1
2	Outline Introduction Concepts and Calculations: The problem we wish to address Interactive Examples (IEs) Our approach to the problem Demonstration Student Interactions with IEs What do they think of them? How do they use them? Student Performance Assessment tools Initial Results Thoughts for the Future
3	Introduction The Problem to be Addressed: It’s been our experience that too many students see “concepts” and “calculations” as two totally separate and unrelated activities. When given a quantitative question, most students will NOT think about the CONCEPTS that are involved. When given a qualitative question, most students will never consider writing down an appropriate equation… math is NOT seen as a TOOL What are we trying to do about this situation? We are creating instructional materials that we hope will promote “concept-based” problem solving among our introductory students. What Approach are we taking? Create interactive web-based exercises (Interactive Examples) Why have we chosen this direction?? It’s our strength: (long history of CAI at Illinois) Immense Dissemination Potential (asynchronous learning)
4	What is an Interactive Example? Base question is a quantitative problem (multi-step). Students can request help which comes in the form of more questions. These questions are designed to guide the student along a path suggested by the UMASS PERG: Conceptual Analysis: What concepts and/or principles determine what will happen in the physical situation? Graphical representations? Qualitative Behavior? Strategic Analysis: What general approach to take? Develop a plan for applying the principles identified in conceptual analysis. Quantitative Analysis: What are the appropriate equations for this problem. Work out the mathematical solution. Meta Analysis: What have we done? Reflect on, make sense of the previous analyses. IEs use the optional follow-up questions to do this task. Students can opt to answer the base question at any time. Eventually, enough help is given to solve the problem (it is an “example”!) Once the base question is answered correctly: A Recap is given (Conceptual, Strategic and Quantitative Analyses). Follow-Up Questions (optional, i.e. no credit) are asked.
5	Implementation Status Fall 2002 Illinois: 125 IEs in 5 Courses (total of 1800 sudents) Created over past 2½ years by UI PERG: 12 Students, 3 Faculty Elsewhere: Washington, Wisconsin, Ohio State, Rochester, Seattle Pacific, Chicago State, Bradley, Northeastern Ill., College of DuPage, Parkland
6	How Do Students Respond to IEs? They love them! We introduced some IEs to a class in another research university that had been using a well-known web-based homework system (XXX) We Classified the free response Comments from the class: Enthusiastic Positive Neutral Negative Awful 57% 38% 5% 0 0
7	More Comments from Students “This is a better system than XXX because of the feedback/hints given. Also the summary statements at the end are helpful.” “I like these problems because they give you feedback and they help you on reasons you got the answer right or wrong.” “I found that the explainations were definitely helpful in understanding the concepts; however, the fact that many of the answers were multiple choice could possibly lead to abuse of the system by some students.” “The explanations are extremely helpful. I appreciate the hints one is given when the wrong answer is chosen, and the questions are quick.”
8	The REAL Data: Student Logs We record all student submissions on IEs:
9	Initial IE Analysis Analyse IE Logs to determine nature of student interaction with IEs
10	Initial IE Analysis How Do Students Obtain the Correct Answer to the Base Question?
11	Initial IE Analysis How far into Help sequence do students go?
12	Do Students Learn More from IEs? This is a difficult question.. It is a major subject of our current research. Initial assessment tools: Final Exam questions and “Homework B” Questions We can reuse final exam questions because we collect exam copies and have never seen any significant evidence for information transmittal Homework B’s are web-based quizzes. They are not proctored, but, on average there is a strong correlation between homework B performance and exam performance.
13	Sample Homework B Given C₁, C₃, and C₄ Three Qualitative Compare Q₁ and Q₄ Compare Q₁ and Q₃ How does energy change if dielectric added to C₂? Two Quantitative Given Q₁ and E, what is C₂? Given Q₁ and E, what is V₃?
14	Homework B Analysis The Homework B questions have a wide variation in difficulty
15	Homework B: Pre IEs Compare performance on Homework B questions (269 total!) in Physics 112 between Spring99 and Spring00 (pre IEs)
16	Distribution of Differences Histogram the points on the previous plot to get a distribution of the normalized differences: d º (<sp99>-<sp00>)/s
17	All Pre-IE Comparisons Pre-IE Semesters:
18	Post IE – Pre IE Comparisons Physics 112 Post-IE Semesters: Spring 02 and Fall02
19	Physics 112 Analysis Combine Fall and Spring Analyses: Take Sp99 + Sp00 + Sp01 to be baseline for Spring Take Fa99 + Fa00 to be baseline for Fall Form differences: Sp02 – Sp baseline, Fa02 – Fa baseline for each question Combine Fall and Spring differences for each question
20	Questions with Significant Improvement Were improvements made in qualitative or quantitative questions? Qualitative/Conceptual: 35 questions Quantitative: 12 questions Improvement occurs uniformly in both areas fraction of all HWB questions that are quantitative = 25% In what topics were improvements made? Improvements seen in all weeks About 40% of improved questions came in two weeks: Capacitors: no formulas.. Need to reason Polarization and Refraction: Very relevant IEs?
21	Very Preliminary Exam Analysis Compare Two Pre-IE: Form normalized differences of average scores on repeated final exam questions.
22	Very Preliminary Exam Results Combine Sp00 and Sp01 to form Pre-IE baseline
23	Conclusions/Future Plans We have implemented a total of 125 IEs in 5 courses at Illinois Students perceive them to be very helpful to their learning. We have observed statistically significant improvements in student performance, in both weekly quizzes and the final exam, after IEs were added to courses. Next Step? Refine analysis… try to remove fluctuations due to population selection by scaling averages by exam average of population. Extend exam analysis to fall semesters of 112 and combine results with those of spring. Investigate correlations between content/context of IEs with that of improved homework B and final exam questions. Investigate correlations between the nature of student use of IEs (as observed in IE logs) and improved student performance. Once understood, feed back information to improve existing IEs and create new ones.