EFFECTIVE FEEDBACK TO THE INSTRUCTOR FROM ONLINE HOMEWORK

The paper describes different feedback mechanisms available to instructors during the deployment of online formative assessment exercises.


I. INTRODUCTION
Technology has enabled instructors to create and distribute a wide variety of educational materials efficiently.These include numerous types of formative conceptual and algorithmic exercises for which prompt feedback and assistance can be provided to students.While several meta-analyses of the effects of assessment with immediate feedback to the student on learning are positive [1,2], the range of effect size is considerable [3], and can even be negative [2][3][4][5][6].Even within our own model systems CAPA, LectureOnline, and LON-CAPA, when used just for homework, a range of partly contradictory observations were made [7,8].There will not be a general answer to the question of whether or not systems like LON-CAPA are beneficial-after all, they are just tools, not a curriculum.Instead, effectiveness will depend on how they are used, and with which material.There is no doubt however that timely feedback to the instructor, as discussed in this paper, is crucial for ensuring effective use-both during selection and deployment of online educational materials.
Course management systems can and often do record all information transmitted to and from students.That large amount of data, especially in large courses, is much too diluted for instructors to interpret and use without considerable pre-processing [9].

II. THE TOOL
LON-CAPA, (The LearningOnline Network with a Computer-Assisted Personalized Approach) [10], while similar to many others in most aspects, differs in three important ways: • The first is its capability to randomize problems, both algorithmic numerical exercises as well as problems that are qualitative and conceptual, so that numbers, options, images, graphs, formulas, labels, etc., differ from student to student [11].The students can thus (and are encouraged to) discuss the assignments, but cannot simply exchange answers.• The second is in the tools provided that allow instructors to collaborate in the creation and sharing of content in a fast and efficient manner, both within and across institutions, thus implementing the initial goals of the WWW [12].The majority of course management systems are built around the course as the main entity, and learning content is then uploaded to the courses.At the end of the semester, most systems allow export of the content to an instructor's personal computer, and then require re-uploading in another semester.Within LON-CAPA, content is stored independently of a specific course in a shared cross-institutional content pool.• The third is its one-source multiple target capabilities, that is, its ability to automatically transform one educational resource, for example a numerical or conceptual homework question, into a format suitable for multiple uses: the same source code, which is used to present problems for online homework, can also generate them for an online examination, or for a printed version suitable for a proctored bubble sheet examination which is later machine scored [13].A summary of performance results obtained this past decade using our systems for homework, quizzes, and summative as well as formative examinations, has been published [14,15], including studies on the early detection of students-at-risk [16,17].

III. AUTOMATED FEEDBACK MECHANISMS
The amount of data gathered from large enrollment courses (200-400 students) with over 200 randomizing homework problems, each of them allowing multiple attempts, can be overwhelming.Figure 1 shows just a small excerpt of the homework performance in an introductory physics course, students in the rows, problems in the columns, each character representing one online homework problem for one student.A number shown is the number of attempts it took that particular student to get that particular problem correct-"*" means more than nine attempts; "." denotes an unsolved problem; blank denotes an un-attempted problem.This view is particularly useful before the problem deadline, where columns with a large number of dots or blank spaces indicate problems that the students have difficulties with.An important task of the feedback tools for the instructor is to help identify the source of difficulties and the misconceptions students have about a topic.There are basically three ways to look at such homework data: by student, by problem, or cross-cutting.For a per-student view, each of the items in the table in Figure 1 is clickable and shows both the students' version of the problem (since each is different), and their previous attempts.Figure 2 is an example of this view, and indicates that in the presence of a medium between the charges, the student was convinced that the force would increase, but also that this statement was the one he was most unsure about: His first answer was that the force would double; no additional feedback except "incorrect" was provided by the system.In his next attempt, he changed his answer on only this one statement (suggesting that he was convinced of his other answers) to "four times the force"-however, only ten seconds passed between the attempts, showing that he was merely guessing by which factor the force increased.The per-problem view in Figure 3 shows which statements were answered correctly course-wide on the first and on the second attempt, respectively, the graphs on the right show which other options the students chose if the statement was answered incorrectly.Clearly, students have the most difficulty with the concept of how a medium acts between charges, with the absolute majority believing the force would increase, and about 20% of the students believing that the medium has no influence.The concept should be dealt with again in class.
In the analysis illustrated in Figure 3, a simple item-analysis on statements was performed, with only the added difficulty of keeping track of the randomized order in which these statements appeared to individual students.A more sophisticated analysis involves keeping track of the concepts each statement addresses, especially if there is more than one statement addressing the same concept, and different students see different versions of it.To this end, internally, the statements can be grouped into six socalled "concept groups," each focusing on a particular physics aspect of the problem.Every student gets one statement (with the correct labels filled in) from each one of these concept groups.The item analysis on the result in this mode is done by concept group, not by statement, and can thus be carried out independently of the randomization.
The simplest function of cross-cutting statistics tools in the system is to quickly identify areas of student difficulties.This is done by looking at the number of submissions students require in reaching a correct answer, and is especially useful early after an assignment is given.A high degree of failure indicates the need for more discussion of the topic before the due date, especially since early responders are often the more dedicated and capable students in a course.Figure 4 shows a plot of the ratio of number of submissions to number of correct responses for 17 problems, from a weekly assignment five days before it was due.About 15% of the 400 students in an introductory physics course had submitted part or most of their assignment.The data of Figure 4 are also available as a table, which in addition lists the number of students who have submissions on each problem.Figure 4 shows that five of the questions are rather challenging, each requiring more than 4 submissions per success on average (for example, problem 1 requires a double integral in polar coordinates to calculate a center of mass).Note that an error in the unit of the answer or in the formatting of an answer is not counted as a submission-in those instances, students re-enter their data with proper format and units, a skill that students soon acquire without penalty.For the third part, use the y-component of the initial velocity in the equation v^2 -v0^2 = -2*g*(x -h), where v is the y-component of the velocity at the tip of the arc path (...therefore, equals z...), v0 is the ycomponent of the initial velocity, x is the height to find, and h is the initial height (a.k.a.x0; it's given).Now for the y component.My logic was that at half the total airtime, the object would be at the peak of the arc, and thus would be moving at 0m/s, (being in transition from going up to coming down).I tried solving for the initial y velocity using this information and the Vfinal=Vinitial + (A)(t) equation.But still no luck.Any pointers would be greatly appreciated.
Student E: Hey EVERYBODY, whoever did this FORGOT to divide whatever your total change in x is by 2 and use that as the displacement in x to find the V nought x, just a little heads up i.e. i used 175 m/ 2 = 87.5 m (since the object isn't an even parabolic function, its not all of the upside down U shape on the graph) as my displacement for finding V nought x Student F: When using the equation y = y0 + v0*t -.5*g*t^2 the v0 in the second term on the right side is really the initial velocity in the y direction not the total initial velocity.
In general the equation x = x0 + v0*t + .5*a*t^2! is always a one dimensional equation so when you use it in the y-direction all variables are for the y-direction only initial y, initial y velocity, and constant y acceleration ONLY WAY!!! 2.) Sort of kind of eye ball the total distance the object traveled from start to finish.

3.) In sample problem 4-7 in the book they used the
Horizontal Range equation in order to find the answer, but you have to adjust the problem to find Gravity or G.
Here is the ADJUSTED equation so all you have to do is plug in the numbers that you got.G = Initial speed * sin(2*your angle) / Total distance^2 Now the computer gives you some lead way due to the "eye-balling" you have to do, but it gave me my answer and I was 0.08 off.
Hope this helps you guy's.

Student F: what are the units used for this?
Student G: Gravity is acceleration, so the units should be m/s^2.

Student H: Once you plot your points how does this determine your angle?
Student I: I had to do 3 iterations of this problem before getting it right.Assuming the math is done correctly there is not much tolerance in this problem in regards to calculating the launch angle (theta).When I was off by more than 3 degrees I got it wrong.Be VERY careful when drawing the tangent.

IV. FREE-FORM FEEDBACK
Within LON-CAPA, every online resource in a course is automatically associated with a threaded online discussion, which is attached to the bottom of the page.Students and instructors can post anonymously, or with names or screen names.Students are extremely vocal online, even though they are made aware that instructors are always able to see their full names, independent of posting mode.Figure 5 shows the online student discussion associated with two problems which address the same physics, but are different in nature: the left problem is numerical in nature, the right one conceptual.Due to the randomization of the questions, student cannot simply exchange answers with each other, and are forced to discuss the questions on a level which is more insightful to the instructor.Reading at least some of the online discussions before class gives instructors some insight into the general climate in the course, as well as student areas of difficulty.

V. EVALUATIVE FEEDBACK FOR MATERIALS SELECTION
Feedback to the instructor should begin during the resource selection process, particularly when considering material from other authors.The LON-CAPA shared resource pool currently spans over 20 universities and colleges, as well as over 20 high and middle schools and several publishing companies.
As the resource pool grows, selecting an appropriate content resource becomes an increasingly challenging task.In addition to the "Browse" view of the resource pool, instructors can search the cataloguing information.LON-CAPA has two categories of cataloguing or metadata ("data about data") mechanisms: static metadata provided by the authors, such as title, subject, keywords, etc, and dynamics metadata, gathered by the system based on the use of the resource.
The latter provides information similar to amazon.com'sdynamic metadata; that is, it provides information in which context a resource has been used by other instructors, see Figure 6.Besides providing contextual information, the dynamic metadata provides what amounts to a peer-review mechanism: resources that have been selected by a number of instructors presumably are the ones having passed careful consideration by a number of peers.
In addition, the resource selection interface provides evaluative free-form feedback: Figure 7, left side, shows the user interface that is presented to learners and that enables them to submit subjective evaluation data.For each of the statements presented the user can select simple responses from a pull-down menu, ranging from "strongly disagree" (1) to "strongly agree" ( 5).An educator wishing to utilize a given resource is able to look up the metadata on statistical assessment and evaluation, as shown in the right side of Figure 7. Individual comments are only visible to the author of the resource.The comments shown here were actual student responses, and we blacked out their user-ids for privacy reasons.In order to collect these metadata with feedback from individual students, we found that it worked best to assign the production of evaluations as parts of students' honors projects.We were very careful not to make any student grades dependent on participation in the collection of metadata.Assigning even a small part of class credit to completion of metadata evaluation information might compromise the integrity of the data collected in this way.Any instance in which a given resource is used in an exam setting thus collects information on degree of difficulty and discrimination.This information can be archived and used to create random tests that are generated from a large bank of testing resources.The computer can then create tests that do not rely on the selection of the instructor, but instead allow for a comparison relative to an objective standard.This is important particularly when one allows for creation of individual tests for students, in which the questions are allowed to vary from student to student.

VI. CONCLUSIONS
Technology does indeed provide means to get considerable feedback on many aspects of teaching and learning.To make good use of that feedback is a far greater challenge.We have been using LON-CAPA for both formative and summative assessment.
The LON-CAPA course management software has reached a state of maturity; its resource pool has reached a size that now allows novel approaches to old problems.With over 60,000 individual resources, with many tens of thousands of students enrolled each semester at approximately 50 institutions, with automated metadata collection, and with resource sharing across the LON-CAPA member network we have entered a new phase in the use of educational technology.It has now become possible to think about multiple content representations to provide a more customized accommodation of individual learning styles.In addition, we are now in the position to establish more objective measurement tools for learning outcomes that utilize large test banks of individual test items for which standardized statistical information has been collected across many educational settings.
Our ability to detect, to understand, and to address student difficulties is highly dependent on the capabilities of the tool.Feedback from numerous sources has considerably improved the educational materials, which is a continuing task.Analysis mechanisms like the ones provided by LON-CAPA can facilitate research in physics education.Finally, as a result of feedback on students' work, those doing very poorly can be identified quite early.

Figure 1 :
Figure 1: A Small Excerpt of the Performance Overview for a Small Introductory Physics Class

Figure 2 :
Figure 2: Student-centered View of a Problem

Figure 3 :
Figure 3: Compiled Student Responses to a Problem

Figure 4 :
Figure 4: One Early Measure of a Degree of Difficulty

Student C :
How do we use y = y0 + v0*t -.5*g*t^, when we dont have two of the variables (y and v0)?How do we use that formula to get the vo in the y direction?(i.e.what numbers and such do we use?) Student A: What does the magnitude of the gravitational field mean?Student B: i'm guessing acceleration Teaching Assistant: That is correct.You need to calculate 'g' for Planet X. Student C: How are you supposed to do this problem?I am confused, it seems like we have learned nothing during lecture to help us understand these problems, we never do any examples and work thru problems in lecture.please help.Student D: Yeah, I'm totally lost on this one and all I have to look at in my notes are a bunch of variables in an equation, I don't know where to plug in half of the numbers I have.This problem and the catapult one I'm totally lost on cuz all I have to go by are these equations with like 5-6 variables such as y=(tan[d^2y0/dt^2]*x-(g*x^2/2*(vocos[d^2yo/dt^2])^2) and then all the problem tells me is "you threw the rock at 22.8 m/s" or something...I got all the other problems done easily, but this one and the catapult one...I dunno, I just can't figure them out.I worked on them for a while the other day and then got up at like 8:30 today to work on them and still haven't figured them out.:/ Student E: Here is a simple answer to the question, go to sample problem 4-7 in your book and you'll get the answer.But I'll be nice enough to help you out a little more.1.) Lecture we talked about getting the tangent line in order to find the angle, DO THIS!!! Print out the paper and find the angle, IT'S THE Thanks.Student D: Ok.Someone tell me what I'm doing wrong.I figure since they give you the distance traveled in the x direction, and the time it was in the air, you should be able to get the x component of velocity with distance/time.

Figure 5 :
Figure 5: Online Student Discussions on Two Problems, One Numeric and One Conceptual in Nature

Figure 7 :
Figure 7: Left Side: User Interface Used by LON-CAPA to Collect User Evaluation Data; Right Side: Excerpt from the Summary Information Metadata Presented to the Resource Creator