USING RUBRICS AND CONTENT ANALYSIS FOR EVALUATING ONLINE DISCUSSION : A CASE STUDY FROM AN ENVIRONMENTAL COURSE

This paper presents a case study of using course-specific rubrics combined with content analysis, together with instructor and student feedback, to assess learning via online discussion. Student feedback was gathered via Small Group Instructional Diagnosis, and instructor feedback was collected through formal interviews. Content analysis used emergent coding with different assessment criteria for each phase of the online discussion. Student participation was high, with a number of students feeling they learned beyond what was discussed in class. Some students however were overloaded by the large number of postings and repetitiveness during some of the phases of the discussion. The instructor was pleased to find students who were quiet in class being active in the online discussion. However, he found that student contributions demonstrated insufficient reflection and critical thinking. Content analysis showed that students met, on average, 59-82% of the essential assessment criteria in their postings, and that their contributions significantly improved as the online discussion progressed. However, a limited number of postings reflected critical thinking. In using online discussion, the use of assessment criteria is therefore commendable, as it was found that content analysis gave an insight beyond student and instructor perceptions. The insights gleaned from the methodology indicate its usefulness in assessing online discussion activities more objectively, and with respect to specific learning objectives.


I. INTRODUCTION
The American University in Cairo (AUC) is an American liberal arts university based in Cairo, Egypt.Its language of instruction is English.It follows a semester system, encompassing 15 working weeks.Classes usually meet two to three times per week, and consist of up to 40 students.No degree courses are offered via distance learning, but some instructors add an online component to their courses without reducing face-to-face contact time, using the learning management system WebCT.
The course 'Man and the Environment' is an introductory course to environmental science that students of non-science/engineering majors can choose to take in order to fulfil the general science requirement of their liberal arts degree.The course aims at presenting the principles of environmental science, together with the primary concerns of natural resource management and the major challenges for environmental protection on a global scale.At AUC, this is carried out through four modules.Module I presents the fundamentals of ecosystems and how they operate particularly with respect to energy and nutrient cycles.Module II focuses on the major natural resources and their management.Module III is concerned with environmental degradation and global initiatives to combat it.Module IV relates environmental management and protection to diverse issues such as economic development, policy, law, and sustainable development.The four modules are of different sizes with module III being the largest.This primarily stems from the fact that environmental degradation is reaching serious dimensions globally, and particularly so in Egypt.Students' understanding of details and dynamics of such degradation, as well as ways and initiatives to combat it, is of crucial importance to their commitment to environmental protection.In this respect, it was considered important that students be able to identify issues of primary environmental concern and critically reflect on why these issues persist, while realizing the complexity of their interdependence.Egypt was used as an example of a developing economy faced with serious environmental challenges.Proposing possible solutions to issues of concern would help students appreciate the complementary roles of different entities such as individuals, civil society, regulatory bodies, private businesses, the political establishment, and others.
Class time was not enough for extensive discussion and exchange of experiences and opinions, especially concerning a topic of such complexity.The instructor therefore needed to find an 'innovative' way to carry this out outside of class time.

II. WHY ONLINE DISCUSSION?
Asynchronous online discussion seemed to be an appropriate pedagogical tool to meet the above needs.It allows students to discuss the issues online and outside of class time [1]; it is praised for promoting student reflection [2] and for encouraging those students who are reluctant to speak up in class [3].Moreover, conversations are recorded [4,5], which allows both the instructor and the students to refer to them later.
In this respect, and for the purposes of this course, online discussion was particularly suitable as a significant number of students seemed hesitant to speak up in class.In addition, recent research [6] found that 71% of surveyed AUC students (n = 99) who had online discussion as part of a course they took agreed or strongly agreed that it enhanced their learning experience.However, the investigators were aware of the most commonly cited limitations of online discussion [1,7] , namely, the effect of its textbased dryness on learning particularly for visual learners; too many unread postings causing information overload, potentially de-motivating students and causing them to stop participating; the reluctance of some students to participate altogether; and the fact that some students read but do not post (also known as 'lurking') which is often seen as unfair to the other students.
A majority of students taking the course were found to be already familiar with WebCT, the learning management system used at AUC (74% were moderately or very experienced with it).They were also generally experienced with web technologies such as email (100%) and internet (89%), so no technical access issues were anticipated which might have reduced their participation in the online discussion.
Online discussion had been used within the context of this course for three semesters before this research was carried out.Students were awarded bonus marks for submitting 'good quality' postings.However, this was subjective, and it was considered important to measure the impact of this pedagogical innovation on student learning and motivation.In this respect, it was decided that an 'action research' project be carried out in the Spring 2005 semester, as a collaboration between the course instructor and the Center for Learning and Teaching (CLT) at AUC, to collect feedback from students and conduct a content analysis of the online discussion transcripts.This would result in data gauging student perceptions and measuring actual student learning that had occurred.A significant number of studies have been conducted on online discussion.However, they have either focused on totally online courses (e.g.[2,8,9]) or on blended courses where in-class engagement was difficult because of large numbers of students (a number of the examples in [5,10].In addition, there is insufficient research addressing explicit learning gains achieved [5].Some researchers conducting a study on blended courses with small classes, focused on learner perceptions only [11][12][13], and did not go further to conduct content analysis on student learning.Others [14,15] did recommend the use of rubrics to assess student learning objectively, but did not go further to use rubrics in conducting content analysis.Although some researchers conducted content analysis of online discussion for small numbers of graduate students in a blended learning context, they based their content analysis on a predetermined framework rather than a course-specific rubric set [16,17].Others developed rubrics and analyzed them to assess the effectiveness of interaction in distance courses, but not learning [18].Meyer developed a rubric set to assess the quality of online courses, but again, this was not directly applied for assessing learning [19].
The investigation presented here is unique: course-specific rubrics combined with content analysis, as well as student and instructor perceptions, are used in order to measure learning via online discussion.This is carried out in a course with a small number of non-native English-speaking students meeting three times per week.

III. PURPOSE OF THIS STUDY
This study aims to investigate how online discussion can be used to achieve learning goals and specific learning objectives.In this respect, one of the overarching learning goals for using online discussion was to encourage students to apply what was presented in class to specific examples from their daily experiences in order to deepen their awareness of the issues at hand.A second goal was to assist them in developing their reasoning skills, and encourage critical thinking and analysis instead of memorization.These overarching goals were also translated into specific learning objectives as specified in section IV-B.
This study is unique in that content analysis of the online discussion is carried out based on 'rubrics' specific to the activity's learning objectives, rather than applying a general framework.It is hoped that the results of this study demonstrate the benefit of combining rubrics with content analysis in assessing learning.Moreover, this study explores the usefulness of using online discussion in small classes where students meet regularly, as opposed to large classes and distance courses.

IV. RESEARCH METHODOLOGY
The two known methods of measuring quality of online discussions are 'learner feedback' [11] and 'content analysis' [16,17] with content analysis considered to be the more revealing [9,16].In this respect, an action research approach was followed, triangulating the results of the content analysis of online discussion transcripts with feedback from students and the instructor's own perceptions.

A. Structure of the Online Discussion Activity
The online discussion was structured into four separate but related cycles, each cycle addressing a certain aspect of the environmental status in Egypt.The length and timing of each cycle would depend on student feedback, lasting from about one to two weeks, and beginning on the same day the preceding cycle ended.The topic tackled in cycle one ("What do you think the major environmental challenges in Egypt are?") was set in advance, but the exact topics tackled within cycles two to four were not pre-set.The progression of these topics was developed in response to the way student participation and feedback proceeded.Students would get 'bonus' marks for posting 'meaningful contributions'.However, they were not given any assessment criteria.The cycle topics and learning objectives are included in Table 1.

B. Content Analysis
Some researchers use pre-existing cognitive frameworks to conduct content analysis on online discussion transcripts, for example comparing Bloom's taxonomy and Perry's Intellectual Development model [17], while others use frameworks specifically developed for online discussion or develop their own [16,20].Although this approach has the advantage of focusing on certain aspects of the learning activity, it limits the scope of conclusions, hides other aspects of data not present in the framework, and is unlikely to directly meet every learning activities' unique objectives [21,22].It was decided not to use any of the available analysis frameworks in the literature, such as that of Henri [20], or Gunawardena, Lowe and Anderson [23], or the Transcript Analysis Tool [22], because they were either too sophisticated for the level of the students (e.g.Henri's metacognition category was unlikely in any of the postings) or they were developed for a different type of learning activity (e.g.Gunawardena's framework refers to a learning activity that involves argumentation and resolving a problem collaboratively).Using any of these would not allow effective assessment of learning objectives specific to the particular course and activity at hand.However, some features of Henri's [20] model were adopted in the current research, by including quantitative data on participation, and keeping track of postings that were 'reactive', i.e. offering a reaction to the contents of other postings (corresponding roughly to Henri's 'interactive' category).
In this respect, a number of assessment criteria, 'rubrics,' were developed by emergent coding to be used in the content analysis.Emergent coding is a methodology that builds coding categories from existing transcripts instead of applying a pre-existing external framework to the transcripts.In the case of this investigation, the transcripts are the online discussion transcripts, and this approach was deemed more appropriate for the investigation since none of the existing frameworks could directly meet the learning objectives set by the instructor.
The authors read through the online discussion transcripts together, keeping in mind the primary learning objectives, as specified in Table 1, and then devised a set of rubrics to measure these learning objectives.The rubrics for the different cycles were therefore different, depending on the set learning objectives for that cycle.However, they also encompassed some generic criteria applicable to all cycles, such as correctness of information supplied, and originality.The posting was used as the unit of analysis, and all the cycle rubrics were applied to it.Scoring was carried out in a binary fashion with a rubric receiving a score of either '1' (yes) or '0' (no) depending on the content of the posting.Some of the rubrics were essential in assessing the learning objectives, whereas others were a 'bonus' (i.e. the students were not expected to meet them, but those who did were considered more advanced).For example, one rubric "Original" was defined as "posting contains at least one original idea," as opposed to one which is repetitive of another posting, even if reworded, and this "originality" criterion was expected of all students.An example of one of the "bonus" rubrics is one that was named "Building" and is determined on whether a "posting builds upon what is in another posting, developing it and/or adding new information," with the purpose of demonstrating if students have built on what their peers have contributed as opposed to either ignoring it or repeating it.The details of these rubrics and the points they investigate are presented in Table 2.

Rubric Name Explanation
[AwareAccounts] Does the posting demonstrate an awareness of the accountability of different entities?
[Building] Does the posting build upon what is in another posting, developing it and/or adding new information?
[CauseEffect] Does the posting identify the relationship between the cause and the effect?

[Connections]
Does the posting make the connection between several challenges?
[Critical] Does the posting demonstrate critical evaluation of the solution it suggests?
[Direct] Does the posting refer to a direct environmental challenge?(as opposed to an indirect one resulting from another more basic challenge).
[Effects] Does the posting specify the effect(s) of the environmental challenge(s)?
[Implementable] Is the method suggested in the positing for addressing the causes of persistence of environmental challenges implementable?

[JumpsToSolutions]
Does the posting specify solutions?[MultipleChallenges] Does the posting specify more than one challenge?

[MultipleReasons]
Does the posting specify several reasons to the persistence of environmental challenge, and/or several effects of an identified reason?
[MultipleRoles] Does the posting demonstrate an awareness of multiple roles and responsibilities?

[MultipleSolutions]
Does the posting suggest several solutions to the identified reason?

[NotIncorrect]
Does the posting contain incorrect information?
[OnTopic] Is the posting within the topic under discussion?(as opposed to addressing a topic unrelated to the discussion).
[Original] Does the posting contain at least one original idea?(as opposed to one which is repetitive of another posting, even if reworded).

[OverallCauses]
Does the posting refer to general overall causes of the environmental challenges (such as lack of awareness, legislation, enforcement of legislation, financial constraints)?
[RelatesCauses] Does the posting specify how several causes interact to render an environmental challenge persistent?

[RelatesDiff]
Does the posting establish a connection between different causes for the persistence of environmental challenges and any one particular solution?

[Relevant]
Does the posting address issues that exist in Egypt?
[Sources] Does the posting specify the source(s) of the environmental challenge(s)?
[Totality] Does the posting demonstrate that the student is aware where/how the specified challenge fits in the total environmental status?The rubrics were developed taking into account the learning objectives, and then applied to sample sets of postings for each cycle by both investigators collaboratively.These postings were chosen so as to form as much of a representative sample with regards to length and content as possible.This collaborative coding approach was carried out to hone the rubric definitions, increase objectivity, and achieve a level of interrater agreement on how to interpret each rubric and assess postings.Rubrics developed for each of the four cycles are presented in Table 1.All postings, including the sample set, were then coded using the finalized rubrics.To ensure intra-rater reliability, the resulting coding of the sample sets was then compared to their initial coding.Coding was not carried out by both authors on purpose, since in an authentic situation, assessment of online discussions is likely to be carried out by the course instructor only, with the emergent coding approach ensuring consistency of the application of each rubric.The finalized coding was then used to analyze patterns in all postings.A tailor-made Access-Excel package was developed for coding and for analyzing patterns in all postings.

C. Instructor and Student Perceptions
The instructor's perceptions were collected via two formal interviews and several informal discussions throughout the semester.On the other hand, student feedback, giving an insight into the educational situation [24] was collected via Small Group Instructional Diagnosis (SGID).This is a structured group interview process [25] that takes place in the absence of the instructor.Groups of students are asked about what they believe helps them learn in a course and how improvements could be made.SGIDs produce both written and oral feedback.For this research, a number of questions directly pertaining to the students' perceptions of the online discussion activity were added: 1. Do you feel that the online discussions have helped you connect the course with real life (Egyptian environmental problems) more than if you hadn't had them? 2. Did you know beforehand that the instructor uses online discussions and that it could improve your grade? 3. Do you feel the online discussions have motivated you? 4. Is your opinion online affected by what the instructor seems to think is a good answer? 5. What could have been done to improve online discussions?[this question was created on the spot to address the issues that students said came up with online discussions] In addition, a demographic survey was conducted at the beginning of the semester.Laurillard [26] advocates conducting such surveys to understand more about students' backgrounds and what they bring to the course.The survey included questions on students' familiarity with technology, to rule out its lack as a possible hindrance to effective student participation.The majority of students were very familiar with web technology, and therefore it can safely be concluded that it does not present a hindrance to any of them for contributing to the online discussion.

A. Instructor Perceptions
According to the course instructor, student engagement in online discussion can demonstrate significant variations from semester to semester.For the Spring 2005 semester, during which the current research was conducted, a significant number (25%) of students were quiet in class but very active online.This was primarily due to limited spoken English skills, which made these students reluctant to speak up in class.The course instructor was satisfied with the overall student performance in online discussion, although his perception was that their critical thinking skills were not developed well enough, and that some students just posted without giving their writing much thought.This was based on the absence of clear indications in the postings that students were capable of recognizing relations existing between environmental challenges which they had successfully identified; that they were aware of multiple causal relations, as for example between one cause and several effects, and/or several causes to one effect; that they were aware of the limitations and/or difficulties of some of the solutions they were putting forward for addressing identified environmental challenges; and that they were aware that some challenges could be addressed through different possible solutions.Another issue of concern was the recurring convergence of a significant number of students on one issue, failing to look beyond it into others, or to develop it further.This was of particular concern when the issue was an obvious one, not necessitating much debate, or discussion (e.g. the lack of awareness being a cause of environmental degradation), thus leading to considerable repetitiveness in the postings.

B. Student Perceptions
When students were asked for feedback about online discussion in an SGID conducted during the fourth cycle of the discussion, ten students wrote that online discussion was an important factor in helping their learning in the course; six said online discussion helped them connect the course with environmental issues in Egypt, and six said the activity motivated them.These numbers need to be considered in light of the number of students who contributed to online discussion (32 students) rather than the total number of students in class (40 students).Furthermore, some students complained of 'information overload', stating that they stopped contributing to online discussion when postings became too many and too repetitive, with postings concurring to stated opinions without any further contributions.This is indeed valid feedback considering that the SGID conducted during the fourth discussion cycle, came right after the third cycle, which had the highest percentage of repetitive postings (Figure 1).Not applicable

Table 3. Quantitative Details for Each of the Four Cycles of the Online Discussion
Table 3 presents quantitative details for each cycle.The number of student postings and the number of students participating are indicative of the degree of engagement of the class as a whole, whereas the average number of postings per student and the average word count per student posting are indicative of the level of engagement for each of those students participating in the discussion.Figure 1 presents the number of 'original' versus 'repetitive' postings, and Figure 2 presents the number of student postings and those of the instructor for each cycle.The overall trends in these variables throughout the four cycles are summarized in Table 4.

Student Engagement
The results show that as online discussion progressed, an increase in student engagement occurred.This is reflected in the increase of the total number of postings, as well as the average number of postings per student.The decrease in the average number of student postings per day should not be considered a contradiction to the general increase in student engagement.It is believed that this is due to external factors such as the approach of the end of the semester with all the associated deadlines of term papers and projects.Students getting busier with other required course work (as opposed to the 'bonus' online discussion), would post fewer messages per day.Nevertheless, the total number of postings per cycle increased and the average number of postings per student increased steadily, demonstrating a continued interest and engagement in the online discussion.Fewer student postings per day resulted in prolonging the last two cycles: it took more days to discuss the topics at hand thoroughly.Instructor engagement in the first three cycles was similar, and decreased significantly in the fourth cycle (Figure 2).This ruled out the possibility of the instructor's engagement in the discussion as being the cause of the increased student participation.

Repetitiveness
Regarding the repetitiveness issue which some students raised during the SGID, it was found that it started by being exceptionally low (13%) in the first cycle, increasing to 46% in the second cycle, and reaching a noticeable maximum of 64% for the third cycle, then decreasing once more to 40% in the fourth cycle.These figures need to be considered in light of the instructor encouragement (in his postings) of students reacting to one another's postings.Though a number of students succeeded to do so, using the content of their colleagues' postings to further develop ideas and express opinions, a significant number agreed to others' posting contents, repeating (though through rewording) the same ideas of the posting reacted to with no additions.The correlation between the exceptionally high occurrence of repetitions and the topic of Cycle III is noteworthy and is further discussed below.

Meeting Rubrics
Tables 5 and 6 present details of posting content analysis for the four cycles of the online discussion.It is clear from  The mode of rubrics met, as well as the frequencies at these modes, also improved as cycles progressed.
In Cycle I, six messages (out of 23, i.e. 26%) met five of the seven essential rubrics.For Cycle IV, 28 messages (out of 55, i.e. 51%) met all six essential rubrics.Again, Cycle III was the exception to this upward trend, where 15 messages (out of 51, i.e. 29%) met only three of the six essential rubrics.
Cycle III data could be a reflection of a limited understanding of students of the topic under discussion "How do you think the causes for the persistence of environmental challenges in Egypt today could be addressed?"(i.e.why are these environmental challenges persisting?).Indeed, the percentage of postings meeting the rubric [OnTopic] is the lowest (82%) for all four cycles, as shown in Table 6.This limited understanding might explain the high level of repetitions seen in Cycle III.Some students not being very clear about the topic of discussion for the cycle, would tend to react to their colleagues' postings (particularly if they have been encouraged to do so by the instructor

Connecting the Course to Egypt's Environmental Problems
The overall objective for using online discussion was to help students make connections between the course contents and Egypt's environmental problems, as an example for a developing country with significant environmental challenges.The rubric [Relevant] was one directly addressing this point.It was met by 78% and 90% of postings in Cycles I and II respectively, and then met by 100% of postings in Cycles III and IV, demonstrating that this overall objective was successfully met by those students taking part in the online discussion, and that student increased engagement through the four cycles assisted in this.

VI. CONCLUSIONS
This investigation combines the use of rubrics and content analysis to both assess individual student learning and to gauge the entire class learning outcomes from online discussion.This is applied to a small class meeting regularly.
Using rubrics allowed objective assessment of learning against the instructor's cycle-specific learning objectives.Rubrics proved more beneficial than the instructor's own subjective assessment because they showed more learning than expected: the instructor could see how well students were doing with respect to the over-arching learning goals, but the detailed rubrics pointed out specific areas of strength and weakness in each student's learning.
The use of emergent coding also proved more beneficial in practice than the use of pre-determined frameworks as it allowed an assessment specific to the activity's learning objectives.In addition, content analysis captured the wider picture of the entire class learning from the online discussion.It allowed the instructor to identify the learning objectives which were most difficult to achieve and the ones which were widely met.In this respect, the combination of rubrics and content analysis demonstrate that student learning was greater than student and instructor perceptions, and pinpoints a number of possible improvements to the activity which the instructor can apply in the future.These encompass informing students of the assessment criteria (rubrics) in advance in order to guide them towards the learning objectives, which is common practice for rubrics [27]; focusing questions and feedback on addressing specific deficiencies; dividing students into a number of smaller discussion groups to limit repetitiveness and information overload, as suggested by McConnell [1]; and referring to ideas expressed in the online discussion during class meeting times, thus encouraging students who made contributions [16].
In addition to the above, results also demonstrate that online discussion can be beneficial even when classes are small and meet regularly as it allows more in-depth appreciation of content implications, and a venue for quiet students to interact.Although some research [28] warns that learners for whom English is not a first language may suffer in online discussion, this investigation shows that, on the contrary, online discussion seems to be a better forum for such learners to express themselves, as they have more time to reflect and compose their answers than they would have for an oral discussion in the classroom.

Figure 1 .
Figure 1.Number of 'Repetitive' Postings for the Total of Student Postings

Figure 2 .
Figure 2. Student and instructor postings for Cycles I to IV of the online discussion

Table 4 . General Trends for Contributions to Online Discussion
Table 5that as the online discussion progressed, the quality of the postings generally improved.The percentage of rubrics met, on average, increased.In Cycle I, 63% essential rubrics were met on average, whereas for Cycle IV 81% essential rubrics were met on average.The exception to this upward trend is Cycle III where 60% of essential rubrics were met.

Table 5 . Summary of Posting Content Analysis
), by mostly agreeing to what is in them without further elaboration, criticism or development.These reactions would present the basis for further reactions, etc.

Table 6 . Percentages of Essential and Bonus Rubrics Met in Student Postings 4. Critical Reasoning Skills A
number of rubrics reflecting critical reasoning skills need to be more closely considered.For Cycle I, the essential rubrics [Totality] and [Effects] were met by 39% and 30% of the postings respectively, demonstrating that a significant number of students, though successfully identifying valid environmental challenges, were not clear about how the identified challenges related to one another and/or contributed to the detrimental environmental conditions.For Cycle III, the essential rubrics [Implementable], [Original], and [RelatesDiff] were met by 41%, 35%, and 8% of the postings respectively.The [Original] rubric reflects the high level of repetitions as discussed above.The [RelatesDiff] and [Implementable] rubrics scores demonstrate that multiple cause-effect relations were very seldom considered by students and solutions were suggested without giving much thought to whether they were implementable.Indeed this is also reflected by the very low scores of the bonus rubrics [Critical], met by 12% and 9% of postings in Cycles III and IV respectively, and [MultipleSolutions], met by 10% of postings in Cycle III.