Exploring the Interaction Equation: Validating a Rubric to Assess and Encourage Interaction in Distance Courses

The degree of interaction among participants in distance courses is widely acknowledged to be an indicator of successful learning experiences; interaction has been found to contribute to both achievement and student satisfaction. As an increasing number of distance courses move to a completely asynchronous, online format, providing for better interaction is an important means of assuring course quality. To allow measurement and study of interaction in online courses, five elements were identified that contribute to interaction, and a rubric was designed to assess the degree of each in distance courses [1]. A previous use of the rubric in one online class [1] indicated the instrument had good convergent and divergent validity and reliability in terms of consistency of results across students. The current study focused on establishing the rubric's concurrent validity and consistency of results across four distance courses. Recommendations are made to increase the usefulness and reliability of the rubric in practice.


I. INTRODUCTION
Substantial distance learning theory and research has focused on the role of interaction in successful distance learning courses [2,3,4,5,6,7]. Findings indicate that increased interaction in distance courses is associated with higher achievement and student satisfaction [8,9,10,11]. For example, Swan [11] found that two of the three course design factors that contributed significantly to student satisfaction and perceived learning in online courses were related to instructor and student interaction. Although distance learning continues to increase in popularity, and course offerings abound [12], critics of distance learning express doubts that instructors separated by distance from their students can provide the same quality of education as in face-to-face courses [13]. Providing for high interaction among participants in distance courses is a key way to address these criticisms and assure equivalent quality for distance offerings.
Zhang and Fulford [9] offered evidence that interaction is a psychological construct, rather than a technological one, finding that number of completed messages or actual time spent in messaging had little relation to students' perceptions of the degree of interaction in an online course. An analysis of the literature on interaction in distance courses revealed that interaction is not one element, but several interrelated ones and is achieved "through a complex interplay of social, instructional, and technological variables" [1, p85]. From this analysis, five observable, measurable elements were derived that were seen as essential to the "interaction equation," i.e., indicators of qualities that determine interaction in distance courses: 1. social and rapport-building designs for interaction; 2. instructional designs for interaction; 3. interactive capabilities (interactivity) of the technologies used in the course; 4. evidence of learner engagement; and 5. evidence of instructor engagement.
To allow more meaningful examination of the role of interaction in enhancing achievement and student satisfaction in distance, these elements were used to develop a Rubric for Assessing Interactive Qualities in Distance Courses. This rubric was formulated and tested for use in distance learning environments in which an instructor is available to a specific group of students, rather than in courses set up for students to complete in a self-instructional manner. An initial use of the rubric in an online course revealed that it had good convergent/divergent validity, with rubric ratings diverging from course evaluations, as expected, only on the element that measured interactivity of technology resources being used in the course. Consistency across users also was high, with 95% of the students rating the course within four points out of the total 25 [1]. However, that study did not consider concurrent validity or traditional measures of inter-rater reliability for the rubric.
Since past research found a correlation between perceived high interaction in a course and course satisfaction [10], a measure of interaction in distance courses also should correlate highly with course evaluations. Thus, a rubric designed to measure interaction in distance courses should be found to demonstrate concurrent validity if it correlates with students' evaluations of the course. To explore this premise and further validate the usefulness of the rubric as a measure of course interaction, this study expanded the use of the rubric to additional courses and gathered data to indicate concurrent validity and inter-rater reliability.

II. BACKGROUND LITERATURE AND THEORETICAL FRAMEWORK
Much useful work has been done to define interaction as a construct and to identify a theoretical basis for the characteristics of interaction associated with distance learning. However, course design guidelines and subsequent impact research has not kept pace with theory and research. One reason for the lack of transfer from theory to practice in this area is the complex nature of interaction in distance courses and the difficulty of designing assessment and evaluation tools that build on a solid theoretical framework, yet provide sufficiently practical guidelines to make the concept of interaction measurable and useful to distance instructors and researchers.
Jonassen, Peck, and Wilson [14] recommend rubrics as tools for assessing complex performance in a way that gives input and feedback to help improve the performance. Rubrics, which consist of a set of elements that describe a performance, together with a scale (e. g., 1-5 points) based on levels of performance for each element, have become increasingly popular in educational technology as a means of assessing the quality of complex tasks such as digital portfolios [15] and web page creation [16]. When these rubrics are combined with descriptions and examples of effective performances for each component quality, they become a powerful means to clarify expectations and guide performance. Although rubrics have been designed to assess the overall quality of online courses (e.g., see http://www.csuchico. edu/celt/roi/), the rubric described here marks the first time such an instrument has focused on quality of interaction in distance courses and has collected data to validate the instrument in practice.
Of the elements that were determined to reflect observable, measurable qualities of interaction to include in a rubric, two are those over which the instructor has control: designs for social interaction and designs for instructional interaction. A third element focused on interactive qualities of the technologies used in a distance course, while two other elements measure kinds and qualities of messages: learner engagement and instructor engagement. Articulating these elements and providing a scale to measure them is essential to permitting better, more comparable research on the hypothesis that higher interaction of one or more kinds leads to higher achievement and course satisfaction.

A. Social and Rapport-Building Designs for Interaction
Several distance-learning researchers have found that social interactions play a key part in increasing the total interaction of a distance course and can enhance and further learning. Picciano [17] notes that in non-online courses, students often learn from each other in many outside-class situations, sharing insights and information in social exchanges in the library or cafeteria. Since they cannot meet in this way in virtual courses, he says that online courses should be designed to simulate and promote this interaction.
Woolcott [18] says that increasing social rapport among participants helps decrease the "psychological distance" and isolation often experienced by distance students. She finds that decreasing psychological distance has the effect of increasing both motivation and observed interaction in the course and, thus, enhancing learning.
Vrasidas and McIsaac [4] say that students have a need to interact socially, as well as to learn, in a course setting, and Gilbert and Moore find that "social interaction can directly foster instructional interaction" [2, p31]. Activities designed to increase social rapport among course participants may help meet this need, as well as facilitate the learner-to-learner interaction that Moore [3] identifies as one of the essential types of interaction. The scale of interaction for this element ranges from very limited, one-time information exchanges of background information among students to ongoing course activities designed to encourage social and personal interactions. Although social interaction among students is the primary focus of this element, instructor participation is also a factor in decreasing psychological distance. In light of this, higher levels of interaction on this element include both activities designed to increase social rapport among students and instructor.

B. Instructional Designs for Interaction
Wagner [5] points out that instructional design theory and practice impact directly the kind and extent of interaction possible in a distance course. Highly interactive learning environments include specific activities that are designed to encourage, support, and even require interaction. Wagner [6] lists 13 possible outcomes of these activities (e.g., interaction to increase participation, interaction to enhance learner control and self-regulation). These outcomes are especially helpful when viewed in the context of designs that enable collaboration among students and instructor as co-participants in the course, rather than only between the instructor and individual students.
Since Vrasidas and McIsaac [4] find that larger class sizes inhibit student-to-student interaction, distance course designers often cite collaborative learning strategies as a way to increase interaction among participants [19,20,21,22,23]. Small-group, collaborative designs not only require students to interact with each other, they also make frequent, meaningful interaction more manageable. Thus, the scale of interaction for this element ranges from designs that require simple, one-way delivery of information to individual students to those that maximize instructor-student exchanges and make possible collaborations within and among groups.

C. Interactivity of Technology Resources
Wagner [5] noted that technologies vary greatly in their potential to promote interaction. Thus, she differentiates between technology characteristics, which she refers to as "interactivity," and outcomes of using them, which she calls "interaction." While exploiting the potential of technologies depends on the instructional designs in which they are employed, Simonson, Smaldino, Albright, and Zvacek [24] feel that differences in technology characteristics impact critical learning variables such as types and immediacy of feedback mechanisms and methods of presenting and elaborating on information [5], as well as the amount and kind of collaboration techniques possible among participants of small groups. Horn says that "interactivity varies based on the transmission medium" [25, p15], and that "…absence of immediate feedback and nonverbal cues leads to unnecessary anxiety and hostility among students" [25, p13].
Since technologies that permit more visual cues, hypermedia presentations, and two-way, more immediate communications are also assumed to permit higher interactivity than those that allow only written or audio messages and one-way communications, the rubric scale focuses on the range of these qualities. However, it is important to emphasize again that levels of technological interactivity are only potential contributors to interaction. They become meaningful components to promote interaction only in the context of course designs that take full advantage of them.

D. Evidence of Student Engagement
Distance course designers acknowledge that instructors share responsibility with students to promote interactive learning in online learning communities [26,27]. Moore and Kearsley [28] emphasize that instructors can create an environment conducive to high interaction and learner autonomy. Hillman, Willis, and Gunawardena [29] propose that instructors must give students assistance and practice to increase the likelihood of successful use of technologies that allow interaction. However, student response to these requirements and efforts is variable. Although student engagement can vary considerably, despite the instructor's efforts to encourage it, it nonetheless has an impact on the total interaction possible in a distance course. Vrasidas and McIsaac [4] and Brown [30] found that less experienced distance learners participated less frequently and less spontaneously, either for social or instructional purposes. The more comfortable the students are with distance formats, the more likely they are to interact. Roblyer [31] found high student engagement was characterized by voluntary, as well as required, messages that were responsive to the purpose of the discussion. In the rubric, the range of interaction as reflected in student engagement considers three different qualities: required versus voluntary responses, responding versus initiating messages, and the quality of constructed responses.

E. Evidence of Instructor Engagement
As noted previously, Zhang and Fulford [9] illustrated that students' perceptions of interaction in a distance-learning course do not correlate with actual interaction (number of messages, time spent interacting). Rather, perceptions seem shaped by quality and usefulness of interaction. Yacci [7] characterized interaction as a psychological construct influenced both by lag time and coherence (i.e. perceived instructional value) of responses. Kearsley [26] and Simonson and colleagues [24] also find that instructors can either enhance or decrease course interaction depending on how consistently, quickly, and helpfully they respond. As noted previously, Swan [11] found that the most successful online courses are those in which instructor-to-student interaction is both frequent and productive. Thus, evidence of high instructor engagement in the rubric ranges from low to high in the quality, speed, and usefulness of feedback to students.

F. Summary of Elements in the Interaction Equation
These findings indicated that instructors, learners, and technologies are all essential parts of the interaction equation. The interrelationship among these factors in a distance course is shown in Figure  1.This figure depicts interaction as "a created environment in which social and instructional messages are exchanged among entities of a course, and in which messages are both carried and influenced by the activities and technology resources" [1, p81].

III. STUDY METHODS AND DATA SOURCES
Although methods of establishing reliability and validity of rubrics are relatively new, several strategies have been used in recent years. Taggart, Phifer, Nixon, & Wood [32], recommend improving rubric content validity by involving instructors in its development and helping to ensure reliability by training raters. In studies to validate rubrics to assess student writing samples, Baker and Abedi [33] and Novak, Herman, and Gearhart [34] used a variety of methods, including establishing concurrent validity by correlating rubric scores to other measures of writing performance and obtaining an indication of interrater reliability with the "classical techniques" [34] of calculating alpha levels.
Per Taggart colleague's recommendation [32], the Rubric for Assessing Interactive Qualities in Distance Courses was reviewed by 42 distance instructors, and their feedback was used to improve the clarity and comprehensiveness of rubric elements [1]. In the current study, students in four classes in two universities used it as part of post-course evaluations. To make comparisons more consistent, the post-course evaluation instrument usually used at University #1 was used for all courses. This evaluation had two major parts: Part I focused on the instructor; Part II focused on other aspects of the course. Also, students were asked if this was their first online course and their preferred delivery system (online, face-to-face, combination, no preference, or other), and they were allowed to make open-ended comments. For the course at University #1, evaluation materials were mailed to students with a self-addressed, stamped envelope. For courses in University #2, evaluations were administered during the final (face-to-face) class period. In all cases, evaluation results were received anonymously.
One of the four courses in the study was completely online; students and instructor never met, and all interaction took place in asynchronous activities in a web-based course space. Other courses had limited face-to-face interaction (first and last classes meetings in one course, four sessions in the other), with 80-90% of course activities being done online in asynchronous activities in a web-based course space. The final course met face-to-face only twice but had weekly synchronous (videoconferencing) and asynchronous (web-based course space) activities. Of the 92 students who took the courses, 71 agreed to complete the rubric as part of the post-course evaluation activity. Of completed post-course evaluations, 68 (74%) had usable data. The numbers of students and completed evaluations in each course are shown in Table 2.

IV. DATA ANALYSES AND RESULTS
Three kinds of analyses were done to explore the rubric's reliability and validity as a measure of interaction across various kinds of distance courses and to examine the possible contributions of various elements of interaction to total course interaction. These included calculating the following: alpha levels across ratings in each of the four courses, correlations between overall rubric scores and course evaluation scores, and correlations between scores on rubric elements and course evaluation scores. Additional correlations were done between student characteristics (distance learning experience and course format preferences) and course ratings, to gather evidence of whether these affected students' perceptions of course quality or interaction levels. Finally, student comments were analyzed to gain further insights on student perceptions of courses' interactive qualities and the importance of interaction to course quality.

A. Reliability Levels
Using the strategy recommended by Baker and Abedi [33] to determine the level of inter-rater reliability of rubrics, Chronbach's alpha levels were calculated across rubric scores within each online course. For Courses #1-#4, these were: .88, .64, .93, and .95. These results indicate high consistency of ratings across student raters within each course.

B. Correlations Between Rubric Scores and Course Evaluation Scores
To obtain indications of the rubric's concurrent validity, Pearson correlations were done between the rubric scores and the post-course evaluation scores in each course. These results are shown in Table 3. All correlations were found to be significant at the levels shown in the table.

Correlations Between Course Evaluation Scores and Rubric Scores Within Courses
Correlations also were done between rubric scores and Part I and Part II post-course evaluation subscores, across courses, and between rubric scores and the total scores on the post-course evaluations across courses. Also, to examine the contribution of each kind of interaction to total course interaction and help determine whether or not certain types of interaction seemed more related to course satisfaction, correlations were done between each rubric element and the total post-course evaluation scores. These data are shown in Table 4.
All correlations were found to be significant at the levels shown in the

C. Correlations Between Student Characteristics and Course Ratings
Correlations also were done between scores on course evaluations, rubric ratings, and students' experience in distance courses (i.e., item asking "first course or not?") to determine whether students' assessment of interaction or overall course quality was related to their experience as distance learners. Similarly, correlations were done between the evaluation results and their preference of learning format (online, face-to-face, combination, no preference, or other) to determine if their assessment was related to course format preference. No significant correlations were found between any of these variables.

D. Qualitative Analysis of Student Comments
This analysis was intended to provide more insight into student perceptions of each course's interactive qualities and of the importance of interaction to overall course quality. Approximately 40% of the students (27 of the 68) offered comments, but all were fairly terse. Most were complimentary, and only a few offered any information helpful to the purpose of this study. The latter expressed a desire for more direction from the instructor, especially for how they were to use the various components of the online format. These comments had benefit primarily as formative feedback to instructors on ways they could improve their courses.

V. INTERPRETATION AND CONCLUSIONS
Results received to date from uses of the rubric in online or mostly-online courses reflected a fairly high correlation between post-course evaluations and rubric scores, indicating good concurrent validity. Although all correlations were significant, Element #4 (evidence of student engagement) exhibited the highest correlations with the overall course evaluation and with certain components of post-course evaluations. One of the most interesting findings is that technology interactivity (Element #3) correlates with overall evaluations of course quality less than any other aspect of the course. This could indicate that the actual technologies used are less important than instructor and student engagement and the ways activities are designed to promote social rapport and enhance learning.
High inter-rater reliabilities on rubric scores in all courses indicate that results are consistent across students who use the rubric to rate interactive qualities. To decrease variations in students' understanding of the rubric's purpose and meaning and to help accomplish the "training" in instrument use, that Taggart, Phifer, Nixon, & Wood [32] indicate, is important to insure a rubric is used as the developers intended. The authors recommend giving students a set of written directions that has been developed to communicate the purpose of the rubric and what each element measures. Instructors can read these directions aloud and answer any questions students may have before they complete their assessment.
As Roblyer and Wiencke [1] indicate, a rubric which is a valid, reliable measure of various aspects of interaction in distance courses can be used to encourage interaction by students and instructors, as well as to measure how much students perceive it has occurred in given courses. It can also provide valuable feedback for instructors seeking to improve the quality of their online courses. This study used methods employed previously for other kinds of rubrics to help validate the usefulness of these non-traditional, but increasingly popular, instruments.
The results from this study provide further, though still tentative, evidence that the rubric designed to assess interactive qualities is a valid and reliable measure of one important aspect of distance courses. If its usefulness is established in additional courses and larger samples, such an instrument can help further both the design and research of optimal distance learning environments by helping to define and quantify observed interaction and allow empirical assessments of its contributions to course effectiveness. development of the AERA paper on which this article was based.

IX. APPENDIX A RUBRIC FOR ASSESSING INTERACTIVE QUALITIES IN DISTANCE COURSES (© 2004, M. D. ROBLYER)
RUBRIC DIRECTIONS: The rubric shown below has five (5) separate elements that contribute to a course's level of interaction and interactivity. For each of these five elements, circle a description below it that applies best to your course. After reviewing all elements and circling the appropriate level, add up the points to determine the course's level of interactive qualities (e.g., low, moderate, or high) In addition to providing for exchanges of information and encouraging studentstudent and instructor-student interaction, the instructor provides ongoing course structures designed to promote social rapport among students and instructor.
In addition to the requiring students to communicate with the instructor, instructional activities require students to develop products by working together cooperatively (e.g., in pairs or small groups) and share results and feedback with other groups in the class. Permission is granted to use this instrument, either in paper or electronic form, under the following conditions: (1) the purpose is nonprofit research or education; (2) any research summary based on the instrument use cites the research report published in the 2004 article in the Journal of Asynchronous Learning Networks entitled "Exploring the Interaction Equation: Validating a Rubric to Assess and Encourage Interaction in Distance Courses" by Roblyer and Wiencke; and (3) the instrument itself is used in its complete form and displays the following statement: