Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE)

Student engagement is critical to student learning, especially in the online environment, where students can often feel isolated and disconnected. Therefore, teachers and researchers need to be able to measure student engagement. This study provides validation of the Online Student Engagement scale (OSE) by correlating student self-reports of engagement (via the OSE) with tracking data of student behaviors from an online course management system. It hypothesized that reported student engagement on the OSE would be significantly correlated with two types of student behaviors: observational learning behaviors (i.e., reading e-mails, reading discussion posts, viewing content lectures and documents) and application learning behaviors (posting to forums, writing e-mails, taking quizzes). The OSE was significantly and positively correlated with application learning behaviors. Results are discussed along with potential uses of the OSE by researchers and online instructors.


Introduction
Despite the decrease in higher education enrollment overall, online instruction is still growing.A recent report funded by the Online Learning Consortium (Allen & Seaman, 2013) found that 6.7 million students (about 32% of all college students) were taking at least one online course.Indeed, in the analysis of 2,820 institutions (of a potential 4,527 active degree-granting institutions of higher education in the United States), over 69% of chief academic officers felt that online learning is important for the future of their institutions.The vast majority of these officers (77%) believe that online learning is as good as or better than traditional brick-and-mortar learning.
Certainly, research comparing face-to-face and online courses has demonstrated that online courses can be as effective as traditional face-to-face courses (Maki & Maki, 2007;Robertson, Grant, & Jackson, 2005;Zhao, Lei, Lai, & Tan, 2005).Thus, scholarship has moved beyond comparing online and face-to-face classes to exploring ways to enhance teaching and learning in the online environment (Durrington, Berryhill, & Swafford, 2006;Gaytan & McEwen, 2007;Levy, 2008;Young, 2006).Creating and validating research tools to measure various aspects of the online teaching environment is an important part of advancing research about online learning (Roblyer & Wiencke, 2004).Such tools may also provide feedback to instructors about individual courses.
Student engagement is, generally, the extent to which students actively engage by thinking, talking, and interacting with the content of a course, the other students in the course, and the instructor.Student engagement is a key element in keeping students connected with the course and, thus, with their learning (Dennen, Darabi, & Smith, 2007;Kehrwald, 2008;Robinson & Hullinger, 2008;Shea, Li, & Pickett, 2006;Swan, Shea, Fredericksen, Pickett, Pelz, & Maher, 2000).Therefore, the ability to effectively measure student engagement is necessary for online researchers and instructors.The Online Student Engagement scale (OSE) provides that.This paper will discuss the conceptualization of student engagement and the previous steps in creating, assessing the reliability of, and validating the OSE.It then presents the current study, which further validates the OSE's ability to measure student engagement.

Literature Review
Student engagement has been conceptualized in multiple ways across researchers and disciplines (Azvedo, 2015).To explain this abstract concept, the key ideas in the research on online student engagement are explored: social construction, the widely applied Community of Inquiry model, and the particular importance of student engagement to online learning.

Social Construction
Social constructivist theories, such as those created by Vygotsky (1978) and Bandura, Ross, andRoss (1961, 1963), posit that we learn through social interaction.Students may perform a set of actions by themselves but will perform better when allowed to work collaboratively with others.This difference between what students can perform by themselves and what they can perform with others is Vygotsky's "zone of proximal development" (Ally, 2004;Anderson, 2004;Ashcraft, Treadwell, & Kumar, 2008;Hrastinski, 2009;Stacey, 2002;Vygotsky, 1978;Woo & Reeves, 2007).In an online discussion, for instance, students can help each other by filling in the gaps in each other's knowledge and/or by "demonstrating" particular tasks.The zone is the reason interaction with the instructor and with other students is so important to learning.Bandura et al. (1961Bandura et al. ( , 1963) ) illustrated that students can also learn by observing others' behaviors.In the online course, such observational learning may occur when students read arguments posted by other students or the instructor.These become "models" for learning.Similar processes could occur for shared papers, wikis, and so on.This move toward more active learning and interaction with students is particularly important in the online environment, where the challenges of lack of synchronicity (not being online at the same time) and lack of placedness (not being in the same geographical location) have to be overcome (Anderson, 2004).To overcome these challenges, researchers recommend creating courses that encourage three characteristics: social presence, community, and meaningful interaction (Ally, 2004;Bigatel, Ragan, Kenan, May, & Redmond, 2012;Dow, 2008;Hill, Song, & West, 2009).
Briefly, the need for active learning and interaction means that students need to feel as if they are dealing with real people (social presence), that they belong in some way with/to this group of learners (community), and that they are involved in sharing, negotiating, arguing, discussing, and perspective taking (meaningful interaction) (Wang, 2008;Woo & Reeves, 2007).According to social constructivism, this type of interaction/engagement is necessary for learning (Ashcraft et al., 2008;Ally, 2004;Bigatel et al., 2012;Hrastinski, 2009).Hrastinski (2009) defines online learner participation as "a process of learning by taking part and maintaining relations with others.It is a complex process comprising doing, communicating, thinking, feeling and belonging, which occurs both online and offline" (p.1761).
Some of the implications that constructivism has for online learning are similar to the guidelines it creates for traditional instruction: Learning should be active, allow students to construct their own knowledge, make effective use of collaborative and cooperative methods, and be meaningful to students (Ally, 2004).In this way, online learning environments promote social presence and community while creating meaningful interactions.Under these conditions, significant learning is more likely to occur.The Community of Inquiry model (CoI) provides a clear framework for applying social constructivist ideas to the online learning environment.

Community of Inquiry Model
The Community of Inquiry model discusses three "presences" that are necessary for an effective community of learners: social presence, teaching presence, and cognitive presence (Akyol & Garrison, 2011;Akyol & Garrison, 2014;Annand, 2011;Garrison, 2007;Arbaugh, 2008;Garrison & Anderson, 2003;Garrison & Arbaugh, 2007;Shea et al., 2010;Stodel, Thompson, & MacDonald, 2006).Social presence, as discussed earlier, is the ability of learners to share more than "just the facts" and to feel they are communicating with real people in cyberspace (Kehrwald, 2008).When social presence occurs, students feel they are communicating their emotions and attitudes and interpersonally connecting with others (Garrison & Arbaugh, 2007).Online researchers emphasize social presence as a key factor in student engagement.Researchers have found social presence to be positively related to students' learning and their sense of being connected within the class (Shea et al., 2006), higher performance on writing assignments (Picciano, 2002), and student satisfaction (Dennen et al., 2007).Garrison et al. (2000) found that "social presence marks a qualitative difference between a collaborative community of inquiry and a simple process of downloading information" (p.96).Thus, online students need to feel that they are not alone in their learning, but connected to a group of learners.We have yet to reach the stage of technological sophistication (although we are probably close) when one can feel engaged with nothing more than a computer, in the sense of feeling as if one is working with someone else to create knowledge, solutions, and so on.Students need to feel that they are working with real people: their peers and instructors.Social presence is a necessary but not sufficient component of student engagement (Dow, 2008).
Cognitive presence includes the practical inquiry model, which moves students' thinking/discussion from a triggered event that makes them aware of some new idea, concept, or problem to exploration of the new information, integration of ideas, and finally to resolution of the problem (Akyol & Garrison, 2011).Teaching presence is about course design and organization, discourse facilitation, and direct instruction (Akyol & Garrison, 2014;Garrison, 2007).Research has supported the relationship between one or more of the presences and perceived learner support and perceived learning (Akyol & Garrison, 2014;Arbaugh, 2008); student satisfaction and sense of community (Garrison, 2007); and higher order learning outcomes (Akyol & Garrison, 2011).Thus, CoI offers a strong model for researching about online courses as well as designing effective online learning environments.

Importance of Student Engagement to Online Learning
Social construction in general and the CoI framework in particular support the need for student engagement with content, other students, and the instructor.Swan et al. (2000) and Chickering and Ehrmann (1996) agree.Swan and her associates (2000) found that the three factors associated with successful course design and students reporting high levels of learning and satisfaction were (1) frequent and quality interaction with instructors, (2) a dynamic discussion (interaction with classmates), and (3) a transparent interface (easy navigation).Likewise, Chickering and Ehrmann (1996) hold that good practice must include student-faculty contact, cooperation among students, and active learning.In general, the research about online learning supports the idea that student engagement is crucial to student success (Dennen et al., 2007;Kehrwald, 2008;Robinson & Hullinger, 2008;Shea et al., 2006;Swan et al., 2000).Creating a learning environment that is cohesive and interactive (Gaytan & McEwen, 2007) ameliorates a major issue with online courses: students' feelings of isolation (Lewis & Abdul-Hamid, 2006;Ortiz-Rodriguez, Telg, Irani, Roberts, & Rhoades, 2005;Russo & Campbell, 2004;Song & Singleton, 2004) by providing opportunities for students to create connections with the instructor, students, and course content (Young, 2006).

Defining Student Engagement
As stated earlier, while there are strong theoretical foundations and a very useful model for engagement, student engagement, as a term, is not well defined.Kuh (2003) sees engagement as "the time and energy students devote to educationally sound activities" (p.25).His definition gave rise to the National Survey of Student Engagement (NSSE).The NSSE benchmarks five clusters of activities indicating student engagement, including level of academic challenge, a supportive campus environment, enriching educational experiences, student-faculty interaction, and active and collaborative learning (Robinson & Hullinger, 2008).The NSSE perspective on student engagement considers the entire collegiate experience, both inside and outside of the classroom.Other measures focus more on student engagement within the classroom.One of these is Handelsman, Briggs, Sullivan, and Towler's (2005) measure of traditional classroom student engagement.They found four factors illustrating how students devote time and energy in the classroom: skills engagement (keeping up with readings, putting forth effort); emotional engagement (making the course interesting, applying it to their own lives); participation/interaction engagement (having fun, participating actively in small group discussions); and performance engagement (doing well on tests, getting a good grade) (Handelsman et al., 2005, p. 187).They see student engagement as containing both affective and behavioral components.Combining social constructivist notions of learning, the CoI model, and previous incarnations of engagement in the traditional classroom, a description of online student engagement emerges: Engagement involves students using time and energy to learn materials and skills, demonstrating that learning, interacting in a meaningful way with others in the class (enough so that those people become "real"), and becoming at least somewhat emotionally involved with their learning (i.e., getting excited about an idea, enjoying the learning and/or interaction).Engagement is composed of individual attitudes, thoughts, and behaviors as well as communication with others.Student engagement is about students putting time, energy, thought, effort, and, to some extent, feelings into their learning.Therefore, the OSE attempts to measure what students do (actively and in their thought processes) as well as how they feel about their learning and the connections they are making with the content, the instructor, and other students in terms of skills, participation, performance, and emotion.

Previous Work on the OSE
Because the previous work of creating and assessing the reliability and validity of the OSE is detailed elsewhere (Dixson, 2010), a brief summary is presented here.The OSE was initially created using a four-step process: (1) reviewing existing measures of student engagement; (2) conducting a focus group to discuss how those measures would need to be changed for the online environment; (3) creating a pilot of that initial instrument; and (4) performing a test of the instrument.

Existing Measures
The literature about measuring student engagement indicates three potential measurements: Roblyer and Wiencke's (2004) Rubric for Assessing Interactive Qualities of Distance Courses (RAIQDC), Ouimet and Smallwood's (2005) Classroom Survey of Student Engagement (CLASSE), and Handelsman et al.'s (2005) Student Course Engagement Questionnaire (SCEQ).The RAIQDC measures interaction by asking students about other students' behaviors.Given that a good deal of student behavior is not accessible in the online environment, particularly students' affective responses, the RAIQDC was not acceptable.The CLASSE asks students to report on their own behaviors both within and outside of a particular class, but it was designed to be used with a separate faculty CLASSE so that areas of importance to faculty can be matched to areas of importance to students.This allows faculty to see where there may be areas of incongruence between what they feel is important and what their students value (Ouimet & Smallwood, 2005).While this is a useful tool based on NSSE items, it lacks an explanatory foundation (other than NSSE) for inclusion/exclusion of particular items.
The SCEQ was more theoretically sound.As previously mentioned, Handelsman et al.'s (2005) survey does not conceive of engagement as one characteristic or only as behaviors but as based on multiple factors: skills engagement (what students "do"); emotional engagement (how connected they feel to the course/content, which is especially important in online courses; how applicable they feel it is); participation/interaction engagement (interacting with others, enjoying the content/course); and performance engagement (students' desire/goal to succeed in the course).The notion that students get emotionally involved and need others to learn fits with social constructivist ideas about learning as well as the CoI model of social, cognitive, and teacher presence.It also fits with research findings regarding the need for meaningful activities and connection in online classes.So the SCEQ held a stronger theoretical foundation about engagement and measured not just perceptions of attitudes but also perceptions of behaviors.Therefore, Handelsman et al.'s (2005) SCEQ was used to form the basis of the online survey.However, since this measurement includes items such as "coming to class everyday" and "raising my hand in class," adjustments to fit the online environment were needed.Table 1 summarizes essential shortcomings of each measure.Designed to be used with CLASSE Faculty SCEQ Needed to be adapted to online environment

Focus Group
To make these adjustments, a focus group of five online instructors met to discuss what each of the four factors (skills engagement, emotional engagement, participation/interaction engagement, and performance engagement) might "look like" in the online environment.They identified a set of 30 behaviors that might represent Handelsman et al.'s (2005) four factors in an online course (see Dixson, 2010, for complete description).For instance, items such as "taking good notes in class" were replaced with "taking good notes over PowerPoints or video lectures."Likewise, "participating in small group discussions" was revised to "participating in small group discussion forums."From that process, an initial 30-item questionnaire was created.

Pilot
The third step in creating the scale consisted of a pilot of the 30 items created by the focus group.Participants for this pilot were 31 students in online communication courses at a regional midwestern university.The results indicated strong reliability (Cronbach's alpha = .95)and significant correlation with two global items on engagement with the course (r = .73;p < .01),one global item of social presence (getting to know other students), and one global item of teacher presence (getting to know instructor) (r = .38;p < .05)(Dixson, 2010).Thus, initial reliability as well as an expert jury (focus group) and concurrent validity (correlation with the global items) were supported with the pilot.

Test
Once the initial three steps were accomplished, the fourth step was to test the engagement scale with a larger and more diverse pool of students.To that end, 186 students across 38 courses from six campuses of a large midwestern university (the main campus and five regional campuses) completed the OSE.As an incentive for instructor participation, aggregate data was shared with instructors if five or more students from a course participated.To alleviate students' concerns about instructors having access to the data, no demographic data was collected.
Factor analysis yielded four factors: skills, emotion participation, and performance of 19 items loading .60 or higher (Dixson, 2010).These 19 items indicated strong reliability (alpha = .91)and significant correlation with a course global engagement item (r = .67;p < .001).(See Table 2 for a listing of how the items distributed across factors.)As in the pilot, these 19 items were significantly correlated with both instructor presence and social (student) presence.An unexpected finding of this study was that students who spontaneously reported multiple channels for communicating with other students and the instructor (i.e., e-mail, discussion forums, etc.) reported significantly higher levels of engagement.

Get to know other students
Completing this final step meant that the OSE had exhibited face, expert jury, and concurrent validity along with reliability, and could serve as a reliable indicator of student engagement in the online learning environment.It contained the four factors expected: skills (i.e., staying up on readings, listening/reading carefully), emotional (i.e., applying course material to their lives, really desiring to learn the material), participation/interaction (i.e., participating actively in small-group discussion forums, helping fellow students), and performance (i.e., getting a good grade, doing well on tests/quizzes) engagement.One concern remained: There was no external validation of engagement, only students' perceptions.Thus, another step was needed.

Next Step
Since all of the data used to accomplish the previous four steps was self-reported, it was conceivable that, while the OSE was consistently and reliably measuring something, it might not be student engagement.An external (outside student perceptions) measure was needed to further validate the instrument.If the OSE was indeed valid, it should have correlated with actual student behavior within the class.While the OSE measures characteristics of student engagement beyond behaviors, behavioral engagement is part of overall student engagement and, thus, should be correlated with self-reports of student engagement.Another study comparing observable (not self-reported) student behaviors with the engagement measure was needed.In the online environment, those behaviors are often tracked by the learning management system.

Current Study
Because of the need to compare observable behaviors with the engagement measure, the current study proposed to correlate the OSE with actual online student behaviors as tracked by course management software.The tracked behaviors included reading posts, reading/viewing content (reading posted documents or e-mails, viewing links or videos), writing posts or e-mails, and taking quizzes.Social constructionists discuss many types of behaviors that occur during the learning process.Students observe (or read or listen to) new information, and they then need to process and interpret that information before they personalize/apply it and, in short, make it their own and make it fit with their view of the world (Ally, 2004).In the online environment, there are definite opportunities for observational learning (reading posts and content, watching lectures) and for application/interactional learning (posting in response to questions or other posts, taking quizzes, writing papers, etc.).
So, two distinct types of behaviors were accessible: observation (or taking in content), and application (or producing/demonstrating learning).The researcher assumed that in most cases students would need to observe (i.e., listen to the lecture so they could apply the ideas to their own lives, read the posts so they could consider and respond to the posted materials) before they applied that learning (e.g., took the quiz, responded to the discussion forum post, etc.).Thus, students would need to read an initial post or question before creating their own post, and they would need to read content and/or view a video lecture before taking the quiz.Therefore, online behaviors available from the course management system were divided into observation learning behaviors and application learning behaviors.Student engagement should be strongly correlated with both types of behaviors, but since applying learning behaviors requires the learning behaviors to occur first, the students who do more application behaviors should be more actively involved in the course and, thus, more engaged.While the distinctions are somewhat arbitrary since learning can and should occur during both types of behaviors, they are meaningful in creating levels of engagement within the course in terms of behaviors.
The researcher is cognizant of the fact that measuring student behaviors via course management software leaves out time spent thinking and reading, and that engagement goes well beyond behaviors produced with clicks online (indeed, the OSE measures a much broader definition of engagement, as presented above).This test is meant only as a method for providing more objective behavioral validation of the OSE than might be provided by only having students self-report their behaviors.It is not attempting to measure (or claiming to measure) all learning that occurs in an online course.
Thus, given the Community of Inquiry model and the previous work to establish the validity and reliability of this measure, the current study, to further validate the OSE, predicts the following: H 1 : OSE will significantly and positively correlate with observation learning behaviors in an online course.
H 2 : OSE will significantly and positively correlate with application learning behaviors in an online course.

Method Participants
To test these hypotheses, online communication instructors at a regional campus of a midwestern university were asked to forward an e-mail within Blackboard (the learning management system) to their students requesting their participation in a study about online learning.This occurred toward the end of the spring semester.Students were asked to complete the OSE and to give their permission for their instructor to release the tracking information about their online behavior.Thirty-four students (23 females and 11 males) from 13 sections of 5 upper level undergraduate communication courses completed the Qualtrics survey and gave permission for their tracking information to be used.This represented about a 10% response rate.The researcher was not instructing any of the participating sections.Five of the students were excluded from correlation analysis due to missing data on the OSE.

Procedures
Once students had completed the survey and the semester had ended, instructors submitted the tracking information for analysis.Tracking information included a semester's worth of observation learning behaviors (number of e-mails, discussions and assignments read, along with the number of web pages, content pages, and files viewed) and application learning behaviors (number of e-mails sent, discussions posted, assessments finished, and assignments submitted).As discussed earlier, while a student has to do something (e.g., log on, click a link) to accomplish observation learning behaviors, there is a difference between the student who attends class and one who attends and participates.Observational behaviors were differentiated from application learning behaviors.This data analysis examined only the quantity of behaviors, not the degree to which those behaviors affected student learning.Students could have simply clicked on and opened an e-mail without paying much attention to the content, and they could have taken a quiz by guessing at the answers.But for the purposes of seeing if the engagement scale is correlated with actual behaviors, the quantity of behaviors, while a crude measure, is still a valid estimate of student learning behaviors.Thus, the data analysis distinguished between observation behaviors of paying attention to content and application behaviors of actually doing something with the content and/or with classmates or the instructor.The observation learning behaviors and application learning behaviors were separately summed.
Students also completed the 19-item OSE (see Appendix A).To do so, they reported on a 5-point Likert scale how well each behavior, thought, or feeling was characteristic of them or their behavior.Statements included descriptions of thoughts/emotions such as "really desiring to learn the material"; skills like "being organized" and "taking good notes over readings, PowerPoints, or video lectures"; participation such as "helping fellow students" and "posting in the discussion forum regularly"; and performance items like "doing well on the tests/quizzes" and "getting a good grade."The OSE Cronbach's alpha for this sample was a .86,meaning the instrument demonstrated internal reliability.Results were obtained by running two separate Pearson's correlations: one between the OSE and the number of observation learning behaviors and the second between the OSE and the application learning behaviors.

Results
H 1 : The OSE will significantly and positively correlate with observation learning student behaviors in an online course.
This hypothesis was not supported.See Table 3 for correlation statistics.H 2 : The OSE will significantly and positively correlate with application learning student behaviors in an online course.
This hypothesis was supported.See Table 3 (next page) for correlation statistics.

Discussion
The finding that the application learning behaviors were significantly correlated with the OSE scale strongly supports the validity of the scale in measuring the engagement of students.While the scale goes beyond behaviors (measuring emotional and performance engagement as well as skills and participation), this study provided evidence that the self-reports of students are correlated with their observable (by a course management system) learning behaviors, validating the scale with objective data about behaviors.The validity of self-reports of engagement used by the OSE is supported by actual behaviors in the online class.Thus, the primary purpose of this study was accomplished.
While it was expected that both application behaviors and observation learning behaviors would be positively correlated with self-reported engagement, only application learning behaviors significantly correlated with self-reported engagement.It may be that simply reading posts, e-mails, content, and so on is not enough to be "engaged" in the course.So the number of observation learning activities is less relevant to engagement unless it is followed up by posting in the discussion forums, answering e-mails, and other application learning behaviors, a conclusion that fits with a social constructivist perspective.While notions such as observational learning would indicate that students can learn by reading, listening, and thinking, Vygotsky's ideas about the social nature of learning would indicate that only when we engage in communication with others about what we are learning do we have the opportunities to "test" our skills/knowledge, receive feedback, and engage more deeply with the content.The need for interaction to enhance learning fits with what we know about active learning in the classroom.A straight, traditional lecture with no room for student interaction will, in most cases, simply not be as engaging as some form of active learning (Ghani, 2009;Keyser, 2000;Margurber, 2005).This type of interaction with others, if within the zone of proximal development, can help students move that zone in the direction of becoming more knowledgeable/skilled within the content area.Therefore, social constructionist perspectives are supported with the finding that only application learning behaviors were significantly correlated with students' reports of engagement in online courses.
The sheer number of observation learning behaviors was, however, somewhat surprising.Thirtyfour students logged an average of just over 1,700 observation behaviors.This seems an extraordinary number of e-mails, posts, web pages, and other files to read, or at least click on, during a 16-week semester.The fact that the standard deviation (986.78) was almost one thousand indicates a lot of variability from class to class and student to student.It is possible, of course, that many of the behaviors reported by the course management software were just "clicks" and skimming of e-mails, posts, files, and web pages rather than students actually taking the time to read and consider each one.As previously stated, this data does not distinguish the quality of learning, simply the behavior of accessing the information.That fact may explain why the observation learning behaviors were not correlated with student engagement-they are not necessarily indicative of being engaged with the content or other

Limitations
Of course, there are limitations to this study.As always, a larger and more diverse sample would lead to stronger and more generalizable conclusions.Given the voluntary nature of the sample, it is certainly possible that those who volunteered were more engaged than others.However, since a correlation was used, the volunteer nature of the sample should have little effect on the statistical tests other than to restrict the variability of results.Accessing students' learning behaviors in more depth (are they taking notes or on Facebook while viewing the video lecture?) would provide a stronger sense of the behaviors involved.Future iterations of the scale should pilot more items asking students to consider their cognitive activities.At this point, students were asked about finding ways to make the course material relevant and applying the material to their lives.Other items measuring cognitive efforts (e.g., "I spend time thinking about the readings, content, etc.") might better tap into this important component.
In the previous study (Dixson, 2010) with 186 students from six campuses using multiple learning management systems, the mean engagement was 3.41.The current sample was quite a bit higher at 4.02, likely due to the smaller size and the use of communication courses that may require more student engagement in terms of discussion forums, chats, and so on.Of course, both means may be affected by the voluntary nature of the samples.More engaged students are more likely to volunteer for the study.Ideally, the next study would be used as part of a regular class and, thus, would require everyone to take the study, creating a sample that is not potentially biased by volunteerism.Even given this limitation, the data from both studies can begin to provide some benchmarks about student engagement in different types of classes that instructors can use to assess the engagement of their own students.

Potential Uses of the Scale
The OSE has three primary functions: (1) to aid research into online course design, (2) to provide feedback to instructors about the level of engagement of their students given the course design choices made, and (3) to provide evidence of teaching effectiveness for merit arguments, teaching awards, and promotion and/or tenure cases.
As a research tool, it has, so far, shown strong reliability and validity.The current study strengthens the argument regarding the validity of the OSE scale.If particular activities, designs, or teaching methods are said to increase student engagement, the OSE can be used to test those claims.
If an individual instructor or set of instructors wished to measure the level of engagement of their students, it could provide that type of feedback as well.The instrument should be especially useful in providing feedback before and after course design modifications to increase student engagement.
University teachers are increasingly being asked to provide multiple measures of teaching effectiveness beyond traditional student evaluations (multiple items with standard Likert scales asking how knowledgeable, organized, etc., the instructor was).The OSE is a valid and reliable scale that taps into something beyond student satisfaction while still using a student report method.Thus, the OSE may be used as an indirect measure of teaching effectiveness.It measures, in more depth than most traditional student evaluations, the perceived engagement of students in an online course.Engagement is a necessary (but not sufficient) step in student learning given that students must be engaged with the course before they can learn.So, while the OSE does not purport to measure learning, it does measure a necessary part of the learning environment that teachers work to create.Like student evaluations, the OSE should not be used in isolation.Since it measures perceived student engagement, it is vulnerable to all of the factors that affect the ability of student evaluations to be used as valid or reliable measures of teaching effectiveness: level of the course, type of content, preparedness of students, and so on.However, it provides a measure of the environment created by the design choices and responsiveness of the instructor and, thus, is an indirect measure of teaching effectiveness.
The OSE provides information beyond that available from course management software.Course management software offers information about the quantity of behaviors, such as number of e-mails read, posts read or written, quizzes taken, and so on.The OSE taps into students' intellectual efforts, skills, performance, and participation as well as the affective/emotional components of learning.
In conclusion, the OSE scale offers an easy, valid, and reliable way to measure students' engagement in online courses.This information is needed to continue to account for the efficacy of online courses, gauge changes in online courses, and move the scholarship of teaching and learning in the online learning environment forward.

Figure 1 .
Figure 1.Affective and behavioral components of engagement.

Table 3
Statistics for Hypotheses Tests They are indicative of accessing the potential to be engaged.Observation behaviors were, however, significantly correlated with application behaviors (r = .48;p < .001;n = 29).Thus, observation behaviors are likely a necessary but not a sufficient factor in engagement.The numbers provide a rough indication of the amount of work students are doing before enacting behaviors that demonstrate their learning.