Establishing a Student Evaluation of Online Teaching and Learning Framework Through Analysis of Existing Instruments

This study aims to establish a framework for student evaluation of online teaching and learning (SEOTL) through an analysis of SEOTL instruments implemented by universities and colleges. From a list of 131 R1 and 135 R2 institutions, we searched, reviewed, and identified 27 instruments for student evaluation of online teaching. A five-dimensional evaluation framework with 24 categories of elements was developed through an analysis of these instruments. There were 278 evaluation elements among the 27 instruments. We found that most instruments focus more on the Course and Instructor dimensions, with Instructor Facilitation and Learning Goals and Objectives elements occurring most frequently. However, Organization and Technology dimensions with Advising Availability and Adequacy, Registration Procedures, Support Services, and Online Help Desk elements were least included. This study has implications for administrators, instructors, instructional designers, and students.

The sudden shift in course delivery modality to a fully online learning environment in response to the COVID-19 pandemic has cemented online learning as one of the essential forms of education.As such, many more university faculty members are currently teaching online and will continue to do so.One necessary component of the online modality is evaluation of online teaching.Measuring the quality of online teaching is an essential step in continuous improvement of online teaching which enables better monitoring of efforts to increase student learning and engagement and develop faculty expertise.In many higher education institutions, faculty members are mandated to participate in course evaluations of their online teaching as part of their evaluation process.Though online teaching has been occurring for decades, universities often do not have a differentiated evaluation measure for online teaching and learning (Berk, 2013;Rothman et al., 2011).As such, faculty members, educational program directors, administrators, and online learning researchers are likely to utilize inadequate measures to assess the effectiveness of online courses and programs.Given recent growth in educational programs and research on interventions geared toward improving online teaching and learning outcomes, improved evaluation instrument is a pressing need.
Student evaluation of teaching is one measure used to inform both formative and summative decision making and assists educators in several important ways (McMahon et al., 2007).For example, better instruments are needed to support ongoing efforts to improve and assess online teaching quality.Student course evaluations play an essential role in the ongoing maintenance and improvement of courses for promoting student success, which factors into program and university measures of retention and progression.Additionally, they serve as an important evidentiary source for personnel management such as the reappointment of adjunct and clinical faculty as well as tenure and promotion decisions.However, if evaluation-based educational ratings data are used for high stakes decision making, then more research is needed to support the validity of such measures (Harris et al., 2014).Current research on student perceptions of online teaching effectiveness has relied on general questions which do not consider the complex, systemic nature of online courses (Lowenthal et al., 2015).
Given the accepted practice of using quantitative student evaluations of teaching in Western cultures (Darwin, 2017) and their increasing use for high-stakes evaluation of teaching effectiveness (Kogan, 2014), there is a need to advance the evaluation of teaching for the online modality to consider the full scope of factors contributing to teaching and learning.Due to the unique nature of online teaching (Stewart et al., 2004;Martin, Sun, et al., 2020), existing evaluation systems for online teaching must be expanded to measure and report on the relevant dimensions associated with online teaching effectiveness.In the present study, the research team aims to synthesize existing student evaluations of online teaching instruments and report existing practices by identifying evaluation dimensions and elements.Through this study we also aim to synthesize existing practices into a theoretical framework for online teaching and learning as a necessary first step in establishing a basis for future instrument development and use.

Literature Review Student Evaluation of Teaching
Student evaluation of teaching instruments measures perceptions of course and/or instruction.The first implementation of student evaluation of teaching in universities dates to the 1920s (Galbraith et al., 2012).Evaluation results can be used for both formative and summative purposes.Course instructors use the evaluation results to improve their teaching (Spooren et al., 2013).Institutions and universities also utilize evaluation results to implement administrative and personnel decision making such as hiring and promotion of faculty members (Spooren et al., 2013) based on the assumption that highly rated instructors produce positive learning outcomes among learners.
There is a general agreement that teaching is a multifaceted and complex practice which needs to be evaluated from multiple dimensions (Spooren et al., 2013).Chickering and Gamson (1987) noted that teaching practices that contribute to student success included student-faculty contact, cooperation among students, active learning, prompt feedback, time on task, high expectations, and respect for diverse talents and ways of learning.However, no consensus exists on the number and content of the dimensions.Spooren et al. (2013) reviewed research on student evaluation of teaching (SET) in the context of higher education and found the factor numbers in SET instruments ranging from two to twelve.For example, Students' Evaluations of Educational Quality (SEEQ; Marsh et al., 2009)  Student evaluation of teaching was considered a valid measure for teaching effectiveness and research has been conducted on the validity and reliability of student evaluation of teaching instruments (Onwuegbuzie et al., 2009).Cohen (1981) made an initial effort to examine the relationship between student ratings of instruction and student achievement by conducting a meta-analysis.Both overall course evaluations and overall instructor evaluations were highly and significantly correlated with student achievement in this meta-analysis.Here, overall course evaluations and overall instructor evaluations refer to the overall effectiveness concerning course dimension (e.g., This course is an excellent course), and teaching and instructor dimension (e.g., This instructor is an excellent instructor), respectively.However, this meta-analysis was not specific to online teaching.
A few researchers have examined student evaluations of online teaching based on student and instructor characteristics.Seok et al. (2010) found female students had statistically significantly higher perceptions of the effectiveness of online courses in six subscales (i.e., user interface, getting started, technical assistance, communications, online instructional design, and content).Researchers further noted that students' native language was a factor associated with online course evaluation.There were also statistically significant differences among students with varying educational levels in the rating of instructional design and content.In addition, Carle (2009) employed multilevel growth models to examine student evaluations of teaching effectiveness across time, instruction modes (i.e., online and face-to-face), and faculty characteristics (i.e., gender, ethnicity, tenure status).Data collected from 10,392 classes across three years revealed that although students tended to rate minority instructors significantly lower in face-to-face classes, no statistically significant differences in students' ratings were found between white instructors and minority instructors in online classes.Similarly, in another study, Weinkle et al. (2020), when studying 163 undergraduate students from six institutions and 21 graduate students from one institution, found no statistically significant differences in instructor evaluations across older male, older female, younger male, and younger female instructors.Feistauer and Richter (2018), examining the validity of student evaluations of teaching, found that likeability had a substantial bias on student evaluation of teaching and prior subject introduced a weak bias.These findings show that there are variations in student perceptions and each student might interpret the criteria differently when evaluating online teaching.

Instruments on Students' Perceptions of Online Teaching
While there is a large body of research on the student evaluation of teaching, only a few studies focus on the development and validation of instruments specifically designed for online teaching and learning.Table 1 provides an overview of the literature on the development and validation of student evaluation of online teaching.
The Student Evaluation of Online Teaching Effectiveness (SEOTE) developed by Bangert (2004) was among the initial endeavors to measure student perceptions of online teaching quality based on the framework of Seven Principles of Effective Teaching (Chickering and Gamson, 1987).This instrument has four factors (i.e., student-faculty interaction, active learning, time on task, cooperation among students) with 26 items.Bangert (2005;2006;2008) conducted a series of studies and provided evidence for content validity and internal consistency reliability (Cronbach's alpha ranging from .80 to .95).Specifically, a principal component factor analysis and a confirmatory factor analysis revealed a satisfactory global fit of the four-factor model to the data with various samples of undergraduate and graduate students, providing validity evidence based on internal structure.
In the same year, Stewart et al. (2004) constructed the Questionnaire for Student Evaluation of Web-Based Instruction following four steps: initial instrument development, data collection, validation, and final instrument development.This instrument has 44 items in seven elements.Multiple sources of validity were evidenced.Consultations with four content experts provided validity evidence based on content.Responses from 1,405 participants showed high internal consistency measured by Cronbach's alpha (ranging from .75 to .92).The sevendimensional construct (i.e., instructor and peer interaction, technical issues, appearance of Web pages, hyperlinks and navigation, content delivery, online applications, class procedures and expectations) also displayed empirical support for the internal structure-based validity based on factor analyses.
Studies were also conducted to develop and validate measures for distance teaching.Cheung (1998) identified four factors (i.e., student development, assessment, learning materials, face-to-face components) after a review of existing literature and instruments on distance teaching evaluations.This instrument demonstrated a set of good psychometric properties.Specifically, the instrument was found to have high reliability with respect to interrater reliability (ranging from .759 to .893) and internal consistency reliability (ranging from .824 to .948).A hierarchical confirmatory factor analysis resulted in a satisfactory fit of the four-factor model to the data (e.g., RMSEA = .053,GFI = .90,& CFI = .92).Similarly, Roberts et al. (2005) developed an instrument to evaluate distance education courses based on the methods proposed by Biner (1993), which consists of four procedures: item generation, dimension identification, essential item selection, and instrument writing and presenting.With this measure, students assessed the instructor, overall course effectiveness, and specific technical dimensions of distance education on a five-point Likert scale.
Through a thorough review of literature on best practices in online learning, Rothman et al. (2011) developed a survey measuring students' perceptions of online courses.This instrument consists of six factors (i.e., appropriateness of readings and assignments, technological tools, instructor feedback and communication, course organization, clarity of outcomes and requirements, content format) with 25 items.Satisfactory evidence for reliability (Cronbach's alpha = .98)was reported.
Most recently, Blackman et al. (2019) developed the Online Teaching Effectiveness Scale (OTES) based on a review of literature on measures of online teaching effectiveness.OTES measures student perceptions of online teaching effectiveness in four aspects (i.e., presence, expertise, engagement, facilitation).Multiple sources of validity and reliability evidence of OTES was provided by Reyes-Fournier et al. (2020).Satisfactory internal consistency reliability (ranging from .68 to .95) and test-retest reliability (ranging from .74 to .89) were found with a sample of undergraduate and graduate students.Confirmatory Factor Analysis resulted in a satisfactory fit of the four-factor model to the data (RMSEA = .143,CFI = .912).Validity based on relations to other variables was evidenced by the significant and positive relationship between expertise and course grade (r = .1,p = .05).However, course grade did not significantly correlate with the other three dimensions.
Although a few instruments have been developed to be utilized for SEOTL, the review of the relevant literature suggests that most of the existing instruments on student evaluations of online learning are over ten years old.Besides, the most recent instrument (Reyes-Fournier et al., 2020) focuses only on the dimensions of instructor, course, and student without paying attention to Organization or Technology dimensions, which are important factors pertaining to the student experience in the online learning environment.Thomas and Graham (2017) reviewed literature on online instructor evaluation and found instruments of student evaluation of online instructors focused on two dimensions (i.e., course and instructor) and eight categories of elements (i.e., learner-instructor interaction, instructor expertise, student-student interaction, assignments are meaningful, clear expectations and instructions, technical concerns, visual design and function of the course, effective use of technological tools).

Comparing Evaluations of Online with Face-to-Face Courses
Several studies compared student evaluations of online courses with those of face-to-face courses or blended courses, and mixed findings were noted.First, instructors received different ratings across modalities of course delivery.Lowenthal et al. (2015) analyzed student evaluations of face-to-face and online courses at a university over seven years, and found online instructors were rated statistically significantly lower in each item of the End-of-Course Evaluation Questions (i.e., Course Overall, Instructor Overall, Grading Fairness, Instructor Access, Workload, and Course as Learning Experiences) compared with their ratings of face-to-face courses.This finding, however, contradicted other studies (e.g., Carle, 2009;Liu, 2006).Carle (2009) conducted multilevel analyses with 10,392 classes at a university over three years and found no statistically significant differences in student ratings of teaching effectiveness between the two modes of instruction.Moreover, online course evaluation had lower completion rates compared with face-to-face courses.Online teaching and learning are distinct from face-to-face teaching and learning in other aspects.Martin, and Sun, et al. (2020) conducted a systematic review of literature on online teaching and learning and highlighted the significant role of infrastructure to promote engagement and success, including organizational resources and technology for each course-specific participant.
Existing student evaluations of online teaching instruments are either more than 10 years old or, more recently, focus only on one dimension (e.g., instructor).With an increase of online courses in higher education, there is a need to conduct a review of the instruments for student assessment of online teaching and learning, based on which a multidimensional online course evaluation framework can be constructed.The current study aims to establish a student evaluation of online teaching and learning framework from analyzing existing online course evaluation instruments implemented by universities and colleges.The research questions guiding the current study are as follows: 1. What evaluation dimensions are included in student evaluation of online teaching and learning instruments used by universities?
2. What were the evaluation elements in the university student evaluation of online teaching and learning instruments based on the learner, course, instructor, and technology, and organization dimensions?
3. How is the distribution of the evaluation elements across the dimensions used in student evaluation of online teaching and learning?

Methods
This study used a systematic review process and included a) identifying instruments, 2) screening instruments, and 3) coding and analyzing instruments.

Identifying Instruments Identifying University Lists
A list of Doctoral Universities: Very High Research Activity (R1 universities) and Doctoral University: High Research Activity (R2 universities) were obtained based on the Carnegie Classification of Institutions of Higher Education (2021).A total of 131 R1 universities and 135 R2 universities were included in this list.We initially planned to target a random sample of 26 R1 institutions and 27 R2 institutions.However, this random sample only yielded six evaluations of online teaching in total because not all institutions considered had a specific form of online course evaluation, so we decided to use all the 266 R1 and R2 institutions to identify available instruments.

Search Terms Used
We used the search terms "Online Course Student Evaluation" or "Student Evaluation of Online Teaching" and the university name in the Google search engine to search for publicly linked course evaluation instruments used by a university.We also entered the two search terms in the institutions' websites to identify potential instruments.This procedure was implemented by two researchers, with one researcher searching for instruments in R1 universities and the other searching in R 2 universities.The search endeavor resulted in 17 instruments in R1 universities and 14 instruments in R2 universities, which were publicly available instruments for student assessment of online teaching.

Sending Emails
In a related attempt to locate instruments, we directly contacted directors of the Center for Teaching and Learning or equivalent department at universities.We recorded names and email addresses of directors of the Center for Teaching and Learning for each university from their website.A total of 76 emails were sent and 7 responses were received.From this correspondence, one additional instrument was obtained.Six of the other directors who responded mentioned that they did not have a differentiated instrument for online course evaluation.

Screening Instruments
The inclusion and exclusion criteria used for screening identified instruments are presented in Table 2. First, the instruments were included if they were used to evaluate online/distance courses or includes items evaluating online/distance courses, so instruments for the evaluation of face-to-face courses were excluded.Second, we only examined student evaluation of online teaching, so peer evaluation or self-evaluation instruments were excluded.Two researchers implemented the screening of the 32 instruments independently by applying the two inclusion criteria specified.Five instruments in R2 universities were excluded because three were guidelines or standards for the design of online/distance courses and two were peer evaluations of online teaching.The final sample, consisting of 18 instruments from R1 universities and nine instruments from R2 universities resulting in a total of 27 instruments which was submitted for coding and analysis.

Inclusion Criteria Exclusion Criteria
Evaluation of online/distance courses or have items evaluating online/distance courses Evaluation of face-to-face courses

Student evaluation
Peer evaluation or self-evaluation

Coding and Analyzing Instruments
Instrument coding occurred in two intentional phases.In the first phase, two researchers reviewed the 27 instruments in their entirety.Open coding was adopted to locate and identify codes for each individual item.After code identification and exploration, coded information was reread to identify underlying connections between codes and the codes were categorized into elements and elements were placed into dimensions.The following five dimensions (learner, course, instructor, technology, and organization) were identified and adopted as an overall scheme for sifting and organizing elements emerging in the process.A total of 24 categories of elements emerged from the process.
In the second phase, we coded the 27 instruments based on the framework of five dimensions and 24 categories of elements.The second phase of coding was implemented by recording whether an instrument had a specific element or not.To ensure the quality of coding, a subset of six instruments (22%) was randomly sampled and independently coded by two researchers.We obtained an interrater reliability of 94% for percent of agreement and 89% for Cohen's Kappa, indicating a satisfactory coding consistency.Differences and disagreements were resolved through group-level discussions with the entire research team that included two additional researchers.We encountered a few challenges in the process of sorting some of the elements into one of the dimensions.For example, there was a subtle distinction between the elements of Course Activities and Instructor Facilitation when referring to items related to engagement or discussion.We decided to code an item as Course Activities when the item focused on the course and as Instructor Facilitation when it emphasized instructor.Similarly, items related to course assignment could fall into both the categories of Course Activities and Course Assessment.
Once coding was completed, a series of descriptive analyses were conducted.The number and percentage of instruments for each element were recorded.A distribution was plotted for the 24 categories of elements across the 27 institutions.The variations of the number of elements for each instrument were displayed.Further, a distribution was examined in terms of the number of elements in each dimension for each of the 27 instruments.

Methodological Limitations
This study examining online course evaluations has a few limitations.The research team examined mostly evaluation instruments available publicly online.Though attempts were made to reach directors of Centers for Teaching and Learning to request a copy of instruments not available, in many cases responses were not received.Also, we included only R1 and R2 universities to manage the scope of the project.Further work should explore instruments from other types of institutions such as teaching universities and community colleges.During the coding of the evaluation elements and dimensions, only two researchers were involved.Though interrater reliability was calculated and there were periodic discussions among the researchers, there could be a bias on how these items were coded.Also, while a framework was proposed for the dimensions these were not validated as part of this study.Researchers may explore the factor structure of the construct and evaluate internal validity in future studies.

Overview of the Instruments
A list of instruments identified is presented in Appendix A. Among these 27 institutions, only six have a complete evaluation form while the other 21 institutions have only supplemental items on online teaching added to their original evaluation form for face-to-face courses.The number of items in these instruments ranges from two (supplemental items) to 56 (complete forms).Most items in those instruments are Likert scale questions.

Dimensions and Elements of the Instruments
A Student Evaluation of Online Teaching and Learning (SEOTL) framework was constructed through an analysis of the 27 instruments.There are five dimensions in this framework: learner, instructor, course, organization, and technology.The fifth dimension, technology, could be part of any of the other four dimensions.These dimensions collectively capture all relevant aspects of online teaching and learning in higher education.Figure 1 depicts the framework of student evaluation of online teaching.The elements of each dimension are detailed in Table 3.

Effort to Learn
Effort to Learn has items on measuring the amount of effort that learners devoted to the online course, including the time they spent in and out of classes, and the degree of attendance, participation, and interaction.

Intellectual Challenge
Intellectual Challenge measures the extent to which learners are intellectually challenged or stimulated.It includes if the course helps learners gain knowledge and skills, understand subject matter, and practice abilities in critical thinking and problem-solving.

Interest to Learn
Interest to Learn includes items on measuring the extent to which learners' interest, motivation or enthusiasm was stimulated by the course or the instructor.

Student Readiness
Student Readiness measures learners' preparedness for online learning, such as having prerequisite knowledge, technological skills, digital information literacy skills, or adequacy of living and study setting.

Anticipated grade
Anticipated grade measures learners' expectations of their course grade.

Course Format and Organization
Course Format and Organization measures the design, structure, and presentation of the online course, including instructional balance, instructional alignment, course content planning, appropriateness of instructional pace, and appropriateness of amount of work.

Goals and Objectives
Learning Goals and Objectives measures if the course learning goals and objectives are clearly specified and well accomplished.

Course Materials
Course Materials measures the quality, quantity/workload, relevancy, variety, and accessibility of course materials for an online course.This element also has items measuring the extent to which course materials are aligned with learning goals and objectives and contribute to student learning.

Course Activities
Course Activities measures the quality, frequency, depth, variety, and appropriateness of class activities such as peer work, collaborations, hands-on activities, and discussions in an online course.This element also measures if course activities create opportunities for interactions with content, classmates, and the instructor, and facilitate student learning.

Course Assessment
Course Assessment measures the quality, variety, and appropriateness of assessment of student performance in an online course.Ideally, grading criteria and instruments are explicitly specified; assessment method is fair, accurate and appropriate; and assessment is aligned with learning goals and objectives and contributes to student learning.

Instructor Subject Matter Expertise
Instructor Subject Matter Expertise measures the extent to which the instructor demonstrates the mastery of subject matter expertise in an online course.The Instructor needs to have a good command of knowledge in course content and clearly explain the course subject matter.

Instructor Facilitation
Instructor Facilitation measures the quality and process of the delivery of an online course.Ideally, an online course instructor gives a clear explanation of course content, employs effective teaching methods or strategies, provides prompt and meaningful feedback, and manages classes, discussion, interactions, and communication effectively.

Instructor Readiness
Instructor Readiness measures the instructor's preparedness for online teaching such as if the instructor has skills in the use of technology.

Creation of Inclusive Learning Environment
Creation of Inclusive Learning Environment measures the extent to which the instructor encourages diverse perspectives, creates a positive, inviting, and inclusive learning environment, treats students with respect, and demonstrates cultural awareness.

Enthusiasm and Rapport
Instructor Enthusiasm and Rapport measures the extent to which the instructor demonstrates interest in or enthusiasm about teaching.

Instructor Instructor Use of Technology
Instructor Use of Technology measures instructors' ability in the effective use of technology.

Functionality of Synchronous/ LMS
Functionality of Synchronous/LMS measures the effectiveness of navigating the learning management system and the extent to which features and elements of online learning environments support learning.

Organization Advising Availability and Adequacy
Advising Availability and Adequacy measures the Availability and Adequacy of advising.

Registration Procedure
Registration Procedure measures the smoothness of registration process of online courses Support Services Support Services measures availability and adequacy of services and resources (e.g., financial aid, registration, counseling, career centers) provided by various centers and institutes (e.g., Office of Financial Aid, University Bookstore, Office of Distance Education) that support students' learning.

Online Help Desk
Online HelpDesk measures the availability and adequacy of Help Desk.

Descriptive Analysis of the Instruments
The distribution of the 24 elements is depicted in Figure 2. Table 4 displays the full coding information of the 24 categories of elements in the five dimensions across the 27 instruments.The total number of elements included in the 27 instruments ranges from 4 to 17 with an average of 10.30.The frequency of each element was also calculated.Instructor Facilitation (n = 26, 96.30%) occurs the most frequently, followed by Learning Goals and Objectives (n = 23, 85.19%), Intellectual Challenge (n = 22, 81.48%), Course Material (n = 21, 77.78%), and Course Assessment (n = 21, 77.78%).The elements that have the least frequency include Advising Availability and Adequacy (n = 1, 3.70%), Registration Procedures (n = 2, 7.41%), Support Services (n = 2, 7.41%), Online Help Desk (n = 2, 7.41%), Anticipated Grade (n = 4, 14.81%), and Technical Support Availability (n = 4, 14.81%).Overall, most instruments contain elements in the Course (ranging from n = 19 to n = 23) dimension and very few instruments have elements in the Organization dimension.

Discussion
This study contributes to the research and practice through the development of the SEOTL framework which can be used by both researchers and practitioners.This multidimensional framework, including learner, instructor, course, technology, and organization, provides a holistic and comprehensive model for evaluation based on all necessary aspects.In the literature, three measures exist that used prior conceptual and theoretical frameworks (Biner, 1993;Chickering and Gamson, 1987;Driscoll, 1998;Khan, 1997).However, these works are outdated, and the frameworks used by these instruments fail to capture all relevant aspects of online teaching and learning.Therefore, there is a need for a newer multidimensional framework evaluation of online teaching which this SEOTL framework aims to meet.The consequences of not having an appropriate instrument specifically designed for online courses are evident: 1) failure to provide constructive feedback for teaching, 2) a faculty promotion decision based on invalid teaching measures, and 3) failure to inform instructors with the important aspects to be prepared for quality online courses.The online course evaluation instruments used by universities are usually not validated despite being used for several semesters and often for making high-stakes decisions.Using a research-based and validated instrument based on the SEOTL framework will help universities precisely evaluate their online teaching and learning practices and diagnose weaknesses and deficits in education.While a few research-based online course evaluation instruments currently exist, this study shows the need to develop an instrument that is multidimensional in evaluating online teaching and learning.Also, currently the instruments in the research are from several years ago except for the Reyes-Fournier et al. (2020) instrument which focuses only on the instructor dimension.
Across all the five dimensions, there were a total of 278 evaluation elements.The most frequently considered was the Course dimension followed by the instructor dimension.There were 103 elements (37.05%) in the Course dimension, and there were 88 elements (31.65%) in the Instructor dimension.This shows that when evaluating online teaching and learning, universities focus on the course and the instructor the most.In the Course dimension, learning goals and objectives were included in 23 instruments, and course materials and course assessment were both included in 21 instruments.In the Instructor dimension, instructor facilitation was included in 26 instruments.Though the overall Learner dimension was least evaluated, one element (intellectual challenge) was included in 22 instruments.This finding is consistent with the research-based online course evaluation instruments (i.e., Bangert, 2004;Cheung, 1998;Reyes-Fournier, et al., 2020;Roberts et al., 2005;Stewart et al., 2004).Researchbased instruments have more elements in the dimensions of Course and Instructor.Instructor facilitation was the most frequently occurring element, included in all research-based instruments.Further, learning goals and objectives, course assessment, course activities, and instructor enthusiasm and rapport were included in most research-based instruments.Previous literature suggested that instructor facilitation was critical to students' learning in online courses.Martin, Wang, et al. (2020) noted that the instructors' timely responses to questions and timely feedback on assignments/projects were conducive to instructor presence, engagement, and learning.Learning goals and objectives was also found to be a significant component of online courses (Ndoye & Martin, 2021;Raible et al., 2016).Goals and objectives help to set learner expectations and to align instructional material and assessment.Course assessment, which is to measure student learning outcomes and overall course effectiveness, was included as an integral part of online courses (Martin et al., 2021).
The three least evaluated dimensions were the Organization dimension, followed by Technology and Learner.Organization dimension included only 7 elements (2.52%), Technology had 28 elements (10.07%), and Learner had 52 elements (18.71%).This suggests that when evaluating online learning, universities often care less about the organizational support, the technology, or the learner.All organizational elements were least used in the evaluation instruments.Advising availability and adequacy was included only in one instrument.Registration procedures, support services, and online help desk were each included in two instruments.In the Technology dimension, technical support availability was included in four instruments, and in the Learner dimension, anticipated grade was included in four instruments.This finding is also aligned with research-based online course evaluation instruments.A few instruments focused on the Technology dimension (Roberts et al., 2005;Stewart et al., 2004) and only one instrument reviewed in the literature paid sufficient attention to the Organizational dimension (i.e., Roberts et al., 2005).
It has been consistently documented in the literature that the (unplanned) shift from faceto-face instruction to online teaching, due to the recent COVID-19 pandemic, has caused an increased burden on instructors and educators (Nasri et al., 2020) to design courses, facilitate learning, and provide appropriate instruction (Rapanta, 2020).Organizational support plays an important role to offer quality education in the virtual environments.However, existing course evaluation forms failed to account sufficiently for some important aspects of online teaching and learning, placing too much emphasis on the instructor's responsibility while devaluing support and services that can be offered by organizations.Evaluating teaching practices is often used not only to provide summative assessments of instructors for promotion, tenure, and salary decisions but also to diagnose deficits in a support system for assisting faculty and students.Thus, including all relevant aspects in online course evaluations is crucial.
The thorough review of the literature suggested the unique nature of online learning environments (Kreitzer & Sweet-Cushman, 2021;Martin, Sun, et al., 2020).Unlike face-to-face instruction, additional factors such as technology and organization support come into play in virtual settings, collectively determining the quality of online education.Thus, each dimension of the SEOTL framework must receive appropriate evaluation.We recommend that online course evaluation forms contain enough items assessing each dimension.Also, though it is a common practice to derive a composite overall score by simply summing all ratings on an instrument to represent overall teaching quality, we suggest giving an equal weight to ratings on each dimension.This assists instructors and organizations to correctly identify where and what to improve.

Implications for Practice and Research
This study has implications for administrators, instructors, instructional designers, and students.
Administrators can benefit from reviewing the currently used instruments, comparing their university evaluation instrument with the findings of this study and add/remove items as needed.The findings of this study will benefit administrators at all higher education institutions though only instruments at R1 and R2 universities were examined.It is important for student evaluation of online teaching instruments to include all five categories of items: students, course, instructor, technology, and organization.Administrators also play a role in the elements related to technology and organization and making sure organizational support is available for the students and the instructors for online teaching and learning.
Instructional designers play an important role in supporting instructors in designing online courses.Instructional designers, when designing online courses or supporting online instructors, can use the findings from this study to include the various evaluation elements in the design.In addition to design, instructional designers can also recommend various additional strategies that the instructors can use during the facilitation of the online course.
The findings have direct implications for online instructors as evaluation ratings provide them with feedback to strengthen the courses they teach.Online instructors can examine and implement the various dimensions and evaluation elements that are commonly used and design and deliver their courses.The process of examination and implementation of these evaluation elements earlier in their courses will strengthen their courses.In addition, they can use feedback from initial implementations to add missing elements to support the students.
In the end, students will benefit the most from well-designed and effective online courses.They can specifically also think about the five evaluation elements that were learner focused, effort to learn, intellectual challenge, interest to learn, student readiness and anticipated grade.These are also helpful in a successful online learning experience.

Declarations
The author(s) declare no conflicts of interest or external funding.

Figure 2
Figure 2 Distribution of the Elements Across the Dimensions has nine factors (i.e., Learning/Value, Instructor Enthusiasm, Organization/Clarity, Group Interaction, Individual Rapport, Breadth, Exam/Graded Materials, Readings/Assignments, and Workload/Difficulty) and two overall ratings (i.e., overall ratings of the course and the teacher), whereas Student Instructional Report (SIR II; Centra, 1993) has six factors (i.e., Course and Planning; Communication; Faculty/Student Interaction; Assignments, Exams, and Grading; Course Outcomes; Student Effort and Involvement) and one overall evaluation item.