Evaluation Instruments and Good Practices in Online Education Evaluation Instruments and Good Practices in Online Education

Chickering and Gamson’s (1987) Seven Principles for Good Practice in Undergraduate Education offers extensively researched and validated tenets for best practices in higher education. After a review of the literature, twenty-eight evaluation instruments currently used to design and review online courses in higher education institutions were collected and divided into categories, based on geographical reach and the type of institution for which they were developed. This study investigates how evaluation instruments used in higher education assess the Seven Principles for Good Practice in Undergraduate Education, and what other items are addressed in the evaluation of courses. Findings show that national and statewide evaluation instruments were less institute specific and more closely aligned to the principles of good practice, and that evaluation instruments often measure extraneous items (e.g., student services, navigation, resources, or institutional support). Additional findings and conclusions based on the analysis of the instruments are discussed.


Introduction
created the Seven Principles for Good Practice in Undergraduate Education as guidelines for effective teaching and learning.Chickering and Gamson's (1987) principles state that good practice: 1. Encourages student-faculty contact.2. Encourages cooperation among students.3. Encourages active learning.4. Gives prompt feedback.5. Emphasizes time on task.6. Communicates high expectations.7. Respects diverse talents and ways of learning.
These guidelines represent a philosophy of quality education that has been widely used and accepted for both face-to-face courses and online learning.The Seven Principles have been extensively researched and validated (Institute for Higher Education Policy, 2000;Meyer, 2000;Kuh, 2002;Shea, Pickett, & Pelz, 2003;Sherry, 2003 (as cited by Dayton & Vaughn, 2007).The National Institute for Learning Outcomes and each major regional accrediting association support the use of the Seven Principles (Meyer, 2002).Pascarella and Terenzini (1991) have synthesized thousands of research studies informing us about how students learn.Of all these studies, the Seven Principles for Good Practice in Undergraduate Education appear to be "the best known, certainly the most widely distributed" (Cross, 1999, p. 256) framework available about student learning.
The Seven Principles have been successfully used to guide and develop courses in online education.Newlin and Wang (2002) used the Seven Principles to discuss the importance of developing online courses guided by sound pedagogical practice in order to promote quality.In particular, the emphasis on contact between students and instructors was noted as a way to build an online community and prevent isolation of participants.Chickering and Gamson (2002) provided examples of the Seven Principles in practice for the Florida Engineering Education Delivery System to reflect best practices in education.Using the Seven Principles as a guide, Sowan and Jenkins (2013) designed and delivered a hybrid nursing course for 109 undergraduates.The students' satisfaction with the course was measured and compared with a cohort taking the same course face-to-face.The students were very satisfied and had significantly higher scores than the students in the control group.This suggests the effectiveness of applying the Seven Principles to online course design.In an article written for the Institute for Learning Outcomes Assessment, academic administrators at the University of Maryland University College also suggested that the Seven Principles can be used to guide course design (Prineas & Cini, 2011).The researchers indicated that there is a benefit to using the learner-centered tenets, based on their experience of operating programs serving 67,000 students worldwide.The Seven Principles have been distilled by some researchers into practical advice; Hathaway (2013) analyzed the usefulness of each of the Seven Principles and then provided tips for implementing these principles into online courses.Creasman (2012) offered useful advice to faculty designing online courses to show the benefit of students interacting with faculty: collaborative learning, active learning, instructor social presence, balancing the amount of course information with student commitment and persistence, and providing different learning experiences to engage more students.Chickering and Gamson's (1987) principles have been used in the past to help evaluate online courses.Bangert (2008) developed an institute-specific student evaluation of online teaching instrument based on the Seven Principles.The purpose of this instrument was to provide instructors with feedback about their online teaching practices.In two studies and with 1,037 students, Bangert measured the validity of the instrument and confirmed the importance of student-faculty interaction, active learning, time on task, and cooperation among students.Crews, Wilkinson, and Neill (2015) also formulated an instrument based on the Seven Principles; with 179 students in an undergraduate Computer Applications in Business course at a large southeastern university, they found that what the students valued aligned with the Seven Principles.Drayton and Vaughn (2007) used the Seven Principles as a foundation for a quality assurance checklist they developed to guide the design and assessment of online courses offered by Southern Polytechnic State University (SPSU).The researchers chose the Seven Principles after an extensive literature search and information review on the use of these principles.Billings (2000) adapted the Seven Principles to develop a framework for assessing outcomes and practices in online nursing courses.Çağiltay, Graham, Lim, Craner & Duffy (2001) used the Seven Principles to evaluate four online courses at Indiana University.To do this, they reviewed course materials and interviewed faculty.Achtemeier, Morris, & Finnegan (2003) also used the Seven Principles as a basis to assess the evaluation instruments used by higher education institutions in Georgia.
There is a benefit to using an accepted framework like Chickering and Gamson's principles to evaluate education.Standards serve to guide instructional design and delivery by providing a clear understanding of what is expected to attain success.This is a useful practice for course designers and instructors endeavoring to design effective online courses.Malone et al. (1997) indicated the need for well-researched criteria to guide instructional designers and instructors in instructional design models and methods.Gaytan and McEwen (2007) also indicated that using "effective assessment techniques is an essential part of effective teaching and learning" (p.118) in online courses.Online instruction offers a unique opportunity to strategically direct learners through a framework that reliably leads to the desired result, assuming the course has been created in a manner that identifies and encourages best practices.Given the increasing prevalence of evaluation instruments in distance education, the purpose of this review is to compare Chickering and Gamson's Seven Principles for Good Practice in Undergraduate Education with evaluation instruments currently in use.

Method
A paper by Achtemeier et al. (2003) guided this research.Achtemeier et al. investigated the definition and principles of effective teaching and learning, specifically in online education.They performed a content analysis of instruments used by the University System of Georgia institutions.Achtemeier et al. identified the Seven Principles for Good Practice in Undergraduate Education as a measurement of good practice, and explored the degree to which evaluation instruments try to assess the Seven Principles for Good Practice in Undergraduate Education.We chose to use the Seven Principles because they have been well researched by others and have stood the test of time since published in 1987.Another benefit of using the Seven Principles is that they represent best practices for higher education, for both traditional and online courses.Thus, the research questions that guide this literature are: • To what degree do evaluation instruments used in higher education try to assess whether the Seven Principles for Good Practice in Undergraduate Education are taking place?• What other items are addressed in the instruments to evaluate courses?
We performed a literature search in the EdITLib: Education & Information Technology Digital Library, Education Research Complete (EBSCO), ERIC (Education Resources Information Center), and Web of Science and Google Scholar databases utilizing the keywords "Chickering and Gamson," "quality," "assessment," "evaluation," "online," and "distance education" to provide information for this research.The purpose of this search was to find literature about the use of assessment/evaluation devices in online education, and to determine what has been written about quality practices in online education.It is important to distinguish between the terms "evaluation" and "assessment."According to Duke University (n.d.), assessment is formative and interested in the process of learning to identify areas of improvement.In contrast, evaluation is used in a summative manner to gauge quality, with an emphasis on the product and an intention to arrive at a final grade or score.Not all of the literature we reviewed followed this distinction so there is some crossover in the use of the two terms.
In a separate search, we collected 33 higher education online course evaluation instruments.We used search terms "course design rubric," "course design checklist," "instructional design rubric," instructional design checklist," "course design standards," "instructional design rubric," and "higher education online course design" on the Internet to look for instruments.Our search was informed by the reference of instruments among evaluation tools we found.For instance, the California State University (CSU) Quality Online Learning and Teaching Rubric (QOLT) listed several rubrics that helped to shape its development (e.g.Quality Matters and Quality Online Course Initiative).Five evaluation instruments were eliminated because they focused only on design and failed to address learning practices.In addition, these instruments, primarily produced by individual bloggers, failed to demonstrate the rigor necessary to be utilized as a complete course evaluation tool.California State University's Chico's Rubric for Online Instruction was also eliminated because the California State University (CSU) Quality Online Learning and Teaching Rubric (QOLT) was developed more recently, within the same institutional system.While our sample (N=28) may seem small, the reach of these instruments is broad (e.g., Quality Matters has more than 900 subscribing institutions, according to the Quality Matters website).
To be included in our study, evaluation instruments had to meet the following criteria: (1) evaluate higher education online course design; (2) support student success; (3) publication after 2000.We divided the evaluation instruments into different categories based on their reach and use: geographical and/or institutional, and level (four-year or two-year).We established a separate category for rubrics from online authors and organizations (Table 1).These categories were determined in part by the example set by Achtemeier et al. (2003), and in part by personal logic.Evaluation instruments created by a national learning management systems company might have a different purpose than an instrument created specifically for a four-year or two-year school.A different level of rigor might be expected based on whether the instrument is used at a national or statewide level, by a four-year institution, a two-year institution, or as an online resource.Our goal was to identify patterns by geographical scope and institutional level.The instruments were analyzed in terms of content, development process, scope, and application.A coding scheme was developed to determine whether an evaluation instrument met the criteria to be evaluated.This scheme was based on information from our literature review of examples of the Seven Principles of Good Practice in online education.Standards and criteria within each evaluation instrument were analyzed and coded to identify principles of good practice that had been addressed using the scheme presented in Table 2.We looked for items that utilized the same or similar terms as the Seven Principles or for examples based on the Seven Principles' intent.For instance, while many evaluation instruments mentioned feedback to students, only evaluation instruments that actually mentioned "prompt feedback" or gave a time requirement (e.g., "48 hours") were coded as assessing prompt feedback.When in doubt, we referred to the examples provided by Chickering & Ehrmann (1996) for implementing the principles using technology.We also referred to Watwood, Nugent, & Deihl (2009), who provided vignettes and supporting material to illustrate the Seven Principles in online teaching.

Findings
The number of instruments addressing each principle has been summarized in Table 3.In the review of the evaluation instruments, student-faculty contact and cooperation among students were the two principles most frequently assessed.Active learning was also frequently evaluated.Much less frequently measured (in order), were diverse talents and ways of learning, prompt feedback, and high expectations.Only one evaluation instrument included student's time on task.National or statewide evaluation instruments were found to assess a greater percentage of the Seven Principles than the other categories.For example, the Quality Matters (QM) rubric is part of a subscription-based program that involves professional training.Annually, a committee reviews independent scholarly research related to online course design as part of the information that helps inform the criteria of the QM rubric (Shattuck, 2013).The QM organization boasts "825 subscribing educational institutions and 160 individual subscribers" (Shattuck, Zimmerman, & Adair, 2015, p. 26) and is considered the most pervasive tool used to evaluate higher education course quality.
While none of these instruments addressed all of the principles, two of the instruments (Open SUNY COTE Quality Review and Southern Regional Educational Board Checklist for Evaluating Online Courses) assessed substantially more of the principles than other instruments (Table 4).Other issues that were frequently addressed in this group of instruments include ensuring that navigation is intuitive, incorporating intended outcomes for students, utilizing assessment, and including support for students.The number of online professional development evaluation instruments that assessed the Seven Principles of Good Practice was below the mean (Table 7).While each of the instruments included student-faculty contact, none of the instruments included prompt feedback or time on task.The online professional development evaluation instruments focused on items like course navigation, scalability of assignments, use of media in the course, ongoing assessments, and clear instructions.The Online Learning Consortium was recently awarded $2.5 million by the Bill and Melinda Gates Foundation to foster best practices in post-secondary learning.In particular, the grant will support expansion of the OLC Quality Scorecard and fund a new competition to "recognize and reward exemplar institutions and faculty for effective use of digital courseware" (Online Learning Consortium, 2015).Special emphasis will be added to the scorecard to include steps that focus on attributes that help "minority, first-generation, lowincome or other disadvantaged backgrounds" (Online Learning Consortium, 2015).This example draws attention to how funding from grants and member institution goals can have a large impact on evaluation instruments.

Conclusions
Creating a course involves many elements.But Chickering and Gamson have emphasized seven principles of good practice which have been widely cited and used over the past 25 years.Our findings indicate that higher education assessment tools are not adequately evaluating all of these well-established principles of effective practice.Most of the evaluation instruments we reviewed assess for student-faculty contact (85%), cooperation among students (75%), and active learning (57%).The more broadly used evaluation instruments (i.e., national or statewide compared to instruments from two-year colleges) were more likely to assess for a greater number of the Seven Principles.There is a system-wide deficiency, however.The majority of instruments we reviewed did not assess for prompt feedback (29%) or for time on task (4%), a practice of ensuring that students understand time requirements and deadlines.Most of the evaluation instruments also fail to assess for communicating high expectations (14%).High expectations can be communicated by assigning challenging work, providing examples, and publically praising exemplary results (Watwood et al., 2009).These practices help ensure quality in online learning by clarifying and illustrating expectations to learners.
While the evaluation instruments often failed to cover some of the seven basic principles, this was not due to their brevity.In fact, they were quite long; the average instrument we reviewed featured over seven standards and 53 specific criteria.We found that many of the evaluation instruments measured the use of learning assessments, clear course organization and presentation, and the presence of course goals and objectives.Clear navigation was also noted frequently, as was availability of resources for student support.These items may be important but they are not part of the crucial seven principles linked to good practice.Adding the Seven Principles to these instruments may encourage best practice but may also create instruments that are increasingly onerous to use.Conversely, adding additional items to the Seven Principles may dilute the focus and distract the practitioner's emphasis from providing a quality education based on best practices.
A limitation of the study was that we did not collect evaluation instruments from sources other than the Internet, and we examined only evaluation instruments that were written in English and publicly available.Furthermore, we did not have contact with any of the institutions or creators of the evaluation instruments.Additional studies might be undertaken to solicit rubrics from institutions and organizations.Chickering and Gamson (1999) worked to produce a framework that was "accessible, understandable, practical, and widely applicable" (p.76).Future studies could also investigate the use of a rubric that assesses for the Seven Principles and its impact on student satisfaction and learning.

Table 1
Evaluation Instruments Sorted by Category • Fayetteville State University Online Course Rubric • Florida Gulf Coast University Principles of Online Design • Kansas State E-Learning Quality Checklist • Pennsylvania State Quality Assurance e-Learning Design Standards • Rochester Institute of Technology Online Course Design Checklist • Southern Oregon University Best Practices in Online Course Design and Delivery • Southern Polytechnic State University Course Assessment Checklist • University of California Irvine Best Practices in Online Course Design • University of New Mexico Online Course Standards Rubric • University of North Texas Online Course Design Checklist • University of Wisconsin-La Crosse Online Course Evaluation Guidelines • Utah State Online Course Quality Rubric

Table 2
"A variety of instructional delivery methods, accommodating multiple learning styles are available throughout the course"(Quality Online Course Initiative Rubric, Illinois Online Network, University of Illinois, 2010, p. 1).

Table 3
Number of Evaluation Instruments Assessing the Seven Principles of Good Practice

Table 4
Number of Seven Principles of Good Practice Assessed by National or Statewide Evaluation Instruments

Table 6
Number of Seven Principles of Good Practice Assessed by Institute Specific (Two-Year) Evaluation Instruments

Table 7
Number of Seven Principles of Good Practice Assessed by Online Professional Development Evaluation Instruments