Evaluation Instruments and Good Practices in Online Education
DOI:
https://doi.org/10.24059/olj.v21i2.913Keywords:
online evaluation, assessment, quality, best practicesAbstract
Chickering and Gamson’s (1987) Seven Principles for Good Practice in Undergraduate Education offers extensively researched and validated tenets for best practices in higher education. After a review of the literature, twenty-eight evaluation instruments currently used to design and review online courses in higher education institutions were collected and divided into categories, based on geographical reach and the type of institution for which they were developed. This study investigates how evaluation instruments used in higher education assess the Seven Principles for Good Practice in Undergraduate Education, and what other items are addressed in the evaluation of courses. Findings show that national and statewide evaluation instruments were less institute specific and more closely aligned to the principles of good practice, and that evaluation instruments often measure extraneous items (e.g., student services, navigation, resources, or institutional support). Additional findings and conclusions based on the analysis of the instruments are discussed.
References
Achtemeier, S. D., Morris, L. V., & Finnegan, C. L. (2003). Considerations for developing evaluations of online courses. Journal of Asynchronous Learning Networks, 7(1), 1-13.
Allen, I. E., & Seaman, J. (2015). 2014 Survey of online learning. Grade level: Tracking online education in the United States, 2014. Babson Survey Research Group and Quahog Research Group, LLC. Retrieved from http://onlinelearningconsortium.org/read/survey-reports-2014/
Astin, A. W. (2012). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. Lanham, MD: Rowman & Littlefield Publishers.
Bangert, A. W. (2006). The Development of an Instrument for Assessing Online Teaching Effectiveness. Journal of Educational Computing Research, 35(3), 227-244.
Bangert, A. W. (2004). The seven principles of good practice: A framework for evaluating online teaching. The Internet and Higher Education, 7(3), 217-232.
Billings, D. M. (2000). A framework for assessing outcomes and practices in web-based courses in nursing. Journal of Nursing Education, 39(2), 60–67.
Blackboard. (2015). Community. Retrieved from http://www.blackboard.com/community/catalyst-awards/past-winners.aspx
Boettcher, J. V., & Conrad, R. M. (2010). The online teaching survival guide: Simple and practical pedagogical tips. New York, NY: John Wiley & Sons.
Boettcher, J. V. (2011). Ten best practices for teaching online: Quick guide for new online faculty. Designing for Learning. Retrieved from http://idedtech.pbworks.com/w/file/fetch/83414902/Ten%20Best%20Practices%20for%20Teaching%20Online.pdf
Bluegrass Community and Technical College. (2011, April 13). Quality assurance checklist for online course design and development. Retrieved from http://district.bluegrass.kctcs.edu/ben.worth/distance_learning/QA_Checksheet.pdf
California Community College Online Education Initiative. (2015). Course design rubric for the online education initiative. Retrieved from http://webcache.googleusercontent.com/search?q=cache:r3dS7uSAyo4J:ccconlineed.org/documents/category/6-professional-development-workgroup%3Fdownload%3D46:oei-course-design-rubric+&cd=1&hl=en&ct=clnk&gl=us&client=safari
California State University. (2015). Quality online learning and teaching rubric. Retrieved from http://courseredesign.csuprojects.org/wp/wp-content/uploads/2013/11/QOLT-QM-Alignment-Matrix-.pdf
California State University, Chico. (2015). Welcome to the exemplary online instruction (EOI) website. Retrieved from http://www.csuchico.edu/eoi/
Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as lever. AAHE bulletin, 49, 3-6. Retrieved from http://sphweb.bumc.bu.edu/otlt/teachingLibrary/Technology/seven_principles.pdf
Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE bulletin, 3, 7.
Chickering, A. W., & Gamson, Z. F. (1999). Development and adaptations of the seven principles for good practice in undergraduate education. New Directions for Teaching and Learning, 1999(80), 75-81.
Chickering, A. W., & Gamson, Z. F. (2002). Seven principles of good practice: A FEEDS Evaluation.
Chua, A., & Lam, W. (2007). Quality assurance in online education: The Universitas 21 Global approach. British Journal of Educational Technology, 38(1), 133-152.
Creasman, P. A. (2012). Considerations in online course design. IDEA paper, 52.
Crews, T. B., Wilkinson, K., & Neill, J. K. (2015). Principles for good practice in undergraduate education: Effective online course design to assist students' success. Journal of Online Learning and Teaching, 11(1), 87.
Cross, K. P. (1999). What do we know about students' learning, and how do we know it? Innovative Higher Education, 23(4).
Dayton, D., & Vaughn, M. M. (2007). Developing a quality assurance process to guide the design and assessment of online courses. Technical Communication, 54(4), 475-489.
Dirkin, K. H. (2008). Three professors teaching online: The realization of teaching perspectives (Doctoral dissertation). Retrieved from ProQuest. (UMI Number: 3331894)
Dixson, M. D. (2010). Creating effective student engagement in online courses: What do students find engaging? Journal of the Scholarship of Teaching and Learning, 10(2), 1-13.
Duke University. (n.d.). What is the difference between assessment and evaluation? Retrieved from http://arc.duke.edu/documents/The%20difference%20between%20assessment%20and%20evaluation.pdf
Educational Development, Inc. (2015). EdTech leaders online course elements. Retrieved from http://courses.edtechleaders.org/documents/OCD/Course_Dev_Check.htm
Fayetteville State University. (2015). Planning and developing new online programs. Office of Faculty Development. Retrieved from http://www.uncfsu.edu/facultydevelopment/faculty-services
Florida Gulf Coast University. (2006). Principles of online design checklist. Faculty Development and Support. Retrieved from http://www.fgcu.edu/onlinedesign/Intro.html
Gaytan, J., & McEwen, B. C. (2007). Effective online instructional and assessment strategies. The American Journal of Distance Education, 21(3), 117-132.
Gawande, A. (2010). The checklist manifesto: How to get things right (Vol. 200). New York, NY: Metropolitan Books.
Graham, C., Caglitay, K., Lim, B., Craner, J., & Duffy, T. M. (2001). Seven principles for effective teaching: A practical lens for evaluating online courses. The Technology Source. Retrieved from http://ts.mivu.org/default.asp?show=article&id=839
Hathaway, K. L. (2013). An application of the seven principles of good practice to online courses. Research in Higher Education Journal, 22, 1.
Haynie, D. (2015, February 5). Study shows sluggish online learning growth for second year. U.S. News and World Report. Retrieved from http://www.usnews.com/education/online-education/articles/2015/02/05/study-shows-sluggish-online-learning-growth-for-second-year
Henry, J., & Meadows, J. (2008). An absolutely riveting online course: Nine principles for excellence in web-based teaching. Canadian Journal of Learning and Technology, 34(1).
Higher Learning Commission. (2015). The criteria for accreditation and core components. Retrieved from https://www.hlcommission.org/Criteria-Eligibility-and-Candidacy/criteria-and-core-components.html
Hosie, P., Schibeci, R., & Backhaus, A. (2005). A framework and checklists for evaluating online learning in higher education. Assessment & Evaluation in Higher Education, 30(5), 539-553.
Illinois Online Network. (2010). Quality online course initiative rubric & checklist. University of Illinois. Retrieved from http://www.ion.uillinois.edu/initiatives/qoci/rubric.asp
Johnston, J., Killion, J., & Oomen, J. (2005). Student satisfaction in the virtual classroom. The Internet Journal of Allied Health Sciences and Practice,3(2),1-7.
Kansas State University. (2013). Kansas State e-learning quality checklist. Retrieved from http://elearningfacultymodules.org/uploaded/QualityELearningChecklist.pdf
Krause, J., Dias, L. P., & Schedler, C. (2015). Competency-based education: A framework for measuring quality courses. Online Journal of Distance Learning Administration, 18(1).
Learning Resource Network (LERN). (2015). Online course best practices checklist. Retrieved from http://teachingonthenet.org/best-practices-standards.cfm
Lewis and Clark Community College (2012). Checklist for instructor review of online and hybrid course design. Retrieved from http://www.lc.edu/uploadedFiles/Pages/Audiences/Online%20Course%20Review%20Checklist.pdf
Malone, B. G., Malm, L. D., Nay, F. W., Oliver, B. E., Saunders, N. G., & Thompson, J. C. (1997). Observation of instruction via distance learning: The need for a new evaluation paradigm. Paper presented at the Annual Meeting of the Midwestern Educational Research Association, Chicago, Illinois.
Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation, 7(25), 1-10.
Meyer, K. A. (2002, January 1). Quality in distance education: Focus on online learning. ASHE-ERIC Higher Education Report. San Francisco, CA: Jossey-Bass Higher and Adult Education Series.
Morgan, G. (2003). Faculty use of course management systems (Vol. 2). ECAR, EDUCAUSE Center for Applied Research. Retrieved from http://zmam-almarefah.com/admin/template/images/file2015-04-07-01-12-05.pdf
Newlin, M. H., & Wang, A. Y. (2002). Integrating technology and pedagogy: Web instruction and seven principles of undergraduate education. Teaching of Psychology, 29(4), 325-330.
Northeast Community College. (n.d.). NECC rubric for online course design standards. Retrieved from http://www.northeast.edu/About-Us/Accreditation/References/Basic-Design-Standards-for-Online-Courses.pdf
Online Learning Consortium. (2015). Home. Retrieved from http://onlinelearningconsortium.org
Palomar College. (2012). Online course best practices checklist. Academic Technology Committee. Retrieved from http://www2.palomar.edu/poet/BestPracticesChecklistSP12.pdf
Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students. San Francisco, CA: Jossey-Bass.
Pickett, A. M. (2009). SUNY learning network rubric for formal course review of new online faculty/courses. Retrieved from http://wiki.sln.suny.edu/download/attachments/4032379/Course_Review_Rubric.pdf?version=1&modificationDate=1277483875246
Portland Community College. (2015). Portland Community College online course development guide. Retrieved from http://www.pcc.edu/about/distance/faculty/development/course-development-questionnaire/
Prineas, M., & Cini, M. (2011). Assessing learning in online education: the role of technology in improving student outcomes. National Institute for Learning Outcomes Assessment. Retrieved from http://www.learningoutcomeassessment.org/occasionalpapertwelve.htm
Ralston-Berg, P. (2014). Surveying student perspectives of quality: Value of QM rubric items. Internet Learning Journal, 3(1).
Ralston-Berg, P., Buckenmeyer, J., Barczyk, C., & Hixon, E. (2015). Students’ perceptions of online course quality: How do they measure up to the research? Internet Learning, 4(1), 3.
Reddy, Y. M. & Andrade, H. (2010). A review of rubric use in higher education. Assessment and Evaluation in Higher Education, 35, 435–448.
Reeves, T. C., & Reeves, P. M. (1997). Effective dimensions of interactive learning on the World Wide Web. In Bradual H. Kahn (Ed.), Internet-based instruction (pp. 59–66). Englewood Cliffs, NJ: Educational Technology Publications.
Roblyer, M. D., & Wiencke, W. R. (2004). Exploring the interaction equation: Validating a rubric to assess and encourage interaction in distance courses. Journal of Asynchronous Learning Networks, 8(4), 24-37.
Rochester Institute of Technology. (2014) Online course design checklist. Retrieved from https://www.rit.edu/academicaffairs/tls/sites/rit.edu.academicaffairs.tls/files/docs/RIT%20Online%20Design%20Checklist_Jul2014_v1.pdf
Seaman, J. (2009). Online learning as a strategic asset. Volume II: The paradox of faculty voices--Views and experiences with online Learning. Results of a national faculty survey, part of the Online Education Benchmarking Study Conducted by the APLU-Sloan National Commission on Online Learning. Association of Public and Land-grant Universities.
Shattuck, K. (2015, September 15). Research inputs and outputs of Quality Matters: Update to 2012 and 2014 versions of What we’re learning from QM-focused research. Retrieved from https://www.qualitymatters.org/what-were-learning-paperfinalmay-18-2012dec2012kspdf/download/What%20we're%20learning%20paper_FINAL_May%2018,%202012_Dec2012ks.pdf
Shattuck, K. (2013, November 25). Results of review of the 2011-2013 research literature. Quality Matters. Retrieved from https://www.qualitymatters.org/node/2307/download/2013%20lit%20review%20summary%20report%2011-25-13-revised-ks.pdf
Shattuck, K., Zimmerman, W. A., & Adair, D. (2015). Continuous improvement of the QM rubric and review processes: Scholarship of integration and application. Internet Learning, 3(1), 5.
Shelton, K., & Saltsman, G. (2014). Quality scorecard 2014 handbook: Criteria for excellence in the administration of online programs. USA: Online Learning Consortium.
Sorcinelli, M. D. (1991). Research findings on the seven principles. In A. W. Chickering & Z. F. Gamson (Eds.) Applying the seven principles for good practice in undergraduate education (pp. 13-25). New Directions for Teaching and Learning, No. 47. San Francisco, CA: Jossey-Bass.
Southern Oregon Distance Education Center. (2010, September 10). Best practice in online course design delivery. Southern Oregon University. Retrieved from http://www.sou.edu/distancelearning/SOU%20DEC%20Best%20Practices.pdf
Southern Polytechnic State University (n.d.). Kennesaw State University. Retrieved from http://www.kennesaw.edu
Southern Regional Education Board. (2006). SREB checklist for evaluating online courses. Retrieved from http://publications.sreb.org/2006/06T06_Checklist_for_Evaluating-Online-Courses.pdf
Sowan, A. K., & Jenkins, L. S. (2013). Use of the seven principles of effective teaching to design and deliver an interactive hybrid nursing research course. Nursing Education Perspectives, 34(5), 315-22.
Sunal, D. W., Sunal, C. S., Odell, M. R., & Sundberg, C. A. (2003). Research-supported best practices for developing online learning. Journal of Online Interactive Learning, 2(1).
Ternus, M. P., Palmer, K. L., & Faulk, D. R. (2007). Benchmarking quality in online teaching and learning: A rubric for course construction and evaluation. Journal of Effective Teaching, 7(2), 51-67.
The Pennsylvania State University. (2015). Penn State quality assurance e-learning design standards. Web Learning @ Penn State. Retrieved from https://weblearning.psu.edu/resources/penn-state-online-resources/qualityassurance/
Three Rivers Community College. (n.d.) Three Rivers Community College online course design checklist. Retrieved from http://www.threerivers.edu/Div_IT/EducationalTechnology/PDF/OnlineCourseReviewChecklist.pdf
University of California, Irvine. (2012). Best practices in online course design. UC Irvine, Teaching, Learning & Technology Center. Retrieved from http://www.tltc.uci.edu/pdf/BEST%20PRACTICES_2.pdf
University of New Mexico. (2015, May 1). Online course standards rubric. Retrieved from http://newmedia.unm.edu/common/documents/OnlineCourseStandardsRubric.pdf
University of North Texas. (2012, April 12). UNT online course design checklist. Retrieved from https://clear.unt.edu/sites/default/files/UNTcourse_design_checklist_rev4_12.pdf
University of Wisconsin - La Crosse. (2014, May 23). UW-La Crosse online course evaluation guidelines. Retrieved from http://www.uwlax.edu/uploadedFiles/Academics/online/guidelines.pdf
Utah State. University. (n.d.). Utah State online course quality rubric. Retrieved from https://elearn.usu.edu/courses/online-rubric-form.pdf
Wainscott, L., & Murphy, C. (2006, March). Designing online courses to encourage community with the seven principles of good practice. In Society for Information Technology & Teacher Education International Conference (Vol. 2006, No. 1, pp. 2708-2712).
Watwood, B., Nugent, L., & Deihl, W. (2009). Building from content to community: Rethinking the transition to online teaching and learning: A CTE White Paper. Richmond, VA: Virginia Commonwealth University: Center for Teaching Excellence.
Wiggins, G. P. (1998). Educative assessment: Designing assessments to inform and improve student performance (Vol. 1). San Francisco, CA: Jossey-Bass.
Downloads
Published
Issue
Section
License
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
