The Validity and Instructional Value of a Rubric for Evaluating Online Course Quality: An Empirical Study
DOI:
https://doi.org/10.24059/olj.v24i1.1949Keywords:
Online course quality, Rubric, Online interactions, Rubric Reliability, Rubric ValidityAbstract
This study investigated the validity and instructional value of a rubric developed to evaluate the quality of online courses offered at a midsized public university. This rubric was adapted from an online course quality rubric widely used in higher education, the Quality Matters rubric. We first examined the reliability and preliminary construct validity of the rubric using quality ratings for 202 online courses and eliminated twelve problematic items. We then examined the instructional value of the rubric by investigating causal relationships between 1) course quality scores, 2) online interactions between students, instructors, and content, and 3) student course performance (course passing rates). A path analysis model, using data from 121 online courses enrolling 5,240 students, showed that only rubric items related to learner engagement and interaction had a significant and positive effect on online interactions, while only student-content interaction significantly and positively influenced course passing rates.
References
Anderson, T. (2008). The theory and practice of online learning. Athabasca, Canada: AU Press.
Anderson, T. D., & Garrison, D. R. (1998). Learning in a networked world: New roles and responsibilities. In C. C. Gibson (Ed.), Distance learners in higher education (pp. 97–112). Madison, WI: Atwood Publishing.
Author. (2003).
Author. (2015).
Author. (2016).
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289.
Blackboard Inc. (2017). Blackboard exemplary course program rubric. Retrieved from https://community.blackboard.com/docs/DOC-3505-blackboard-exemplary-course-program-rubric
Borokhovski, E., Tamim, R., Bernard, R. M., Abrami, P. C., & Sokolovskaya, A. (2012). Are contextual and designed student-student interaction treatments equally effective in distance education? Distance Education, 33(3), 311–329.
Britto, M., Ford, C., & Wise, J. M. (2013). Three institutions, three approaches, one goal: addressing quality assurance in online learning. Online Learning Journal, 17(4).
California Community College (2016). Course design rubric for the online education initiative. Retrieved from http://ccconlineed.org/wp-content/uploads/2015/11/OEI_Rubric_Edited-ACC.pdf
California State University (2015). Quality learning and teaching. Retrieved from https://www.csun.edu/it/qlt
Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education (6th ed.). New York, NY: Routledge.
Custard, M., & Sumner, T. (2005). Using machine learning to support quality judgments. D-Lib Magazine, 11(10), Retrieved from http://www.dlib.org/dlib/october05/custard/10custard.html
Egerton, E. O., & Posey, L. (2007). Quality standards inventory. Retrieved from http://learn.gwnursing.org/Education/QSI/QSI.html
Han, L., Neilands, T. B., & Dolcini, M. M. (2001). Factor analysis of categorical data in SAS. Retrieved from https://www.lexjansen.com/wuss/2001/WUSS01044.pdf
Hixon, E., Barczyk, C., Ralston-Berg, P., & Buckenmeyer, J. (2016). The impact of previous online course experience students' perceptions of quality. Online Learning, 20(1), 25-40.
Hoey, R. (2017). Examining the characteristics and content of instructor discussion interaction upon student outcomes in an online course. Online Learning, 21(4), 268–281.
Illinois Center College (2017). Quality online course initiative. Retrieved from https://icc.edu/faculty-staff/files/ICC-QOCI-Version-4.4-published-05.04.17-2.pdf
Jaggars, S. S., & Xu, D. (2016). How do online course design features influence student performance? Computers & Education, 95, 270–284.
Ke, F. (2013). Online interaction arrangements on quality of online interactions performed by diverse learners across disciplines. Internet and Higher Education, 16(1), 14–22.
Kuo, Y., Walker, A. E., Belland, B. R., & Schroder, K. E. E. (2013). A predictive study of student satisfaction in online education programs in online courses. International Review of Research in Open and Distance Learning, 14(1), 1–39.
Lee, J. (2014). An exploratory study of effective online learning: Assessing satisfaction levels of graduate students of mathematics education associated with human and design factors of an online course. International Review of Research in Open and Distance Learning, 15(1), 111–132.
Lee, Y., & Choi, J. (2011). A review of online course dropout research: Implications for practice and future research. Educational Technology Research and Development, 59(5), 593-618.
Liu, I. F., Chen, M. C., Sun, Y. S., Wible, D., & Kuo, C. H. (2010). Extending the TAM model to explore the factors that affect intention to use an online learning community. Computers & Education, 54(2), 600–610.
McFarland, J., Hussar, B., Wang, X., Zhang, J., Wang, K., Rathbun, A., Barmer, A., Forrest Cataldi, E., and Bullock Mann, F. (2018). The Condition of education 2018 (NCES 2018-144). U.S. Department of Education. Washington, DC: National Center for Education Statistics.
Moore, M. G. (1989). Three types of interaction. American Journal of Distance Education, 3, 1–7.
Murray, M., Pérez, J., Geist, D., & Hedrick, A. (2012). Student interaction with online course content: Build it and they might come. Journal of Information Technology Education: Research, 11, 125–140.
New Mexico State University (2011). Online course design rubric. Retrieved from http://hpdr.education.nmsu.edu/files/2013/07/on-line-course-design-rubric-.pdf
Palomar College (2012). Online course best practices checklist. Retrieved from http://www2.palomar.edu/poet/BestPracticesChecklistSP12.pdf
Quality Matters (2018). Quality matters higher education rubric.
Retrieved from https://www.qualitymatters.org/sites/default/files/PDFs/StandardsfromtheQMHigherEducationRubric.pdf
Raubenheimer, J. E. (2004). An item selection procedure to maximize scale reliability and validity. SA Journal of Industrial Psychology, 30(4), 59–64.
Reigeluth, C. (1999). Instructional design theories and models: A new paradigm of instructional theory. Mahwah, NJ: Lawrence Erlbaum Associates.
Roblyer, M., & Wiencke, W. (2003). Design and use of a rubric to assess and encourage interactive qualities in distance courses. American Journal of Distance Education, 17(2), 77–98.
Sher, A. (2009). Assessing the relationship of student-instructor and student-student interaction to student learning and satisfaction in web-based online learning environment. Journal of Interactive Online Learning, 8(2), 102–120.
Southern Regional Education Board (2006). Checklist for evaluating online courses. Retrieved from https://www.sreb.org/publication/checklist-evaluating-online-courses
Sun, P. C., Tsai, R. J., Finger, G., Chen, Y. Y., & Yeh, D. (2008). What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1183–1202.
Swan, K., Matthews, D., Bogle, L., Boles, E., & Day, S. (2012). Linking online course design and implementation to learning outcomes: A design experiment. The Internet and Higher Education, 15, 81–88.
University of North Dakota (2016). Rubric for evaluating online courses. Retrieved from http://www1.und.edu/academics/center-for-instructional-and-learning-technologies/_files/docs/bestpracticesrubric.pdf
Yong, A. G., & Pearce, S. (2013). A beginner’s guide to factor analysis focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79–94.
Downloads
Published
Issue
Section
License
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
