Evaluating Online Course Quality: A Study on Implementation of Course Quality Standards


  • Whitney Zimmerman The Pennsylvania State University
  • Barbara Altman Texas A&M University
  • Bethany Simunich Quality Matters
  • Kay Shattuck Quality Matters
  • Barbra Burch Quality Matters




course design, professional development, quality assurance, quality standards


This study examined the relationship of intentional faculty professional development, intentional online course design, and informal course reviews to the results of official interinstitutional peer review within higher education institutions. Quality MattersTM (QM) provided the setting for this exploration of the relationship of three independent variables at the course level at institutions that have voluntarily implemented QM online learning quality assurance tools and processes. Data for this study were extracted from a larger statistical project conducted regularly by QM, which included the results of 5,436 online course reviews completed between September 2014 and May 2020 at 360 institutions. These courses were assessed for meeting quality standards in structured, interinstitutional, reviews, conducted by three faculty peer reviewers. QM provided the setting and data for this study; however, the study was not about QM. Instead, it was about exploring the relationships of variables within an institution’s control in the quest for benchmarking and improving online learning. Having and disseminating online course quality standards does not ensure implementation of those standards and quality assurance processes. This observational study provides a better understanding of how the implementation of those standards and quality assurance processes might impact outcomes.


Adair, D. (2014). A process to improve course design: A key variable in course quality. In K. Shattuck (Ed.), Assuring quality in online education: Practices and processes at teaching, resource, and program levels (pp. 81–90). Stylus.

Adair, D., Shattuck, K., Burch, B., & Zimmerman, W. A. (2019, November). 15 years of data on improving online learning design. [Conference presentation]. ICDE World Conference on Online Learning, Dublin, Ireland.

Adair, D., & Shattuck, K. (2019). Ensuring quality while creating and innovating. In K. E. Linder (Ed.), The business of innovating online: Practical tips and advice from industry leaders (pp. 97–112). Stylus.

Baldwin, S. (2019). Assimilation in online course design. The American Journal of Distance Education, 33(3), 195–211. https://doi.org/10.1080/08923647.2019.1610304

Bazluki, M., Gyabak, K., & Uderman, B. (2018). Instructor feedback on a formal online course quality assurance review process. Online Journal of Distance Learning Administration, 21(2). https://www.westga.edu/~distance/ojdla/summer212/bazluki_gyabak_udermann212.html

Bogle, L., Day, S., Matthews, D., & Swan, K. (2014). The power of collaborative, collegial approach to improving online teaching and learning. In K. Shattuck (Ed.), Assuring quality in online learning (pp. 110–123). Stylus.

Bond, J., & Dirkin, K. (2018). Instructional design: Study of a widening scope of practice. Online Journal of Distance Learning Administration, 21(4). https://www.westga.edu/~distance/ojdla/winter214/bond_dirkin214.html

Brinkley-Etzkorn, K. E. (2019). The effects of training on instructor beliefs about and attitudes toward online teaching. The American Journal of Distance Education, 34(1), 19–35. https://doi.org/10.1080/08923647.2020.1692553

Christie, B., & Skylar, A. (2017, September 26). QM statewide systems talk data: Inputs, outputs, and outcomes [Conference presentation]. QM Connect Conference, Fort Worth, TX, United States. https://www.qualitymatters.org/qa-resources/resource-center/conference-presentations/qm-statewide-systems-talk-data-inputs-outputs

Cowan, J., Richter, S., Miller, T., Rhode, J., Click, A., & Underwood, J. (2017). Building a sustainable Quality MattersTM community of practice through social network analysis. American Journal of Distance Education, 31(3), 160–172. doi: 10.1080/08923647.2017.1316154

Fishman, B. J., Penuel, W. R., Allen, A.-R., Cheng, B. H., & Sabelli, N. (2013). Design-based implementation research: An emerging model for transforming the relationship of research and practice. In B. J. Fishman & W. R. Penuel (Eds.), National Society for the study of education: Vol 112 (pp. 136–156). http://isls-naples.psy.lmu.de/video-resources/guided-tour/15-minutes-penuel/fishman_penuel.pdf

Esfijani, A. (2018). Measuring quality in online education: A meta-synthesis. American Journal of Distance Education, 32(1), 57–73. https://doi.org/10.1080/08923647.2018.1417658

Garrett, R., Legon, R., Fredericksen, E. E., & Simunich, B. (2020). CHLOE 5: The pivot to remote teaching in spring 2020 and its impact, the changing landscape of online education, 2020. Quality Matters. https://www.qualitymatters.org/qa-resources/resource-center/articles-resources/CHLOE-5-report-2020

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education model. The Internet and Higher Education, 2(2–3), 87–105.

Geiger, L., Morris, D., Subocz, S. L., Shattuck, K., Viterito, A. (2014). Effect of student readiness on student success in online courses. Internet Learning, 3(1), 73–83. https://joom.ag/w08b/p72

Gibbons, S., Yauk, M., & See, K. K. (2019). Trends and directions in continuing professional education, In M. G. Moore & W. C. Diehl (Eds.), Handbook of distance education (4th ed., pp. 456–472). Routledge Press. https://doi.org/10.4324/9781315296135

Goodyear, P. (2015). Teaching as design. Herdsa Review of Higher Education, 2(2), 27–50. https://www.academia.edu/32521987/Teaching_as_design?email_work_card=view-paper

Gregory, R. L. (2018). Influence of Quality Matters™ professional development on faculty members’ perceptions of design standards and their course development abilities. [Doctoral dissertation, The University of Memphis]. https://www.proquest.com/docview/2049753884

Harkness, S. S. J. (2014). Program administration and implementation of an online learning initiative at a Historically Black College University: A case study [Webinar]. EDUCAUSE/Quality Matters Online and Blended Learning: Institutional Case Studies on Implementing a Quality Assurance Program and Designing Research on Effective Practice Webinar Series. http://www.educause.edu/sites/default/files/library/presentations/ELI143/OL01/Harkness_Online%2BLearning%2BInitiative.pdf

Hollowell, G. P., Brooks, R. M., & Anderson, Y. B. (2017). Course design, quality matters training, and student outcomes. American Journal of Distance Education, 31(3), 207–216. https://doi.org/10.1080/08923647.2017.1301144

International Board of Standards for Training, Performance and Instruction. (n.d.). About us. http://ibstpi.org/about-us/

Kamenetskiy, M. (2015). Evaluating faculty perceptions of teaching practices in online asynchronous courses: An action research study. (Publication No. 3719498) [Doctoral dissertation, Capella University]. ProQuest Dissertations and Theses Global.

Kennedy, A. (2015). Faculty perceptions of the usefulness of and participation in professional development for online teaching: An analysis of faculty development and online teaching satisfaction. (Publication No.3722998) [Doctoral dissertation, University of Wyoming]. ProQuest Dissertations and Theses Global.

Kearns, L. R. (2015). The experience of teaching online: Its impact on faculty professional development and innovation. [Doctoral dissertation, The Pennsylvania State University] Penn State Libraries. https://etda.libraries.psu.edu/files/final_submissions/10474

Kearns, L. R., & Mancilla, R. (2016). The impact of Quality Matters professional development on teaching across delivery formats. Quality Matters. https://www.qualitymatters.org/sites/default/files/research-docs-pdfs/Impact-of-QM%20Professional-Development-on-Teaching-across-Delivery-Formats.pdf

Kumar, S., Martin, F., Budhrani, K., & Ritzhaupt, A. (2019). Award-winning faculty online teaching practices: Elements of award-winning courses. Online Learning, 23(4), 160–180. doi:10.24059/olj.v23i4.2077

McKay, S. (2017, March 15). Quality improvement approaches: Implementation science. Carnegie Foundation for the Advancement of Teaching. https://www.carnegiefoundation.org/blog/quality-improvement-approaches-implementation-science/

Means, B., et al. (2020). Suddenly online: A national survey of undergraduates during the COVID-19 pandemic. Digital Promise.

Mohr, S. C., & Shelton, K. (2017). Best practices framework for online faculty professional development: A Delphi study. Online Learning Journal, 21(4), 123–140. https://olj.onlinelearningconsortium.org/index.php/olj/article/view/1273

Moore, M. G. (2019). The theory of transactional distance. In M. G. Moore & W. C. Diehl, (Eds.), Handbook of distance education, (4th ed., pp. 32–46). Routledge. https://doi.org/10.4324/9781315296135

Moore, M. G., & Kearsley, G. (2012). Distance education: A systems view of online learning (3rd ed.). Wadsworth.

McNeal, L., & Gray, J. (2019). A new spin on quality: Broadening online course reviews through coaching and slow thinking. Online Journal of Distance Learning Administration, 22(4). https://www.westga.edu/~distance/ojdla/winter224/mcnealGray224.html

Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10(53). doi 10.1186/s13012-015-0242-0 https://implementationscience.biomedcentral.com/track/pdf/10.1186/s13012-015-0242-0

O’Brien, E. (2015). A case study evaluation of quality standards and online faculty development [Doctoral dissertation, University of Central Florida]. University of Central Florida Library. https://stars.library.ucf.edu/etd/1239

Piña, A. A., & Harris, P. (2019) Utilizing the AECT instructional design standards for distance learning. Online Journal of Distance Learning Administration, 22(2). https://www.westga.edu/~distance/ojdla/summer222/pina_harris222.html

Quality Matters, (n.d.). Member institutions/organizations. https://www.qmprogram.org/qmresources/subscriptions/subscribers.cfm?program=0

Quality Matters, (n.d.). Quality assurance begins with a set of standards. https://www.qualitymatters.org/qa-resources/rubric-standards

Rhode, J., Richter, S., & Miller, T. (2017). Designing personalized online teaching professional development through self-assessment. TechTrends, 61(5), 444–451. http://www.aeraotl.com/otl-news

Richardson, J. W., Hollis, E., Pritchard, M., & Lingat, J. E. M. (2020). Shifting teaching and learning in online learning spaces: An investigation of a faculty online teaching and learning initiative. Online Learning 24(1), 67–91. https://doi.org/10.24059/olj.v24i1.1629

Ryan, P. (2015). Quality assurance in higher education: A review of literature. Higher Learning Research Communications, 5(4). http://dx.doi.org/10.18870/hlrc.v5i4.257

Sanga, W. M. (2018). Getting to master online teaching: Insights from purposefully organized course development training. Quarterly Review of Distance Education, 19(2), 15–22.

Scarpena, K., Riley, M., & Keathley, M. (2018). Creating successful professional development activities for online faculty: A reorganized framework. Online Journal of Distance Learning Administration, 21(1). https://www.westga.edu/~distance/ojdla/spring211/scarpen_riley_keathley211.html

Seaman, J. E., Allen, I. E., & Seaman, J. (2018). Grade increase: Tracking distance education in the United States. Babson Survey Research Group. https://onlinelearningsurvey.com/reports/gradeincrease.pdf

Shattuck, K. (2019). Teaching online: Where? what? who? when? why? how? In M. G. Moore & W. C. Diehl (Eds.), Handbook of distance education (4th ed., pp. 428–442). Routledge Press. https://doi.org/10.4324/9781315296135

Shattuck, K., Zimmerman, W., & Adair, D. (2014). Continuous improvement of the QM Rubric and review processes: Scholarship of integration and application. Internet Learning, 3(1), 25–34. http://joom.ag/w08b/p24

Shearer, R. L., Park, E. (2019). Theory to practice in instructional design. In M. G. Moore & W. C. Diehl (Eds.), Handbook of distance education (4th ed., pp. 260–280). Routledge Press. https://doi.org/10.4324/9781315296135

Shelton, K. (2010). A quality scorecard for the administration of Online Education programs: A Delphi study (Publication No. 3423965). [Doctoral dissertation, University of Nebraska]. ProQuest Dissertations and Theses Global.

Swan, K., Day, S. L., Bogle, L. R., & Matthews, D. B. (2014). A collaborative, design-based approach to improving an online program. Internet and Higher Education, (21), 74–81. https://doi.org/10.1016/j.iheduc.2013.10.006

Taylor, C., Roehrich, H., & Grabanski, J. (2018). External factors that impact online instructor performance: A study measuring the impact of decision fatigue & quality matters recognition of courses on online instructor evaluation. Online Journal of Distance Learning Administration, 21(3). https://www.westga.edu/~distance/ojdla/fall213/taylor_roehrich_grabanski213.html






Special Conference Issue: AERA Online Teaching and Learning SIG