Advancing Sociotechnical-Pedagogical Heuristics for the Usability Evaluation of Online Courses for Adult Learners

Isa Jahnke, Nathan Riedel, Kanupriya Singh, Joi Moore


Online courses often include interface designs that do not support a positive learner experience. Literature shows a variety of heuristics to detect issues of online courses. While heuristic-based inspection of usability is a dominant method for evaluating digital systems, these methods cannot be easily transferred to online courses. To close this gap, we identified an initial set of social, technical, and pedagogical related items (STP) heuristics based on literature. Next, we analyzed this set using empirical data from two online courses. In total, we analyzed 195 problems with the goal to substantiate a final set of 14 STP heuristics. This new set allows for efficiently evaluating online courses by supporting evaluators and instructional designers in uncovering the most crucial issues and improving the learner experience. Finally, based on this work, we discuss a definition of learner experience for the emerging field of learner experience design and research.



Online Courses, Evaluation, Heuristics, Usability, Online Learning, Adults

Full Text:



Albion, P. (1999a). Heuristic evaluation of educational multimedia: From theory to practice. In J. Winn (Ed.), Proceedings of the 16th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 9–15). Australasian Society for Computers in Learning in Tertiary Education.

Baldwin, S. J., Ching, Y. H., & Friesen, N. (2018). Online course design and development among college and university instructors: An analysis using grounded theory. Online Learning, 22(2), 157–171. doi:10.24059/olj.v22i2.1212

Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, S., &

Reeves, T. C. (2002). Usability and instructional design heuristics for e-learning evaluation. In P. Barker & S. Rebelsky (Eds.), Proceedings from ED-MEDIA 2002: World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 1615–1621). AACE.

Bloom, B. (1956). Taxonomy of educational objectives handbook I: The cognitive domain. Addison Wesley Longman, Inc.

Botella, F., Rusu, C., Rusu, V., & Quiñones, D. (2018, Sept.). How novel evaluators perceive their first heuristic evaluation. Proceedings of XIX International Conference on Human Computer Interaction. Association for Computing Machinery.

Boyle, T. (1997). Design for multimedia learning. Prentice Hall.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101.

Chao, T., Saj, T., & Tessier, F. (2006). Establishing a quality review for online courses. Educause Quarterly, 29(3), 32–39.

Chen, B., Chang, Y. H., Ouyang, F., & Zhou, W. (2018). Fostering student engagement in online discussion through social learning analytics. The Internet and Higher Education, 37, 21–30.

Clement, F. J. (1985). Technology & instruction: The rule of seven revisited. Performance + Instruction, 24(2), 6–8.

Das, J. (2012). Role of syllabus in higher education: A critical study. Global Research Methodology Journal, 7, 1–2.

Demmans Epp, C., Phirangee, K., Hewitt, J., & Perfetti, C. A. (2020). Learning management system and course influences on student actions and learning experiences. Education Tech Research Dev, 68, 3263–3297.

Dringus, L. P., & Cohen, M. S. (2005, Oct. 19–22). An adaptable usability heuristic checklist for online courses. The Frontiers in Education 35th Annual Conference. IEEE.

Douglas, S. (2017, Dec. 6). 7 usability heuristics that all UI designers should know. UsabilityGeek.

Fink, S. B. (2012). The many purposes of course syllabi: Which are essential and useful? Syllabus Journal, 1(1), 1–12.

Garrison, D. R., Anderson, T., & Archer, W. (2003). A theory of critical inquiry in online distance learning. In M. Moore & W. Anderson (Eds.), Handbook of distance education (pp. 113–128). Lawrence Erlbaum.

Guo, P. J., Kim, J., & Rubin, R. (2014, Mar., 4–5). How video production affects student engagement: An empirical study of MOOC videos. The First Association for Computing Machinery Conference on Learning @ Scale Conference. ACM.

Horila, M., Nokelainen, P., Syvänen, A., & Överlund, J. (2002). Criteria for the pedagogical usability, version 1.0. Häme Polytechnic and University of Tampere.

Jahnke, I., Schmidt, M., Pham, M., & Singh, K. (2020). Sociotechnical-pedagogical usability for designing and evaluating learner experience in technology-enhanced environments. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books.

Jahnke, I. (2015). Digital didactical designs: Teaching and learning in CrossActionSpaces. Routledge.

Jahnke, I., Ritterskamp, C., Herrmann, T. (2005). Sociotechnical roles for sociotechnical systems—A perspective from social and computer sciences. In Proceedings of the AAAI Fall Symposium on Roles, an Interdisciplinary Perspective: Ontologies, Programming Languages, and Multiagent Systems. AAAI press.

Kaufmann, R., Sellnow, D. D., & Frisby, B. N. (2016). The development and validation of the online learning climate scale (OLCS). Communication Education, 65(3), 307–321.

Khajouei, R., Gohari, S. H., & Mirzaee, M. (2018). Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. Journal of Biomedical Informatics, 80, 37–42.

Lenzner, A., Schnotz, W., & Müller, A. (2013). The role of decorative pictures in learning. Instructional Science, 41(5), 811–831.

Lim, C. J., & Lee, S. (2007). Pedagogical usability checklist for ESL/EFL e-learning websites. Journal of Convergence Information Technology, 2(3), 67–76.

Martin, F., & Bolliger, D. U. (2018). Engagement matters: Student perceptions on the importance of engagement strategies in the online learning environment. Online Learning, 22(1), 205–222. doi:10.24059/olj.v22i1.1092

Mayer, R. E. (2002). Multimedia learning. Psychology of Learning and Motivation, 41, (pp. 85–139). Academic Press.

Moore, J. L., Dickson-Deane, C., & Liu, M. Z. (2014). Designing CMS courses from a pedagogical usability perspective. In A. Benson & A. Whitworth (Eds.), Research on course management systems in higher education (pp. 143–169). Information Age Publishing.

Nielsen, J. (1994). 10 heuristics for user interface design. Nielsen Norman Group.

Nielsen, J. (2004). Guidelines for visualizing links. Nielsen Norman Group.

Nielsen, J., & Loranger, H. (2006) Prioritizing web usability. Pearson Education.

Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Educational Technology & Society, 9(2), 178–197.

Obsidian Learning. (2017). Transforming learning: using video for cognitive, emotional, and social engagement.

Online Learning Consortium. (2018). OLC OSCQR course design review.

Orcutt, J. M., & Dringus, L. P. (2017). Beyond being there: Practices that establish presence, engage students and influence intellectual curiosity in a structured online learning environment. Online Learning, 21(3), 15–35. doi: 10.24059/olj.v%vi%i.1231

Preece, J. (2001). Sociability and usability in online communities:

Determining and measuring success. Behavior and Information Technology, 20(5), 347–356. doi: 10.1080/0144929011008468 3

Quality Matters (2018). Rubrics & Standards. QM.

Quinn, C. N. (1996). Pragmatic evaluation: lessons from usability. In A. Christie & B. Vaughan, (Eds.) Proceedings of Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 96) (pp. 437–444). University of South Australia.

Quiñones, D., Rusu, C., & Rusu, V. (2018). A methodology to develop usability/user experience heuristics. Computer Standards & Interfaces, 59, 109–129.

Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In M. Wild (Ed.), Computer education: New perspectives (pp. 219–246). Edith Cowan University.

Reeves, T. C., Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., & Loh, S. (2002, June, 24–29). Usability and instructional design heuristics for e-Learning evaluation. ED-Media 2002 World Conference on Educational Multimedia, Hypermedia & Telecommunications.

Robinson, H. A., Sheffield, A., Phillips, A. S., & Moore, M. (2017). Introduction to teaching online: Usability evaluation of interactivity in an online social constructivist course. TechTrends, 61(6), 533–540.

Safie, N. (2007). An investigation on the relationship between e-learning usability attributes towards motivation to learn. International Islamic University Malaysia.

Sauro, J. (2014) The relationship between problem frequency and problem severity in usability evaluations. Journal of Usability Studies, 10(1), 17–25.

Schade, A. (2014). Video usability. Nielsen Norman Group.

Schuler, D., & Namieka, A. (2017). Participatory design: Principles and practices. CRC Press.

Silius, K., & Tervakari, A-M. (2003, May, 8–9). An evaluation of the usefulness of web-based learning environments. The evaluation tool into the portal of Finnish virtual university. In V. Peñarrocha (Ed.), International Conference on Network Universities and E-Learning, Universidad Politécnica de Valencia.

Sims, R., Dobbs, G., & Hand, T. (2002). Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation. Distance Education, 23(2), 135–148.

Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11(5), 467–483.

Stein, J., & Graham, C. (2014). Essentials of online learning. Routledge.

Stone, D., Jarrett, C., Woodroffe, M., & Minocha, S. (2005). User interface design and evaluation. Morgan Kaufmann.

van der Meij, H., & van der Meij, J. (2013). Eight guidelines for the design of instructional videos for software training. Technical Communication, 60(3), 205–228.

van Merriënboer, J. J. G., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13.

Xavier University. (2018). Online learning heuristics.

Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. International Review of Research in Open and Distributed Learning, 16(2), 69–93.

Zhang, D., Zhou, L., Briggs, R. O., & Nunamaker, Jr., J. F. (2006). Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information and Management, 43, 15–27.


Copyright (c) 2021 Isa Jahnke, Nathan Riedel, Kanupriya Singh, Joi Moore

License URL: