Advancing Sociotechnical-Pedagogical Heuristics for the Usability Evaluation of Online Courses for Adult Learners
DOI:
https://doi.org/10.24059/olj.v25i4.2439Keywords:
Online Courses, Evaluation, Heuristics, Usability, Online Learning, AdultsAbstract
Online courses often include interface designs that do not support a positive learner experience. Literature shows a variety of heuristics to detect issues of online courses. While heuristic-based inspection of usability is a dominant method for evaluating digital systems, these methods cannot be easily transferred to online courses. To close this gap, we identified an initial set of social, technical, and pedagogical related items (STP) heuristics based on literature. Next, we analyzed this set using empirical data from two online courses. In total, we analyzed 195 problems with the goal to substantiate a final set of 14 STP heuristics. This new set allows for efficiently evaluating online courses by supporting evaluators and instructional designers in uncovering the most crucial issues and improving the learner experience. Finally, based on this work, we discuss a definition of learner experience for the emerging field of learner experience design and research.
References
Albion, P. (1999a). Heuristic evaluation of educational multimedia: From theory to practice. In J. Winn (Ed.), Proceedings of the 16th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 9–15). Australasian Society for Computers in Learning in Tertiary Education.
Baldwin, S. J., Ching, Y. H., & Friesen, N. (2018). Online course design and development among college and university instructors: An analysis using grounded theory. Online Learning, 22(2), 157–171. doi:10.24059/olj.v22i2.1212
Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, S., &
Reeves, T. C. (2002). Usability and instructional design heuristics for e-learning evaluation. In P. Barker & S. Rebelsky (Eds.), Proceedings from ED-MEDIA 2002: World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 1615–1621). AACE.
Bloom, B. (1956). Taxonomy of educational objectives handbook I: The cognitive domain. Addison Wesley Longman, Inc.
Botella, F., Rusu, C., Rusu, V., & Quiñones, D. (2018, Sept.). How novel evaluators perceive their first heuristic evaluation. Proceedings of XIX International Conference on Human Computer Interaction. Association for Computing Machinery. https://doi.org/10.1145/3233824.3233835
Boyle, T. (1997). Design for multimedia learning. Prentice Hall.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. https://doi.org/10.1191/1478088706qp063oa
Chao, T., Saj, T., & Tessier, F. (2006). Establishing a quality review for online courses. Educause Quarterly, 29(3), 32–39. https://er.educause.edu/articles/2006/1/establishing-a-quality-review-for-online-courses
Chen, B., Chang, Y. H., Ouyang, F., & Zhou, W. (2018). Fostering student engagement in online discussion through social learning analytics. The Internet and Higher Education, 37, 21–30.
Clement, F. J. (1985). Technology & instruction: The rule of seven revisited. Performance + Instruction, 24(2), 6–8. https://doi.org/10.1002/pfi.4150240203
Das, J. (2012). Role of syllabus in higher education: A critical study. Global Research Methodology Journal, 7, 1–2. http://www.grmgrlaranya.com/Journals/7th%20issue/Role%20of%20syllabus%20in%20higher%20education_%20A%20critical%20study.pdf
Demmans Epp, C., Phirangee, K., Hewitt, J., & Perfetti, C. A. (2020). Learning management system and course influences on student actions and learning experiences. Education Tech Research Dev, 68, 3263–3297. https://doi.org/10.1007/s11423-020-09821-1
Dringus, L. P., & Cohen, M. S. (2005, Oct. 19–22). An adaptable usability heuristic checklist for online courses. The Frontiers in Education 35th Annual Conference. IEEE.
Douglas, S. (2017, Dec. 6). 7 usability heuristics that all UI designers should know. UsabilityGeek. https://usabilitygeek.com/usability-heuristics-ui-designers-know/
Fink, S. B. (2012). The many purposes of course syllabi: Which are essential and useful? Syllabus Journal, 1(1), 1–12. https://www.pfw.edu/dotAsset/bdf64113-75c4-487a-ba68-dfcc3044ce9c.pdf
Garrison, D. R., Anderson, T., & Archer, W. (2003). A theory of critical inquiry in online distance learning. In M. Moore & W. Anderson (Eds.), Handbook of distance education (pp. 113–128). Lawrence Erlbaum.
Guo, P. J., Kim, J., & Rubin, R. (2014, Mar., 4–5). How video production affects student engagement: An empirical study of MOOC videos. The First Association for Computing Machinery Conference on Learning @ Scale Conference. ACM.
Horila, M., Nokelainen, P., Syvänen, A., & Överlund, J. (2002). Criteria for the pedagogical usability, version 1.0. Häme Polytechnic and University of Tampere.
Jahnke, I., Schmidt, M., Pham, M., & Singh, K. (2020). Sociotechnical-pedagogical usability for designing and evaluating learner experience in technology-enhanced environments. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/sociotechnical_pedagogical_usability
Jahnke, I. (2015). Digital didactical designs: Teaching and learning in CrossActionSpaces. Routledge.
Jahnke, I., Ritterskamp, C., Herrmann, T. (2005). Sociotechnical roles for sociotechnical systems—A perspective from social and computer sciences. In Proceedings of the AAAI Fall Symposium on Roles, an Interdisciplinary Perspective: Ontologies, Programming Languages, and Multiagent Systems. AAAI press.
Kaufmann, R., Sellnow, D. D., & Frisby, B. N. (2016). The development and validation of the online learning climate scale (OLCS). Communication Education, 65(3), 307–321.
Khajouei, R., Gohari, S. H., & Mirzaee, M. (2018). Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. Journal of Biomedical Informatics, 80, 37–42. https://doi.org/10.1016/j.jbi.2018.02.016
Lenzner, A., Schnotz, W., & Müller, A. (2013). The role of decorative pictures in learning. Instructional Science, 41(5), 811–831. https://doi.org/10.1007/s11251-012-9256-z
Lim, C. J., & Lee, S. (2007). Pedagogical usability checklist for ESL/EFL e-learning websites. Journal of Convergence Information Technology, 2(3), 67–76.
Martin, F., & Bolliger, D. U. (2018). Engagement matters: Student perceptions on the importance of engagement strategies in the online learning environment. Online Learning, 22(1), 205–222. doi:10.24059/olj.v22i1.1092
Mayer, R. E. (2002). Multimedia learning. Psychology of Learning and Motivation, 41, (pp. 85–139). Academic Press.
Moore, J. L., Dickson-Deane, C., & Liu, M. Z. (2014). Designing CMS courses from a pedagogical usability perspective. In A. Benson & A. Whitworth (Eds.), Research on course management systems in higher education (pp. 143–169). Information Age Publishing.
Nielsen, J. (1994). 10 heuristics for user interface design. Nielsen Norman Group. https://www.nngroup.com/articles/ten-usability-heuristics/
Nielsen, J. (2004). Guidelines for visualizing links. Nielsen Norman Group. https://www.nngroup.com/articles/guidelines-for-visualizing-links/
Nielsen, J., & Loranger, H. (2006) Prioritizing web usability. Pearson Education.
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Educational Technology & Society, 9(2), 178–197. https://www.semanticscholar.org/paper/An-empirical-assessment-of-pedagogical-usability-Nokelainen/ea96b628f440642d72026c14710a67ccd06f41f1#extracted
Obsidian Learning. (2017). Transforming learning: using video for cognitive, emotional, and social engagement. https://elearningindustry.com/free-ebooks/transforming-learning-using-video-for-cognitive-emotional-and-social-engagement
Online Learning Consortium. (2018). OLC OSCQR course design review. https://onlinelearningconsortium.org/consult/oscqr-course-design-review/
Orcutt, J. M., & Dringus, L. P. (2017). Beyond being there: Practices that establish presence, engage students and influence intellectual curiosity in a structured online learning environment. Online Learning, 21(3), 15–35. doi: 10.24059/olj.v%vi%i.1231
Preece, J. (2001). Sociability and usability in online communities:
Determining and measuring success. Behavior and Information Technology, 20(5), 347–356. doi: 10.1080/0144929011008468 3
Quality Matters (2018). Rubrics & Standards. QM. https://www.qualitymatters.org/qa-resources/rubric-standards
Quinn, C. N. (1996). Pragmatic evaluation: lessons from usability. In A. Christie & B. Vaughan, (Eds.) Proceedings of Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 96) (pp. 437–444). University of South Australia.
Quiñones, D., Rusu, C., & Rusu, V. (2018). A methodology to develop usability/user experience heuristics. Computer Standards & Interfaces, 59, 109–129. https://doi.org/10.1016/j.csi.2018.03.002
Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In M. Wild (Ed.), Computer education: New perspectives (pp. 219–246). Edith Cowan University.
Reeves, T. C., Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., & Loh, S. (2002, June, 24–29). Usability and instructional design heuristics for e-Learning evaluation. ED-Media 2002 World Conference on Educational Multimedia, Hypermedia & Telecommunications.
Robinson, H. A., Sheffield, A., Phillips, A. S., & Moore, M. (2017). Introduction to teaching online: Usability evaluation of interactivity in an online social constructivist course. TechTrends, 61(6), 533–540.
Safie, N. (2007). An investigation on the relationship between e-learning usability attributes towards motivation to learn. International Islamic University Malaysia.
Sauro, J. (2014) The relationship between problem frequency and problem severity in usability evaluations. Journal of Usability Studies, 10(1), 17–25.
Schade, A. (2014). Video usability. Nielsen Norman Group. https://www.nngroup.com/articles/video-usability/
Schuler, D., & Namieka, A. (2017). Participatory design: Principles and practices. CRC Press.
Silius, K., & Tervakari, A-M. (2003, May, 8–9). An evaluation of the usefulness of web-based learning environments. The evaluation tool into the portal of Finnish virtual university. In V. Peñarrocha (Ed.), International Conference on Network Universities and E-Learning, Universidad Politécnica de Valencia.
Sims, R., Dobbs, G., & Hand, T. (2002). Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation. Distance Education, 23(2), 135–148. https://doi.org/10.1080/0158791022000009169
Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11(5), 467–483.
Stein, J., & Graham, C. (2014). Essentials of online learning. Routledge.
Stone, D., Jarrett, C., Woodroffe, M., & Minocha, S. (2005). User interface design and evaluation. Morgan Kaufmann.
van der Meij, H., & van der Meij, J. (2013). Eight guidelines for the design of instructional videos for software training. Technical Communication, 60(3), 205–228. https://pdfs.semanticscholar.org/47a8/a0502d3ac8a636277b2ba48cba4e2aa14edd.pdf
van Merriënboer, J. J. G., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13. https://doi.org/10.1207/S15326985EP3801_2
Xavier University. (2018). Online learning heuristics. https://www.xavier.edu/id/online-blended-classes/
Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. International Review of Research in Open and Distributed Learning, 16(2), 69–93.
Zhang, D., Zhou, L., Briggs, R. O., & Nunamaker, Jr., J. F. (2006). Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information and Management, 43, 15–27. https://doi.org/10.1016/j.im.2005.01.004
Downloads
Published
Issue
Section
License
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
