Using LMS Log Data to Explore Student Engagement with Coursework Videos

Suzanne Maloney, Megan Axelsen, Linda Galligan, Joanna Turner, Petrea Redmond, Alice Brown, Marita Basson, Jill Lawrence

Abstract


Driven by the increased availability of Learning Management System data, this study explored its value and sought understanding of student behaviour through the information contained in activity level log data. Specifically, this study examined analytics data to understand students’ engagement with online videos. Learning analytics data from the MoodleTM and Vimeo® platforms were compared. The research also examined the impact of video length on engagement, and how engagement with videos changed over the course of a semester when multiple video resources were used in a course. The comparison in platform learning analytics showed differences in metrics thus offering a caution to users relying on unidimensional metrics. While the results support the notion that log data do provide educators with an opportunity for review, the time and expertise in extracting, handling, and using the data may stifle its widespread adoption.


Keywords


learning analytics, student engagement, LMS, higher education

Full Text:

PDF

References


Alfayez, Z. (2021). Designing Educational Videos for University Websites Based on Students’ Preferences. Online Learning, 25(2). doi:doi.org/10.24059/olj.v25i2.2232. https://olj.onlinelearningconsortium.org/index.php/olj/article/view/2232

Au, T., Li, S., & Ma, G. (2003). Applying and evaluating models to predict customer attrition using data mining techniques. Journal of Comparative International Management, 6(1), 10-22. https://journals.lib.unb.ca/index.php/JCIM/article/view/442/736

Axelsen, M., Redmond, P., Heinrich, E., & Henderson, M. (2020). The evolving field of learning analytics research in higher education: From data analysis to theory generation, an agenda for future research. Australasian Journal of Educational Technology, 36(2), 1-7. http://doi:10.14742/ajet.5510

Barua, P. D., Zhou, X., Gururajan, R., & Chan, K. C. (2018). Determination of factors influencing student engagement using a Learning Management System in a tertiary setting. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (pp. 604-609). Santiago: Institute of Electrical and Electronics Engineers (IEEE). https://www.semanticscholar.org/paper/Determination-of-Factors-Influencing-Student-Using-Barua-Zhou/06ffdc5b4b987ba39c3184c204886f6055b9c80d

Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. In Proceedings ASCILITE 2010 (pp. 75-86). Sydney: Australasian Society for Computers in Learning in Tertiary Education. https://ascilite.org/conferences/sydney10/procs/Beer-full.pdf

Bodily, R., Graham, C. R., & Bush, M. D. (2017). Online learner engagement: Opportunities and challenges with using data analytics. Educational Technology, 57(1), 10-18. http://www.jstor.com/stable/44430535

Bodily, R., Nyland, R., & Wiley, D. (2017). The RISE framework: Using learning analytics to automatically identify open educational resources for continuous improvement. International Review of Research in Open and Distributed Learning, 18(2), 103-122. http://www.irrodl.org/index.php/irrodl/article/view/2952

Bohan, J., & Stack, N. (2014). Do weekly online assignments promote student engagement and enhance learning in first-year psychology students? Psychology of Education Review, 38(2). Retrieved from https://shop.bps.org.uk/the-psychology-of-education-review-vol-38-no-2-autumn-2014

Brame, C. J. (2016). Effective educational videos: Principles and guidelines for maximizing student learning from video content. CBE Life Science Education, 15(4), 1-6. http://doi:10.1187/cbe.16-03-0125

Brozina, C., Knight, D. B., Kinoshita, T., & Johri, A. (2019). Engaged to succeed: Understanding first-year engineering students’ course engagement and performance through analytics. IEEE Access, 7, 163686-163699. http://doi:10.1109/ACCESS.2019.2945873

Bulathwela, S., Pérez-Ortiz, M., Lipani, A., Yilmaz, E., & Shawe-Taylor, J. (2020). Predicting engagement in video lectures. In A. N. Rafferty, J. Whitehill, C. Romero, & V. Cavalli-Sforza (Eds.), Proceedings of the 13th International Conference on Educational Data Mining (pp. 50-60). Montreal: International Educational Data Mining Society. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_62.pdf

Carmichael, M., Reid, A.-K., & Karpicke, J. D. (2018). Assessing the impact of educational video on student engagement, critical thinking and learning: The current state of play. A SAGE White Paper. SAGE Publishing. https://us.sagepub.com/sites/default/files/hevideolearning.pdf

Casey, K., & Azcona, D. (2017). Utilizing student activity patterns to predict performance. International Journal of Educational Technology in Higher Education, 14(4). http://doi:10.1186/s41239-017-0044-3

Dixson, M. D. (2015). Measuring student engagement in the online course: The Online Student Engagement Scale (OSE). Online Learning, 19(4). http://doi:10.24059/olj.v19i4.561

Fincham, E., Whitelock-Wainwright, A., Kovanović, V., Joksimović, S., van Staalduinen, J.-P., & Gašević, D. (2019). Counting clicks is not enough: Validating a theorized model of engagement in learning analytics. In LAK19: The 9th International Learning Analytics & Knowledge Conference (pp. 501-510). New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/3303772.3303775

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71. https://link.springer.com/content/pdf/10.1007/s11528-014-0822-x.pdf

Gašević, D., Mirriahi, N., Long, P., & Dawson, P. (2014). Editorial: Inaugural Issue of the Journal of Learning Analytics. Journal of Learning Analytics, 1(1), 1-2. https://learning-analytics.info/index.php/JLA/article/view/3910

Giannakos, M. N., Sampson, D. G., & Kidziński, Ł. (2016). Introduction to smart learning analytics: foundations and developments in video-based learning. Smart Learning Environments, 3:12. http://doi.org/10.1186/s40561-40016-40034-40562

Glance, D. G., Hugh, P., & Barrett, R. (2014, November 23-26). Attrition patterns amongst participant groups in Massive Open Online Courses. Paper presented at the ASCILITE 2014 (Australasian Society for Computers in Learning in Tertiary Education), Dunedin. https://www.ascilite.org/conferences/dunedin2014/files/fullpapers/16-Glance.pdf

Gómez-Aguilar, D. A., Hernández-García, A., García-Peñalvo, F. J., & Therón, R. (2015). Tap into visual analysis of customization of grouping of activities in elearning. Computers in Human Behavior, 47(June), 60-67. http://doi:10.1016/j.chb.2014.11.001

Greenland, S., & Moore, C. (2014). Patterns of online student enrolment and attrition in Australian open access online education: A preliminary case study. Open Praxis, 6(1), 45-54. https://www.learntechlib.org/p/130685/

Guo, P. J., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In L@S '14: Proceedings of the first ACM conference on Learning @ scale conference (pp. 41-50). Atlanta: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/2556325.2566239

Henrie, C., Bodily, R., Larsen, R., & Graham, C. R. (2018). Exploring the potential of LMS log data as a proxy measure of student engagement. Journal of Computing in Higher Education, 30, 344-362. http://doi:10.1007/s12528-017-9161-1

Henrie, C., Bodily, R., Manwaring, K. C., & Graham, C. R. (2015). Exploring Intensive Longitudinal Measures of Student Engagement in Blended Learning. International Review of Research in Open and Distributed Learning, 16(3), 133-155. http://www.irrodl.org/index.php/irrodl/article/view/2015/3386

Hochheimer, C. J., Sabo, R. T., Krist, A. H., Day, T., Cyrus, J., & Woolf, S. H. (2016). Methods for evaluating respondent attrition in web-based surveys. Journal of medical Internet research, 18(11), e301. http://doi:10.2196/jmir.6342

Ismail, S. N., Hamid, S., & Chiroma, H. (2019). The utilization of learning analytics to develop student engagement model in learning management system. Journal of Physics: Conference Series, 1339. http://doi:10.1088/1742-6596/1339/1/012096

Jordon, M. M., & Duckett, N. D. (2018). Universities confront ‘tech disruption’: Perceptions of student engagement online using two learning management systems. The Journal of Public and Professional Sociology, 10(1), Article 4. Retrieved from https://digitalcommons.kennesaw.edu/jpps/vol10/iss1/4

Karaksha, A., Grant, G., Anoopkumar-Dukie, S., Nirthanan, S. N., & Davey, A. K. (2013). Student engagement in pharmacology courses using online learning tools. American Journal of Pharmaceutical Education, 77(6), Article 125. http://doi:10.5688/ajpe776125

Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 170-179). New York: Association for Computing Machinery. https://web.stanford.edu/~cpiech/bio/papers/deconstructingDisengagement.pdf

Knestrick, J. M., Wilkinson, M. R., Pellathy, T. P., Lange-Kessler, J., Katz, R., & Compton, P. (2016). Predictors of retention of students in an online nurse practitioner program. The Journal for Nurse Practitioners, 12(9), 635-640. http://doi:10.1016/j.nurpra.2016.06.011

Li, N., Kidziński, Ł., Jermann, P., & Dillenbourg, P. (2015). MOOC video interaction patterns: What do they tell us? . In G. Conole, T. Klobučar, C. Rensing, J. Konert, & E. Lavoué (Eds.), Design for teaching and learning in a networked world. 10th European Conference on Technology Enhanced Learning (pp. 197-210). Cham: Springer International Publishing Switzerland. https://www.springerprofessional.de/en/mooc-video-interaction-patterns-what-do-they-tell-us/2540034

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an ‘‘early warning system’’ for educators: A proof of concept. Computers & Education, 54(2), 588-599. http://doi:10.1016/j.compedu.2009.09.008

Marks, A., Al-Ali, A., & Rietsema, K. (2016). Learning management systems: A shift toward learning and academic analytics. International Journal of Emerging Technologies in Learning, 11(4), 77-82. http://doi:10.3991/ijet.v11i04.5419

Pardo, A., Ellis, R. A., & Calvo, R. A. (2015). Combining observational and experiential data to inform the redesign of learning activities. In LAK '15: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 305-309). New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/2723576.2723625

Poon, L. K. M., Kong, S.-C., Yau, T. S. H., Wong, M., & Ling, M. H. (2017). Learning analytics for monitoring students participation online: Visualizing navigational patterns on learning management system. In S. K. S. Cheung, L.-f. Kwok, W. W. K. Ma, L.-K. Lee, & H. Yang (Eds.), Blended Learning. New Challenges and Innovative Practices. 10th International Conference on Blended Learning (pp. 166-176). Hong Kong: Springer International Publishing. https://repository.eduhk.hk/en/publications/learning-analytics-for-monitoring-students-participation-online-v-7

Rienties, B., Toetenel, L., & Bryan, A. (2015). "Scaling up” learning design: impact of learning design activities on LMS behavior and performance. In LAK '15: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 345-319). New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/2723576.2723600

Ruhanen, L., Whitford, M., & McLennan, C.-L. (2015). Indigenous tourism in Australia: Time for a reality check. Tourism Management, 48, 73-83. http://doi:10.1016/j.tourman.2014.10.017

Sheeran, P., & Webb, T. L. (2016). The intention–behavior gap. Social and Personality Psychology Compass, 10(9), 503-518. http://doi:10.1111/spc3.12265

Sherer, P., & Shea, T. (2011). Using online video to support student learning and engagement. College Teaching, 59(2), 56-59. http://doi:10.1080/87567555.2010.511313

Smith, D. (2010). Valuation of customer relationships: choice, application and results of various attrition analysis methodologies. Business Valuation Review, 29(2), 44-53. http://doi:10.5791/0897-1781-29.2.44

Stewart, M., Stott, T., & Nuttall, A.-M. (2011). Student engagement patterns over the duration of level 1 and level 3 geography modules: Influences on student attendance, performance and use of online resources. Journal of Geography in Higher Education, 35(1), 47-65. http://doi:10.1080/03098265.2010.498880

Venugopal, G., & Jain, R. (2015). Influence of learning management system on student engagement. In 2015 IEEE 3rd International Conference on MOOCs, Innovation and Technology in Education (MITE) (pp. 427-432). Amritsar: Institue of Electrical and Electronics Engineers (IEEE). http://toc.proceedings.com/29135webtoc.pdf

Vogt, K. L. (2016). Measuring student engagement using Learning Management Systems. (PhD). Ontario Institute for Studies in Education Leadership, Higher & Adult Education, University of Toronto, Toronto. http://hdl.handle.net/1807/73213

Vytasek, J. M., Patzak, A., & Winne, P. H. (2020). Analytics for student engagement. In M. Virvou, E. Alepis, G. A. Tsihrintzis, & L. C. Jain (Eds.), Machine learning paradigms. Advances in learning analytics (pp. 23-48). Cham: Springer Nature Switzerland.

Wang, Z., Bergin, C., & Bergin, D. A. (2014). Measuring engagement in fourth to twelfth grade classrooms: The classroom engagement inventory. School Psychology Quarterly, 29(4), 517-535. http://doi:10.1037/spq0000050

Williams, D., & Whiting, A. (2016). Exploring the relationship between student engagement, twitter, and a learning management system: A study of undergraduate marketing students. International Journal of Teaching and Learning in Higher Education, 28(3), 302-313. https://files.eric.ed.gov/fulltext/EJ1125099.pdf

Winstone, N., Bourne, J., Medland, E., Niculescu, I., & Rees, R. (2020). “Check the grade, log out”: students’ engagement with feedback in learning management systems. Assessment & Evaluation in Higher Education, 631-643. https://www.tandfonline.com/doi/full/10.1080/02602938.2020.1787331

Wu, S., Rizoiu, M.-A., & Xie, L. (2018, June 25-28). Beyond views: Measuring and predicting engagement in online videos. Paper presented at the The 12th International Association for the Advancement of Artificial Intelligence (AAAI) Conference on Web and Social Media (ICWSM-18), California. https://ojs.aaai.org/index.php/ICWSM/article/view/15031

Yang, D., Wen, M., & Rose, C. (2014, July 4-7). Peer Influence on attrition in Massive Open Online Courses. Paper presented at the The 7th International Conference on Educational Data Mining, London. http://www.cs.cmu.edu/~mwen/papers/edm2014-39.pdf

Yukselturk, E., Ozekes, S., & Türel, Y. K. (2014). Predicting dropout student: An application of data mining methods in an online education program. European Journal of Open, Distance and E-Learning, 17(1), 118-133. https://sciendo.com/article/10.2478/eurodl-2014-0008




DOI: http://dx.doi.org/10.24059/olj.v26i4.2998



Copyright (c) 2022 Suzanne Maloney, Megan Axelsen, Linda Galligan, Joanna Turner, Petrea Redmond, Alice Brown, Marita Basson, Jill Lawrence

License URL: https://creativecommons.org/licenses/by/4.0/