Using LMS Log Data to Explore Student Engagement with Coursework Videos
DOI:
https://doi.org/10.24059/olj.v26i4.2998Keywords:
learning analytics, student engagement, LMS, higher educationAbstract
Driven by the increased availability of Learning Management System data, this study explored its value and sought understanding of student behaviour through the information contained in activity level log data. Specifically, this study examined analytics data to understand students’ engagement with online videos. Learning analytics data from the MoodleTM and Vimeo® platforms were compared. The research also examined the impact of video length on engagement, and how engagement with videos changed over the course of a semester when multiple video resources were used in a course. The comparison in platform learning analytics showed differences in metrics thus offering a caution to users relying on unidimensional metrics. While the results support the notion that log data do provide educators with an opportunity for review, the time and expertise in extracting, handling, and using the data may stifle its widespread adoption.
References
Alfayez, Z. (2021). Designing Educational Videos for University Websites Based on Students’ Preferences. Online Learning, 25(2). doi:doi.org/10.24059/olj.v25i2.2232. https://olj.onlinelearningconsortium.org/index.php/olj/article/view/2232
Au, T., Li, S., & Ma, G. (2003). Applying and evaluating models to predict customer attrition using data mining techniques. Journal of Comparative International Management, 6(1), 10-22. https://journals.lib.unb.ca/index.php/JCIM/article/view/442/736
Axelsen, M., Redmond, P., Heinrich, E., & Henderson, M. (2020). The evolving field of learning analytics research in higher education: From data analysis to theory generation, an agenda for future research. Australasian Journal of Educational Technology, 36(2), 1-7. http://doi:10.14742/ajet.5510
Barua, P. D., Zhou, X., Gururajan, R., & Chan, K. C. (2018). Determination of factors influencing student engagement using a Learning Management System in a tertiary setting. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (pp. 604-609). Santiago: Institute of Electrical and Electronics Engineers (IEEE). https://www.semanticscholar.org/paper/Determination-of-Factors-Influencing-Student-Using-Barua-Zhou/06ffdc5b4b987ba39c3184c204886f6055b9c80d
Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. In Proceedings ASCILITE 2010 (pp. 75-86). Sydney: Australasian Society for Computers in Learning in Tertiary Education. https://ascilite.org/conferences/sydney10/procs/Beer-full.pdf
Bodily, R., Graham, C. R., & Bush, M. D. (2017). Online learner engagement: Opportunities and challenges with using data analytics. Educational Technology, 57(1), 10-18. http://www.jstor.com/stable/44430535
Bodily, R., Nyland, R., & Wiley, D. (2017). The RISE framework: Using learning analytics to automatically identify open educational resources for continuous improvement. International Review of Research in Open and Distributed Learning, 18(2), 103-122. http://www.irrodl.org/index.php/irrodl/article/view/2952
Bohan, J., & Stack, N. (2014). Do weekly online assignments promote student engagement and enhance learning in first-year psychology students? Psychology of Education Review, 38(2). Retrieved from https://shop.bps.org.uk/the-psychology-of-education-review-vol-38-no-2-autumn-2014
Brame, C. J. (2016). Effective educational videos: Principles and guidelines for maximizing student learning from video content. CBE Life Science Education, 15(4), 1-6. http://doi:10.1187/cbe.16-03-0125
Brozina, C., Knight, D. B., Kinoshita, T., & Johri, A. (2019). Engaged to succeed: Understanding first-year engineering students’ course engagement and performance through analytics. IEEE Access, 7, 163686-163699. http://doi:10.1109/ACCESS.2019.2945873
Bulathwela, S., Pérez-Ortiz, M., Lipani, A., Yilmaz, E., & Shawe-Taylor, J. (2020). Predicting engagement in video lectures. In A. N. Rafferty, J. Whitehill, C. Romero, & V. Cavalli-Sforza (Eds.), Proceedings of the 13th International Conference on Educational Data Mining (pp. 50-60). Montreal: International Educational Data Mining Society. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_62.pdf
Carmichael, M., Reid, A.-K., & Karpicke, J. D. (2018). Assessing the impact of educational video on student engagement, critical thinking and learning: The current state of play. A SAGE White Paper. SAGE Publishing. https://us.sagepub.com/sites/default/files/hevideolearning.pdf
Casey, K., & Azcona, D. (2017). Utilizing student activity patterns to predict performance. International Journal of Educational Technology in Higher Education, 14(4). http://doi:10.1186/s41239-017-0044-3
Dixson, M. D. (2015). Measuring student engagement in the online course: The Online Student Engagement Scale (OSE). Online Learning, 19(4). http://doi:10.24059/olj.v19i4.561
Fincham, E., Whitelock-Wainwright, A., Kovanović, V., Joksimović, S., van Staalduinen, J.-P., & Gašević, D. (2019). Counting clicks is not enough: Validating a theorized model of engagement in learning analytics. In LAK19: The 9th International Learning Analytics & Knowledge Conference (pp. 501-510). New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/3303772.3303775
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71. https://link.springer.com/content/pdf/10.1007/s11528-014-0822-x.pdf
Gašević, D., Mirriahi, N., Long, P., & Dawson, P. (2014). Editorial: Inaugural Issue of the Journal of Learning Analytics. Journal of Learning Analytics, 1(1), 1-2. https://learning-analytics.info/index.php/JLA/article/view/3910
Giannakos, M. N., Sampson, D. G., & Kidziński, Ł. (2016). Introduction to smart learning analytics: foundations and developments in video-based learning. Smart Learning Environments, 3:12. http://doi.org/10.1186/s40561-40016-40034-40562
Glance, D. G., Hugh, P., & Barrett, R. (2014, November 23-26). Attrition patterns amongst participant groups in Massive Open Online Courses. Paper presented at the ASCILITE 2014 (Australasian Society for Computers in Learning in Tertiary Education), Dunedin. https://www.ascilite.org/conferences/dunedin2014/files/fullpapers/16-Glance.pdf
Gómez-Aguilar, D. A., Hernández-García, A., García-Peñalvo, F. J., & Therón, R. (2015). Tap into visual analysis of customization of grouping of activities in elearning. Computers in Human Behavior, 47(June), 60-67. http://doi:10.1016/j.chb.2014.11.001
Greenland, S., & Moore, C. (2014). Patterns of online student enrolment and attrition in Australian open access online education: A preliminary case study. Open Praxis, 6(1), 45-54. https://www.learntechlib.org/p/130685/
Guo, P. J., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In L@S '14: Proceedings of the first ACM conference on Learning @ scale conference (pp. 41-50). Atlanta: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/2556325.2566239
Henrie, C., Bodily, R., Larsen, R., & Graham, C. R. (2018). Exploring the potential of LMS log data as a proxy measure of student engagement. Journal of Computing in Higher Education, 30, 344-362. http://doi:10.1007/s12528-017-9161-1
Henrie, C., Bodily, R., Manwaring, K. C., & Graham, C. R. (2015). Exploring Intensive Longitudinal Measures of Student Engagement in Blended Learning. International Review of Research in Open and Distributed Learning, 16(3), 133-155. http://www.irrodl.org/index.php/irrodl/article/view/2015/3386
Hochheimer, C. J., Sabo, R. T., Krist, A. H., Day, T., Cyrus, J., & Woolf, S. H. (2016). Methods for evaluating respondent attrition in web-based surveys. Journal of medical Internet research, 18(11), e301. http://doi:10.2196/jmir.6342
Ismail, S. N., Hamid, S., & Chiroma, H. (2019). The utilization of learning analytics to develop student engagement model in learning management system. Journal of Physics: Conference Series, 1339. http://doi:10.1088/1742-6596/1339/1/012096
Jordon, M. M., & Duckett, N. D. (2018). Universities confront ‘tech disruption’: Perceptions of student engagement online using two learning management systems. The Journal of Public and Professional Sociology, 10(1), Article 4. Retrieved from https://digitalcommons.kennesaw.edu/jpps/vol10/iss1/4
Karaksha, A., Grant, G., Anoopkumar-Dukie, S., Nirthanan, S. N., & Davey, A. K. (2013). Student engagement in pharmacology courses using online learning tools. American Journal of Pharmaceutical Education, 77(6), Article 125. http://doi:10.5688/ajpe776125
Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 170-179). New York: Association for Computing Machinery. https://web.stanford.edu/~cpiech/bio/papers/deconstructingDisengagement.pdf
Knestrick, J. M., Wilkinson, M. R., Pellathy, T. P., Lange-Kessler, J., Katz, R., & Compton, P. (2016). Predictors of retention of students in an online nurse practitioner program. The Journal for Nurse Practitioners, 12(9), 635-640. http://doi:10.1016/j.nurpra.2016.06.011
Li, N., Kidziński, Ł., Jermann, P., & Dillenbourg, P. (2015). MOOC video interaction patterns: What do they tell us? . In G. Conole, T. Klobučar, C. Rensing, J. Konert, & E. Lavoué (Eds.), Design for teaching and learning in a networked world. 10th European Conference on Technology Enhanced Learning (pp. 197-210). Cham: Springer International Publishing Switzerland. https://www.springerprofessional.de/en/mooc-video-interaction-patterns-what-do-they-tell-us/2540034
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an ‘‘early warning system’’ for educators: A proof of concept. Computers & Education, 54(2), 588-599. http://doi:10.1016/j.compedu.2009.09.008
Marks, A., Al-Ali, A., & Rietsema, K. (2016). Learning management systems: A shift toward learning and academic analytics. International Journal of Emerging Technologies in Learning, 11(4), 77-82. http://doi:10.3991/ijet.v11i04.5419
Pardo, A., Ellis, R. A., & Calvo, R. A. (2015). Combining observational and experiential data to inform the redesign of learning activities. In LAK '15: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 305-309). New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/2723576.2723625
Poon, L. K. M., Kong, S.-C., Yau, T. S. H., Wong, M., & Ling, M. H. (2017). Learning analytics for monitoring students participation online: Visualizing navigational patterns on learning management system. In S. K. S. Cheung, L.-f. Kwok, W. W. K. Ma, L.-K. Lee, & H. Yang (Eds.), Blended Learning. New Challenges and Innovative Practices. 10th International Conference on Blended Learning (pp. 166-176). Hong Kong: Springer International Publishing. https://repository.eduhk.hk/en/publications/learning-analytics-for-monitoring-students-participation-online-v-7
Rienties, B., Toetenel, L., & Bryan, A. (2015). "Scaling up” learning design: impact of learning design activities on LMS behavior and performance. In LAK '15: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 345-319). New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/2723576.2723600
Ruhanen, L., Whitford, M., & McLennan, C.-L. (2015). Indigenous tourism in Australia: Time for a reality check. Tourism Management, 48, 73-83. http://doi:10.1016/j.tourman.2014.10.017
Sheeran, P., & Webb, T. L. (2016). The intention–behavior gap. Social and Personality Psychology Compass, 10(9), 503-518. http://doi:10.1111/spc3.12265
Sherer, P., & Shea, T. (2011). Using online video to support student learning and engagement. College Teaching, 59(2), 56-59. http://doi:10.1080/87567555.2010.511313
Smith, D. (2010). Valuation of customer relationships: choice, application and results of various attrition analysis methodologies. Business Valuation Review, 29(2), 44-53. http://doi:10.5791/0897-1781-29.2.44
Stewart, M., Stott, T., & Nuttall, A.-M. (2011). Student engagement patterns over the duration of level 1 and level 3 geography modules: Influences on student attendance, performance and use of online resources. Journal of Geography in Higher Education, 35(1), 47-65. http://doi:10.1080/03098265.2010.498880
Venugopal, G., & Jain, R. (2015). Influence of learning management system on student engagement. In 2015 IEEE 3rd International Conference on MOOCs, Innovation and Technology in Education (MITE) (pp. 427-432). Amritsar: Institue of Electrical and Electronics Engineers (IEEE). http://toc.proceedings.com/29135webtoc.pdf
Vogt, K. L. (2016). Measuring student engagement using Learning Management Systems. (PhD). Ontario Institute for Studies in Education Leadership, Higher & Adult Education, University of Toronto, Toronto. http://hdl.handle.net/1807/73213
Vytasek, J. M., Patzak, A., & Winne, P. H. (2020). Analytics for student engagement. In M. Virvou, E. Alepis, G. A. Tsihrintzis, & L. C. Jain (Eds.), Machine learning paradigms. Advances in learning analytics (pp. 23-48). Cham: Springer Nature Switzerland.
Wang, Z., Bergin, C., & Bergin, D. A. (2014). Measuring engagement in fourth to twelfth grade classrooms: The classroom engagement inventory. School Psychology Quarterly, 29(4), 517-535. http://doi:10.1037/spq0000050
Williams, D., & Whiting, A. (2016). Exploring the relationship between student engagement, twitter, and a learning management system: A study of undergraduate marketing students. International Journal of Teaching and Learning in Higher Education, 28(3), 302-313. https://files.eric.ed.gov/fulltext/EJ1125099.pdf
Winstone, N., Bourne, J., Medland, E., Niculescu, I., & Rees, R. (2020). “Check the grade, log out”: students’ engagement with feedback in learning management systems. Assessment & Evaluation in Higher Education, 631-643. https://www.tandfonline.com/doi/full/10.1080/02602938.2020.1787331
Wu, S., Rizoiu, M.-A., & Xie, L. (2018, June 25-28). Beyond views: Measuring and predicting engagement in online videos. Paper presented at the The 12th International Association for the Advancement of Artificial Intelligence (AAAI) Conference on Web and Social Media (ICWSM-18), California. https://ojs.aaai.org/index.php/ICWSM/article/view/15031
Yang, D., Wen, M., & Rose, C. (2014, July 4-7). Peer Influence on attrition in Massive Open Online Courses. Paper presented at the The 7th International Conference on Educational Data Mining, London. http://www.cs.cmu.edu/~mwen/papers/edm2014-39.pdf
Yukselturk, E., Ozekes, S., & Türel, Y. K. (2014). Predicting dropout student: An application of data mining methods in an online education program. European Journal of Open, Distance and E-Learning, 17(1), 118-133. https://sciendo.com/article/10.2478/eurodl-2014-0008
Downloads
Published
Issue
Section
License
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
