Using LMS Log Data to Explore Student Engagement with Coursework Videos

Driven by the increased availability of Learning Management System data, this study explored its value and sought understanding of student behaviour through the information contained in activity level log data. Specifically, this study examined analytics data to understand students’ engagement with online videos. Learning analytics data from the Moodle TM and Vimeo® platforms were compared. The research also examined the impact of video length on engagement, and how engagement with videos changed over the course of a semester when multiple video resources were used in a course. The comparison in platform learning analytics showed differences in metrics thus offering a caution to users relying on unidimensional metrics. While the results support the notion that log data do provide educators with an opportunity for review, the time and expertise in extracting, handling, and using the data may stifle its widespread adoption.

One of the most significant debates in higher education today is the motivation and use of Leaning Analytics (LA).On the one hand, the almost ubiquitous use of Learning Management Systems (e.g., Brightspace® by D2L, Moodle TM , and Blackboard®), to deliver online learning content results in a data harvesting opportunity that could be used to inform teaching and learning.After all, a lack of knowledge about the ways that students interact with learning materials has been identified (Marks, et al, 2016).On the other hand, the fear of misuse of LA for the purpose of institutional surveillance and control gives cause to question the danger of its use and its value in ensuring inclusive educational spaces for learning (Selwyn, 2020;Green, 2018;Keyes, 2019).Caught in the middle are academics navigating new institutional expectations while bringing to bear their own optimism (or pessimism) regarding what the new tool has to offer.A scan of education journals reporting on its use signals that, generally, academics are buoyed by LA claims that, through the collection and analysis of the digital records of students and their interactions with various computer systems, they could better understand and optimise student learning and the environments in which it occurs (Axelsen, et al., , 2020;Marks et al., 2016).A good example is Harindranathan and Folkestad (2019) who demonstrate a meaningful use of LA data in an unsupervised, technology-enhanced platform.However, academics are cautious of the reported time and expertise needed to successfully make use of what LA has the potential to offer (Shibani, et al, 2020).A review of the literature by Panigraphi et al. (2018) indicate the wide spectrum of platforms used across the global e-learning sector.
Despite this debate, pressure on academics to evidence ongoing improvement continues.In online higher education, this proof of improvement relies on student data related to assessment performance and online content use, and confirmation that a feedback loop was created for the improvement of curriculum and the way online learning resources are designed.Kollom et al. (2021) indicate that academics do recognise the possibilities to influence the learning landscape using LA but warn of their reluctance to act on such data, especially with respect to at risk students.While some learning analytics research has focused on using analytics to evaluate courses to improve design (Pardo et al, 2015;Rienties et al,, 2015), little research has focused on continuous improvement of content using learning analytics at the activity (or course resource) level (Bodily et al,, 2017).In relation to the data generated by LMS, log data from these systems are often available for extraction, making these systems a potential source of activity-level data to study student learning using LA.Log data are a record of a user's activity within a system, including click or page view counts, time spent on a given action, keyboard strokes, results of an activity (such as performance on a quiz), and counts of any other activity that may occur within a system.Log data are an activity-level measure, capturing real-time changes in user interactions with the online learning system (Henrie, et al., 2018).
Despite the potential use of LMS data to inform online educational practices and design, researchers have suggested that further work is needed to understand how the value and meaning of LMS log data may best be applied to understanding, informing, and optimising teaching and learning practice (Henrie et al., 2018;Poon et al., 2017).To contribute to discussions in this area, the aims of the research reported in this paper were twofold: (1) examine the value and sufficiency of LMS log data for measuring student engagement at the activity-level, and (2) examine how log data may be used to better understand student engagement at the activity-level to thus inform and optimize how certain course resources/activities are designed.
Within the literature, educational video recordings are widely recognised as an effective pedagogical strategy to support student learning (Noetel et al., 2021;Brame, 2016).Given this, data were collected and analysed in relation to students' access to, and engagement with, coursespecific videos (short video recordings developed and designed to support a particular learning focus within a course).Thus, to LMS log data at the activity-level, the "activity" examined was the way students engaged with the course videos available to learners.
Student engagement is often linked to the time, energy, and effort students dedicate to their learning and learning community (Bond, et al, 2020;Krause, 2005).This study accepted that student engagement is not static, can change during the semester, and occurs along a continuum (Muir et al., 2019).To this end and for the purposes of this study, student engagement is defined as the active choice to access, load, and view course videos as captured by LMS data.
To explore the value and sufficiency of LMS data (aim 1), data on student engagement with, or access to, the course videos were captured using two platforms: the LMS and Vimeo® (a video hosting platform).Examining the types of data available on the two platforms and considering how this data differed in relation to measuring student engagement with the video resources enabled the authors to assess the value and sufficiency of LMS data for this purpose.
To understand how log data may be used to better understand student engagement at the activitylevel and thus inform the design of course resources/activities (aim 2), the more detailed log data collected through Vimeo® analytics were analysed to examine how students engaged with/accessed the video resources.The purpose was to explore the usefulness of LMS data to further our understanding of how students engage with short video content.Relevant to this, although not the focus of this research, is what makes an instructional video effective for student learning.Video effectiveness may affect student engagement and what is of interest in this study is whether the LMS data can be of value in assessing this.For coverage of what makes an instructional video effective there is extensive research (Brame, 2016;Carmichael et al., 2018;Guo et al., 2014;Sherer & Shea, 2011), and for a discussion of the student experience see Alfayez (2021).The focus here was on the value of existing LMS analytic data to a course teaching team.The need for research to understand this is a precursor for fully embracing the potential of smart learning analytics (Giannakos et al., 2016).
Using Log Data to Analyse Engagement Across various educational settings, student engagement has long been viewed as a factor that drives learning and predicts academic, social, and emotional learning outcomes (Fincham, et al., 2019), while lack of engagement has been identified as a contributor to lower completion rates in online learning courses (Kizilcec et al., 2013).Although student engagement is important to any learning experience, it is particularly relevant to technology-mediated learning (Henrie et al., 2018).Knowing what promotes or discourages engagement in technology-mediated learning is therefore important for ensuring that online learning resources are designed to keep students connected with the course and their learning (Dixson, 2015).
The metrics used to measure student engagement in online learning environments broadly align with those used in more traditional classroom settings.These include the time spent on course activities and use of resources (e.g., viewing pages, completing quizzes and assignments), course attendance (or number of logins), the accuracy and completion rate on quizzes and assignments, social interactions, and artifacts produced by learners (Fincham et al., 2019;Vytasek et al., 2020).The literature on engagement and learning analytics have primarily sought to examine student engagement through unidimensional quantitative data metrics, such as discussion forum participation, watching video lectures, completing course assessments, and number of time resources or e-learning tools were used (e.g.Dixson, 2015;Karaksha et al., 2013;Li et al , 2015;Stewartet al., 2011;Vytasek et al., 2020).For student engagement with educational videos, watch time-or the median of normalized engagement time (i.e., the percentage of watch time from the total video)-has been the main measure for quantifying engagement in the literature (Bulathwela et al. , 2020;Guo et al., 2014;Wu et al., 2018).
Learning management systems accumulate vast volumes of data on student behaviour that can be used to inform and improve online student engagement and, as such, a growing body of research has explored the value of LMS log data (Beer et al., 2010;Brozinaet al., 2019;Casey & Azcona, 2017;Gašević, Dawson, & Siemens et al., 2015;Gašević, Mirriahi, Long, & Dawson, 2014;Henrie et al., 2018;Ismail et al., 2019;Macfadyen & Dawson, 2010).While much of this attention has focused on the relationship between log data (i.e., frequency of student LMS use, such as logins, discussion board use, resources used, etc.) and academic performance, there is increasing interest in the relationship between log data and student engagement outcomes.Researchers have examined, for example, the influence of LMS on student engagement (Venugopal & Jain, 2015;Williams & Whiting, 2016), the effects of LMS interface, design, and functionality on online student engagement (Barua et al., 2018;Jordon & Duckett, 2018), differences between students' perceived level of engagement in LMS and their actual online behaviour (Vogt, 2016), and students' engagement with feedback in LMS (Winstone et al., 2020).
When it comes to using LMS log data (such as click counts or number of views) to measure student engagement, one problem is that such measures do not necessarily capture whether learners consume the material (Bulathwela et al., 2020).It has also been argued that student engagement differs greatly from popularity measures such as number of views and cannot be captured by unidimensional metrics because such measures do not necessarily measure the same thing (Fincham et al., 2019).Finding meaningful ways to represent the quantum and quality of engagement in online environments is a current challenge, and as such, research into the development of a reliable model of using click data to measure student engagement is ongoing (Bodily et al., 2017;Henrie et al., 2015;Vytasek et al., 2020).
The level at which engagement is being investigated also has implications for how engagement is conceptualized, operationalized, and measured.Due to the time-intensive nature of collecting engagement data by conventional survey tools, engagement has often been measured at the course level rather than the activity level, therefore limiting its usefulness for making activity-specific interventions that are based on the findings (Bodily, Graham, et al., 2017).Indeed, engagement needs to be measured at the same specificity level as the intervention (Wang, et al., 2014).Thus, if the interest is in better understanding and informing pedagogical practice related to a specific type of online learning resource, such as the use of online videos to support student learning, the most appropriate level of engagement for this focus would be the activity level, where measures focus on students' engagement in specific learning activities (Henrie et al., 2018).
As online learning increasingly moves towards becoming the primary format where students access tertiary education, it is important that the significant volume of data generated is meaningfully utilised by educators to optimise learning experiences.There are, however, ongoing problems with transforming the data into useful information to improve current learning environments.It is a task for which many educators feel they are insufficiently qualified and possess inadequate time to make good use of the data (Poon et al., 2017).Data are also often only available in limited, general formats, such as text rich files or basic data visualization charts, and the LMS regularly do not collect the types of data that are needed for real-time analysis and reporting (Bodily, R., Graham, C. R., & Bush, M. D., 2017;Gómez-Aguilar, et al., 2015).Consequently, the raw data are often not meaningful for educators to diagnose, analyse or predict the usage situation in the LMS.Without effective processing, the large amount of data generated by the system may also lead to information overload for educators who, in turn, do not know what to do with the information, thus further contributing to discouraging assessment of data value (Bodily, R., Graham, C. R., & Bush, M. D., 2017;Poon et al., 2017).
To optimise the learning environment, research needs to focus on developing ways to help educators not only retrieve data about learning processes and relationships between learning agents, but to also help transform the log data gathered from LMS into actionable information.Indeed, few studies have fully exploited the learning data of students from a digital environment such as LMS (Poon et al., 2017), thus reflecting the limited usage of these data to inform teaching and learning.As argued by Bulathwela et al. (2020), a well-designed learning resource should enable the learner to achieve the expected learning outcomes.Research that helps educators design more informed, targeted resources will thus enable the optimisation of learning and the environments in which it occurs.

Methodology
This quantitative study explores the patterns of student engagement with online videos and the differences between the logs provided by Moodle TM and Vimeo®, a video hosting and sharing site.Moodle TM was the LM platform of the institution and the Vimeo® platform was selected as it was an easily available platform, contained the required analytics, offered password protection and was within the budget of the research project.The use of Moodle TM and Vimeo® are commonly available e-learning or video hosting platforms.During 2020, course teams from seven courses developed short videos for their students as part of their course content.The design and development of the videos were in the control of the course team with the stated purpose that the video was to link course theory to real-life practice.This gave a common theme for the videos across all courses while ensuring freedom in design appropriate to each discipline.The course team decided on the number, the duration, and the content of each video to ensure it was appropriate for the targeted course.However, all course team leaders were made aware of the guidelines of effective video design that was developed from previous research (Brame, 2016;Carmichael et al, 2018;Guo et al., 2014;Sherer & Shea, 2011).Courses in which these videos were used, and therefore in which LMS log data were collected, were in the fields of education, accounting, nursing, engineering, and physics.Data about how students accessed and interacted with the course video resources were collected across two semesters and altogether, the data for 77 videos were collected and analysed.Table 1 contains a summary of the courses.The study received ethics approval at the university where the study took place.
The videos were hosted in Vimeo® and were provided to students via hyperlink from the LMS.In the LMS, when a student clicked on a video link, the click was registered as evidence of the student having accessed the resource.Data from the LMS therefore provided cumulative totals of the percentage of students that are enrolled in the course who had accessed (or clicked on) the video link.In Vimeo®, the way the student then interacted with that video was recorded.Data collected in the Vimeo® platform included "loads,", "plays," "finishes" and average percentage of video watched.A load was counted each time the video loaded on any page.A play was registered anytime someone started to play the video.In the instance when a video is played multiple times by the same viewer without a page refresh in Vimeo®, only a single play is counted.This includes scenarios in which viewers click the play button several times, scrub back to the beginning of the timeline, or loop the video.A finish occurred when a viewer watched a video through to the very end.Of these measures (load, play, finish), a "play" in Vimeo® is most like a "click" in the LMS.Both indicate that a student had accessed a video resource, however neither provided any more information about how the student interacted with the video (i.e., did they watch the video after clicking on it).It is also worth noting that while the LMS is only able to record "clicks" that occur within its platform, Vimeo® is able to count "clicks" (loads/plays) across all platforms.
The LMS used in this study was Moodle TM .The data used in this study were obtained from the LMS system retrospectively via an algorithm written specifically for this study and enabled the log (click) data for every resource on a course LMS to be downloaded and displayed in a spreadsheet.The algorithm was written by an IT expert to extract the data required.The data are indicative of click counts only; that is, on any given date, the spreadsheet shows the total percentage of students who have clicked on a resource up to that date.Figure 1 shows how this data are displayed in the spreadsheet: the number of students enrolled in the course on any given day is provided (row 2) and the percentage of students who have accessed the resources are provided, based on current enrolments (columns BC-BT).The learning analytics data from Vimeo® were also downloaded and displayed in a spreadsheet.This allowed the comparison required to explore the value of LMS data for the purpose of research aim 1, and to explore how log data may be used to better understand student engagement at the activity-level for the purpose of research aim 2.

Findings Aim 1: To Explore the Value and Sufficiency of LMS Data
To examine the value and sufficiency of LMS log data for measuring student engagement at the activity-level, log data from the LMS were compared with the log data from Vimeo® to consider whether the LMS log data provide an accurate picture of how students engage with/access course resources at the activity-level.Specifically, click data from the LMS were compared to both plays and finishes on Vimeo®.Figure 2 maps total LMS clicks and total Vimeo® plays per video for semesters 1 and 2; and Figure 3 maps total LMS clicks and total Vimeo® finishes per video for each of the two semesters.In all the graphs, percentages over 100 indicate that some of the students clicked on or played the video multiple times.

Figure 2 Percentage of Vimeo® "Plays" and LMS "Clicks" for Semester 1 (top) and Semester 2 (bottom)
A comparison of the "clicks" in the LMS against the "plays" in Vimeo® -arguably the most similar measurement to a "click" in the LMS-shows that in some cases the Vimeo® plays are higher than the LMS clicks and in other cases this is reversed.On average, across both semesters, Vimeo® recorded a higher number of plays per video compared to LMS clicks (48 videos recorded higher plays on Vimeo®, while 29 recorded higher click counts on the LMS).Where the Vimeo® plays were higher than the LMS click counts, suggests students watching the video multiple times in Vimeo® without necessarily clicking through to it each time from the LMS.Where the LMS click counts were higher than Vimeo® plays, suggests students clicking on the link that was provided to them in the LMS and then choosing not to play the video once it had loaded.Either way, the differences, which were quite large in some cases (e.g., NUR V2, semester 1) show that the log data being captured in the LMS do not provide a complete picture of how students are accessing video resources.

Figure 3
Vimeo® "Finishes" and LMS "Clicks" for Semester 1 (top) and Semester 2 (bottom) A comparison of the "clicks" in the LMS and the "finishes" in Vimeo® further illustrate the problem with relying on click data alone to make assumptions about student engagement.As illustrated in Figure 3, Vimeo® finishes were lower-and in many cases much lower-than LMS clicks in all except one of the videos (in PHY V8 finishes were higher in S2).This shows that while students may have clicked on the link to the video in the LMS and therefore are captured as having accessed the video according to LMS analytics, only some of those students go on to watch the video.Even then, there are questions as to whether students actively engage with the video, or whether they simply have it playing in the background while multitasking (Bulathwela et al., 2020;Guo et al., 2014).Interestingly, a comparison across videos indicates that videos showing an industry professional were less likely to be accessed then those that did not.This was the case within a discipline (for example, education) and between disciplines (for example, urban & regional planning compared to nursing).Also, a comparison across disciplines indicates some difference.For example, the physics videos were not accessed to the same extent as the accounting and nursing videos, but they were much more likely to record a finish.However, this observation needs to be appraised considering the length of the video and the number of videos contained in a course which, as discussed below, was shown to affect the likelihood of student access.

Aim 2: To Understand Student Engagement at the Activity Level Through Log Data
To examine how log data may be used to better understand student engagement at the activity level, log data from Vimeo® were analysed to explore in more detail how students interacted with the video resources.Specifically, the log data were analysed to: compare "loads," "plays," and "finishes"; examine the impact of the video length on engagement; and examine how engagement with videos changed over the course of a semester when multiple video resources were used in the course.Gaining a more in depth understanding of how students engage at the activity (or resource) level with course video resources is important for helping to inform the design of such videos and therefore assist academics to optimise how these resources are used to support students in their coursework.

Does load, play, and finishes analytic data improve our understanding of student behaviour?
Comparing loads, plays, and finishes can provide some insight into students' patterns of behaviour in relation to engagement with course video resources.The number of times a video is loaded ("loads") arguably provides some indication of students' intention to watch (engage with) the video.By clicking on the link to the video, students have taken the first step towards engagement behaviour.After the video has loaded, students who then initiate a "play," and who therefore start to watch the video, are taking the next step towards engaging with that video, thus moving from intention to actual engagement behaviour.Students who then go on to watch the entire video (at least according to log data) are those that have arguably bridged the intentionbehaviour gap (Sheeran & Webb, 2016) to engage with the video resources more fully.

Figure 4
Comparison of Loads, Plays, and Finishes for Total Data Across Both Semesters.
Figure 4 compares "loads," "plays," and "finishes" for all the videos used in the intervention across the two semesters.This figure highlights a definite pattern which shows a drop in the percentage of students who click on the video to load it, compared to the number who initiate a play, compared to the number who then watch that video to completion. Figure 5 compares "plays" against "loads" and "finishes" against "plays" to show the proportion of times a play was initiated for a video out of the number of times the video was loaded, and the proportion of times the video was played until the end out of the number of times it was started.On average, across all the videos for both semesters, 36% of students who clicked the video link to load the video then initiated a play.Of those who did initiate a play, 43% watched the video to the end.

Figure 5 Ratio of Plays: Loads and Finishes: Plays for Semester 1 (top) and Semester 2 (bottom).
The drop-off that occurs between the percentage of students who load the video by clicking on its link (indicated by number of loads) to the number of students who then initiate a play and go onto watch the video in its entirety can be mapped on an attrition curve.In other disciplines such as medicine/health, finance, commerce, economics, and management, attrition models have been used to measure such factors as the loss of clients, customers, or participants over time (e.g.Au et al., 2003;Hochheimer et al., 2016;Ruhanen, et al., 2015;Smith, 2010).In online education, attrition models have been used to measure, for example, retention, attrition and participation in MOOC activities, open access online learning, and online education programs, (e.g.Glance et al., 2014;Greenland & Moore, 2014;Knestrick et al., 2016;Yang et al., 2014;Yukselturk et al., 2014).As the concept of attrition refers to the gradual reduction in size of a variable (such as customers, or in this case the number of students accessing a particular course resource), the attrition curve will usually slope downwards from left to right; that is, it has a negative association.Such a curve can also be used to map student engagement at the resource/activity-level. Figure 6 maps the attrition curve of student engagement with the coursework videos in this study.It highlights the rapid decline that occurs between students displaying an intention to engage with course video resources, as indicated by the fact they clicked on the video link and the video loaded on their computer/device, and their subsequent actual engagement behaviour, as indicated by their behaviour of initiating a play.This rapid decline between the choice to load the video and the choice to watch the video indeed highlights a problem for educators who create and use the videos as part of their course content.It also poses the question-Why do so many students chose not to watch the video once it has loaded on their screen?Perhaps the thumbnail of the video that loads is not appealing or interesting enough to warrant watching.Would students be more likely to initiate a play if the first impression of the video was more appealing or interesting?Were the thumbnails used in these courses not enticing for a student to explore the video?The results suggest that the first impression of a video via its thumbnail is of considerable importance to its eventual use by students.

Does video length matter?
Research has found that in relation to instructional videos, shorter videos are more engaging (Brame, 2016;Carmichael et al., 2018;Guo et al., 2014).Guo et al. (2014), for example, found that student engagement with such videos drops off after about six minutes (360 seconds).In this current study, the videos used ranged in length from 1.5 minutes (85 seconds) to almost 23 minutes (1380 seconds).The impact of video length/duration on engagement was explored in two ways: first, finishes per play-the proportion of times a video was played until the end, out of the number of times it was started-was mapped against video duration; and second, the average percentage of the video that was watched before students clicked off was mapped against video duration.The trend lines in both graphs in Figure 7 show that as video duration increases, the propensity for students to watch the video to the end declines.Indeed, a higher proportion of students are more likely to watch the video to the end when the video is shorter in duration.These findings support the findings of previous research.On average, students watched 74% of the video before they clicked off.It is interesting to note that even for the shorter videos, students tended to click off after watching only 70-80% of the video.Is this because students felt they had watched enough to understand the idea the video was trying to convey?Was it because they felt the video was in its "wrap up" phase?If such a "wrap up" was simply repeating information already raised in the video, then perhaps it was perceived as having no more value to students as they already had the information.Certainly, in some of the videos used in this study, the instructor did spend some time at the end providing a "wrap up" to the video.In considering how to optimise how video resources are designed and produced, this finding suggests that such a "wrap up" is not needed at the end of the video because students will click off anyway.This is something worth exploring to inform how video content can be designed, developed, and produced in a manner that will make the video more engaging for students and to thus ensure instructors optimise video content that students will access in the limited time before their attention wanes.

Does student behaviour change over time during the semester?
As well as exploring the effect of video duration on engagement, how students engaged with videos changed over the course of a semester when multiple video resources were used in the course was also examined.Five of the courses provided students with multiple instructional videos.These videos became available across the semester as part of the course content.Figure 8 shows the percentage of students enrolled in each course who initiated a play for each video.In most of these courses, students' propensity to watch the videos declined over time (the semester) when multiple instructional videos were used to provide course content.
While it is difficult to draw conclusions about the data because they are based on examples from only five courses, the decrease in propensity to watch the videos may suggest that there is such a thing as having "too many" instructional videos as part of a course's content.This would also suggest that although instructional videos may be useful for increasing students' engagement in a course, having too many videos that are too similar may indeed have the opposite effect and students may start to become disengaged from this type of resource.

Summary of Findings
The comparative analysis of learning The comparative analysis of learning analytics data across two platforms showed a difference in the record of the number of clicks, plays and views.This simple comparison of specific resources (videos) across two platforms for several different courses over two teaching periods illustrates the care required in planning, executing, and harvesting LA data.Further, the effort required to extract the LA data required an IT expert to apply a specifically written algorithm.The time and expertise needed would be beyond most staff and thus curtail the widespread adoption of LA data use in a business-as-usual, sustainable way.

Plays Trendline
Percentage of total cohort who accessed the video A trendline has been added to show that engagement with the videos decreased over the semester.Also, note in the Education D course, students were given a 'nudge' (a personal communication) encouraging them to access video 4.This nudge resulted in an increase of students accessing the resource.There is a decline in the number of students who load, then play and then finish watching a video.This is modelled on an attrition curve.Approximately a third of those who "loaded" then "played," and just under half of those "finished."This suggests that effort directed to increasing the "loads" to "plays" would more likely give the greater return then effort directed to increasing the "plays" to "finishes."This effort could be directed to ensuring an enticing thumbnail.Further, the analysis showed that students click-off at about the 70-80% point regardless of video length suggesting that energy directed at the last 20% of the video to encourage a "finish" may not yield higher uptake of "finishes."Longer videos decreased the likelihood of a "finish."Additionally, the number of videos in a course over the same semester reduced the propensity of students initiating a play for each video over the course of the semester.

Discussion
As much time and effort is needed to design and produce course video resources, it is important for educators to be able to seek multiple avenues of assessing how the video was received by students and this would include the use of LA data.This therefore helps educators to assess how well the resource is being utilised (i.e., how many students accessed the video resource and how much of it they watched) and to thus refine the resource if required based on the student access and engagement behaviours.Such analysis and any related refining of the resource will indeed help ensure the optimisation of its design and use.It is not to say that videos with low uptake are not valuable.Indeed, for some students it may provide the much-needed link to the course and what may be needed in such a case is to direct the right students to the video in the first place.While click (or "plays" or "views") data have been used to measure whether students are accessing a resource, such measures are not a reliable measure.As shown in this paper, the two different interfaces used to collect student log data related to whether students clicked on a video resource often recorded quite different click counts.This alone shows that "clicks", "plays," or "views" are not reliable as a measure of student access to a resource, let alone as a measurement of student engagement.Indeed, while click data (or "plays" or "view" of a video) may show that students are clicking on the resource, this data do not show how the student then went on to engage with the resource.This research was limited to two platforms, Moodle TM and Vimeo®.Future research could extend the findings to examine any "platform" effect across the multiple platforms available.
More recently, in relation to how students access course video resources, LMS have started to make available such measures as average percentage of video watched.As shown in this paper, more detailed log data measures, including how many students finish watching a video compared to how many starts watching the video, the average percentage of a video watched before students click off, and the ability to compare loads to plays to finishes are all important in gaining a more in depth understanding of how students interact-or engage-with course video resources.
This finding through the simple comparison of two platforms signals a need to ensure educators (academics and university management) are clear about what they are measuring, why they are measuring it and how it will influence future learning resources and features.Importantly, it should be considered what the LA did not capture, and what simply cannot be captured by the LA.The contrary view put forward by Selwyn (2020) is pertinent here to ensure that generalised macro data, while relevant and useful, are considered and applied in the light of its limitations.This sense of the data not capturing the full story may be one factor that Kollom et al. (2021) was referring to when they found that academics are not wanting to be compelled to act on LA data.
Further to the value and sufficiency of LA data, was the ease of data extraction.There were limits as to how the system users were able to access their course data, which complicated the data analysis and data reporting.As the data were not able to be easily accessed retrospectively, the result was that some portions of the software needed to be modified and an algorithm formulated to ensure that the necessary data were available for analysis.This experience supports existing research that have signaled the time and expertise needed for academics to fully utilize the LA data (Munguia et al., 2020;Kollom, et al., 2021).Shibani, et al. (2020) found that time was a significant factor for integration of LA into teaching.This signals a need for institutions to provide resources in the form of expertise and extra time for academics if usefulness and scalability is to be achieved at any meaningful level.Whilst future technological advancements may improve accessibility, the need to provide suitable expertise and time to analyze and prescribe changes necessary for improvement would remain.
The results also highlight the rapid decline between the choice to load the and to watch the video.It is probable that changing a thumbnail could lead to changes in student behaviour when selecting a video to view.Even a small increase in the number of plays initiated compared to loads would lead to an increase in the number of "finishes."Inspecting Figure 6 shows that effort needs to be focused on converting "loads" to "plays" -or intention to behaviour-because even a small increase in the percentage of students initiating a play could consequently lead to an increase in the percentage of students then engaging with the video resource more fully and potentially watching it to the end, and thus flattening the attrition curve.Future research could explore strategies for converting students' intention to view the video (loads) to actual engagement with the video (plays, leading into finishes).The first impression of a video (based on its thumbnail) could be one factor to explore how to increase a students' propensity to engage.Other factors worth exploring could include the number of videos in a course or program, whether the video contains an industry professional, and differences across program and course level.Additionally, future research could explore the characteristics of those students more or less likely to access and play a video.
The length of the video influenced engagement as did the quantum of videos contained in a single course over a semester.This suggests that a "whole of semester" design approach is needed when seeking to engage students.While this study was concerned with the activity level of analysis, future research could investigate the "whole of semester" student engagement via the analysis of all activity log data.Future research could also explore the influence of activity level across several courses in a program undertaken by a student cohort in the same semester.It may be that a course heavily reliant on video resources may affect the use of resources in courses undertaken by students contemporaneously.
A limitation of this research is that it was limited to one university.Future research could involve other universities that also use Moodle TM as their LMS or to compare other LMS data.A qualitative study would also complement this quantitative study by revealing why students stop watching, why they would start watching and what they do in between.Despite the limitations this study improves our understanding of the value of analytics data and how it can be used to inform educators of student behaviour and thus activity choice.

Conclusion
The aims of this study were to explore the value and sufficiency of LMS data and to seek understanding of student behaviour through the information contained in the log data at the activity level.The analysis of the log data revealed the limited sufficiency of LMS data when compared to another platform.The results also showed a negative attrition curve as the level of commitment to the resource increased.This supports previous research examining attrition models in online education and other disciplines (Glance et al., 2014;Greenland & Moore, 2014;Knestrick et al., Yang, et al., 2014;Yukselturk et al.,, 2014).Comparing loads, plays, and finishes on an attrition curve, for example, can provide insight into student behaviours and possibly provide insight into both how engaging a video might look to students, as well as how engaging it then is.For example, if "loads" significantly exceed subsequent "plays" then this perhaps highlights a problem with the video or its thumbnail (i.e., it does not look interesting enough to students, so they choose not to play it); if "plays" significantly exceed "finishes" then perhaps the content is not engaging or relevant.In both cases, the significant decline between loads then plays or plays and finishes could imply there is a potential problem with the video resource and thus indicate that the resource needs to be modified to better meet student needs.Indeed, such data and the insights it provides in relation to student engagement at the activity/resource level are important for informing both the design of course resources and for optimising their use in the course.

Availability of Data and Materials
The data generated during the current study are available from the corresponding author on reasonable request.

Declaration Competing interests
The authors declare that they have no competing interests.

Ethics
Ethics approval for this work was granted by the University of Southern Queensland.

Figure 1
Figure 1 Example from the Accounting Course of the Spreadsheet Display of LMS Log Data

Figure 6
Figure 6Average Attrition with Respect to the Data for Loads, Plays and Finishes.

Figure 7
Figure 7The Relationship BetweenDuration and Finishes (top)  andPercent of Video Viewed (bottom)

Table 1
Courses and Number of Videos