Learning Analytics to Inform the Learning Design: Supporting Instructor’s Inquiry Into Student Learning in Unsupervised Technology-Enhanced Platforms

Instructors may design and implement formative assessments on technology-enhanced platforms (e

. Learning analytics to inform the learning design: Supporting instructor's inquiry into student learning in unsupervised technologyenhanced platforms.Online Learning, 23(3), 34-55. doi:10.24059/olj.v23i3.2057Learning Analytics to Inform the Learning Design: Supporting Instructor's Inquiry Into Student Learning in Unsupervised Technology-Enhanced Platforms Effective learning strategies are defined as the study approaches that are linked to superior learning and subsequent performance of learners.Examples include study strategies like active retrieval of information via self-testing and distributed or spaced practice of information (Bjork, Dunlosky, & Kornell, 2013).Robust evidence from cognitive psychology literature confirms that learning strategies like active retrieval of information and spaced practice enhance long-term retention when compared to rereading or massed practice of study materials (Carpenter, Pashler, & Cepeda, 2009;Karpicke & Smith, 2012;McDaniel, Agarwal, Huelser, McDermott, & Roediger, 2011;McDaniel, Thomas, Agarwal, Mcdermott, & Roediger, 2013;McDaniel, Wildman, & Anderson, 2012).According to the Institute of Education Sciences (IES), part of the U.S. Department of Education, the above-mentioned effective learning strategies improve learning among all students, in particular struggling learners, irrespective of grade or subject (Pashler et al., 2007).
The metacognitive awareness among learners assumes special importance in higher education, where students have to take an autonomous and active role in learning outside classrooms, such as self-directed environments where there is less guidance from instructors (Bjork et al., 2013;McMahon, 2002).Such settings refer to the unsupervised use of technologyenhanced learning platforms, such as online testing and learning tools.Studies affirm the value of the use of technology-enhanced platforms, like a learning management system (LMS), in conducting self-paced, learner-centered activities outside the classroom (Al-Busaidi, 2013;Chou, Peng, & Chang, 2010;Dias & Diniz, 2014;Islam, 2013;Nguyen, 2017;Wang, 2017;Zhang, Zhao, Zhou, & Nunamaker, 2004).For example, pedagogical tools like low-stakes quizzes can be effectively delivered via LMSs as learning designs which provide structure and opportunities for repeated practice and self-monitoring among learners (Angus & Watson, 2009;Coates, James, & Baldwin, 2005;Doige, 2012;O'Sullivan & Hargaden, 2014).Angus and Watson (2009) point out that certain formative aspects of assessments, like an opportunity for multiple attempts; timely formative feedback, which facilitates the development of mastery goal orientation and selfreflection among learners; and randomized questions could be attainable only in the online format.However, in unsupervised technology-enhanced platforms, instructors may lack access to students' actual learning behaviors and, hence, may not be successful in implementing timely interventions aimed to encourage productive learning behaviors.

Framework for the Study
The conceptual framework chosen for this study is the influential conversational framework proposed by Laurillard (2002), which suggests that interaction and feedback between instructors and students play a key role in enhancing student learning.The following paragraph explores in detail the role which instructors can play in this regard.
Instructors can play a two-pronged role in encouraging productive learning behaviors among students.They can encourage productive learning behaviors through mindful design and implementation of formative assessments (Knight & Sydney, 2018;Wise & Shaffer, 2015).More importantly, they can monitor learning behaviors and intervene by timely and meaningful feedback to support metacognitive awareness among students (Black & Wiliam, 2009;Black & Wiliam, 1998a;Govaerts, Verbert, Duval, & Pardo, 2012;Kaendler, Wiedmann, Rummel, & Spada, 2015).The role of instructors assumes importance in relation to the way assessments are conducted in the classrooms.Traditionally, when assessments are used to gauge students' learning and assign grades, only the final performances are considered as learning outcomes.In this case, instructors usually provide feedback only about the accuracy of the assigned task's outcome.The feedback that focuses on task accuracy may provide minimal guidance to the learners to monitor their learning (Butler & Winne, 1995).Alternatively, formative assessments implemented by the instructor can act as a guide to improve the learning process as well as future instruction (Baleni, 2015;Black & Wiliam, 1998a, 1998b;Leahy, Lyon, Thompson, & Wiliam, 2005;McTighe & O'Connor, 2005).This is because formative assessments provide instructors with ongoing information about learner behaviors and allow instructors to provide timely feedback to encourage productive learning behaviors and alter unproductive ones.

Importance of Interlinking Learning Analytics and Learning Design
As discussed in the previous sections, instructors may design and implement formative assessments intended to improve student learning on technology-enhanced platforms.However, in unsupervised technology-enhanced platforms, instructors remain unaware of students' activities and behavior patterns.Hence, they may not be able to provide students with feedback aimed to encourage the use of effective learning strategies.Possessing an understanding of learner behaviors with respect to the implemented learning design may be a prerequisite for providing meaningful feedback to students (Lockyer, Heathcote, & Dawson, 2013;van Leeuwen, 2015).Learning design is defined as the pedagogical intent and sequencing of an instructional technique (Lockyer et al., 2013).Studies confirm that evaluating the correctness of learners' solutions may be an easy task, while it could be more challenging to evaluate the quality of their learning strategies (Roll et al., 2014).Data gathered from technology-enhanced learning platforms, related to students' activities on those platforms, are required to understand how students interact with the system (Roll, Aleven, McLaren, & Koedinger, 2007).
Learning analytics (LA) allow instructors to access actual student behavioral data, especially when learning happens in unsupervised technology-enhanced learning platforms.LA is defined as "the process of collecting and studying usage data in order to make instructional decisions that will support student success" (Becker, 2013, p. 63).Instructors may need access to student behavior data to evaluate the effectiveness of the implemented pedagogical designs (Dyckhoff, Zielke, Bültmann, Chatti, & Schroeder, 2012;Lockyer, Heathcote, & Dawson, 2013).When instructors have access to students' learning behaviors, they may make pedagogic changes soon enough to impact practice, including modification of the existing instructional design to encourage productive learning behaviors.This cyclical design process is represented in Figure 1.LA has been increasingly used to support learning and teaching.After the implementation of learning designs, LA may help instructors understand the extent to which requirements of the design are met by their students (Kennedy et al., 2014).In short, the data available to instructors that are related to student behaviors allow instructors to reflect on student learning, provide learners with meaningful feedback, and refine the implemented learning design (Kennedy et al., 2014).Despite the potential of LA studies to provide instructors with real-time data related to student behaviors while learning is ongoing, only a few empirical studies explore how LA can support instructors' inquiry into student learning (van Leeuwen, 2015).Several analytic tools are available that collect and analyze data related to student engagement with technology-enhanced platforms (Arnold, 2010;Bakharia & Dawson, 2011;Kuosa et al., 2016;McKay, Miller, & Tritz, 2012;Silius, Tervakari, & Kailanto, 2013).But one of the major limitations of the existing tools is that they do not take into account the implemented pedagogic design, which may primarily determine how students engage with the learning platforms (Kennedy et al., 2014).Interconnecting learning design with the data collected from technology-enhanced learning tools by means of LA remains a largely unexplored area (Lockyer & Dawson, 2012).This limits the effective use of analytic data in meaningful ways.
The following case study from an undergraduate general microbiology class at Colorado State University investigates how meaningful information related to students' learning behaviors with respect to the learning design can be obtained via LA so that instructors can use such information for course-based improvements.

Methods
The following sections provide the context of the implemented learning design and details of the research design and study.

The Implemented Learning Design
This section explains the details of the implemented learning design in the undergraduate microbiology class: Microbiology, Immunology, and Pathology (MIP henceforth) within the Department of Microbiology, Immunology & Pathology at Colorado State University.The instructors of MIP had set up online quizzing on Canvas, the LMS at Colorado State University.The motivation behind the design and implementation of this learning design is the finding that quizzes or tests that require students to actively recall information promote learning and help them remember the information for longer periods.This phenomenon, demonstrated in controlled experiments as well as experimental studies in classrooms, is known as the "testing effect."Similarly, distributing the practice time into multiple sessions is demonstrated to be more effective than massing all the study sessions close together.This finding is referred to as the "spacing effect."In short, the online quizzes were designed based on the benefits of active retrieval and spaced practice of information on long-term retention of information.
The quizzes were to be attempted by the students unsupervised at their own pace and convenience (timing and location of quiz taking was the students' choice).The students were encouraged to watch an instructional video as a prerequisite to taking the online quizzes.This video briefly summarized the benefits of active recall and spaced retrieval on long-term retention and advised the students to learn the material in advance, not use their class notes while attempting the quizzes, and actively retrieve the information required to complete the problem via distributed practice over multiple sessions.The intent of the video was to encourage students to behave in ways that were beneficial for their learning.
Students could attempt the quizzes up to 10 times, as the intention was to promote learning among students rather than test their current knowledge (i.e., use quizzes as a learning tool rather than merely as an evaluative tool).They could retake the quizzes multiple times in order to achieve mastery of the topic and earn the highest score.The highest score achieved was kept in the Canvas gradebook.Each quiz was open for 9 days.The quizzes were low-stakes, contributing to less than 10% of the final grades.Every attempt of the quiz had a set of 10 random questions allotted from a question bank.The timeline sequence of the online quizzes and subsequent summative examinations are shown in Figure 2. As illustrated, each quiz was immediately followed by the relevant summative examination (Quiz 1 by Exam 1, Quiz 2 by Exam 2, Quiz 3 by Exam 3, and Quizzes 4 and 5 by the final comprehensive exam).This quiz structure allowed students to practice content before being tested on the relevant summative exam.In short, this design was implemented as a structure that provided students an opportunity to take part in a flexible, yet focused learning activity.

Problem Statement
The self-administered nature of the quizzes and limited data-reporting options available on the existing Canvas dashboard presented two problems to the MIP instructors.First, the actual quiz-taking behaviors of students were not available to us.Hence, postimplementation of the learning design, we did not possess information regarding whether the actual student behaviors accorded with the pedagogical intent of the implemented design.For example, did the students attempt the quizzes just before the deadlines only to secure credit?Did the students distribute the multiple possible attempts over the period in which the quizzes remained open, or did they mass all attempts together?Second, we were unable to provide meaningful and timely external feedback to students regarding their quiz-taking behaviors and use of strategies.Instead, we provided each student with a comparison of their individual scores with the class averages on exams (or comparison to the student's own scores in the earlier quizzes) and/or motivational feedback, such as "you are progressing well in this course" or "you need to put in more effort."However, motivational feedback is of limited value and may not help students understand deficits in their learning behaviors.

Need for Quiz-Log Analytics
The broad research question identified for the study was the following: Are learners' behaviors aligned to the pedagogical intent of the instructor's implemented learning design?To answer this question, the analysis of quiz logs collected from Canvas was considered necessary for the following reasons.
Providing students with formative feedback about their actual learning behaviors could encourage them to metacognitively monitor their behaviors and regulate their learning better.Students often may not be reliable monitors of their learning strategies and can overestimate the use of a specific tactic (Butler & Winne, 1995;Winne & Jamieson-Noel, 2002;Winne et al., 2002).Therefore, self-reports about the use of learning strategies may be inaccurate and unreliable.A more accurate report of students' learning strategies can be obtained by the analysis of Canvas quiz-log data.
Present-day technology-enhanced platforms log large volumes of metadata related to student activities in these platforms.But the dashboards of these platforms typically have built-in monitoring features that report only limited data.The remaining logged data are unavailable and incomprehensible to instructors, making it difficult to understand students' behavioral patterns.Usually, the information presented in dashboards of LMSs is simple metrics of students' frequency of interaction, such as the first and last login, messages the student has read and posted in discussion threads, number of downloads of study materials, number of pages visited, and scores achieved in assessments (Mazza & Dimitrova, 2007;Bueckle & Börner, 2017).These frequency measures may not provide instructors with meaningful insights into learner behaviors since these do not sufficiently capture student engagement and are not directly correlated to learning.For example, a higher number of logins does not guarantee that a student is more engaged in learning.Instructors may need access to variables closely related to students' learning behaviors to provide meaningful formative feedback, which could act as pointers to alter misguided learning strategies.We assume such variables can be extracted from the available log data.
The two specific research questions identified in the study, related to the quiz-log analysis, were the following: • RQ1: What variables related to students' productive learning behaviors can be identified from Canvas quiz logs?• RQ 2: Are there associations between the identified variables related to productive learning behaviors and exam scores?

Data Collection
Data from Canvas can be collected for data mining at many levels of granularity, ranging from course level to events or actions level (related to each quiz submission).The nature of the problem determines the choice of data collection, which implies that the collected data have to align with the research questions under consideration (Romero & Ventura, 2013).The specific research questions in this study demanded data collection of the quiz-log data from Canvas at the events/actions level.
Canvas data is stored in a "star schema" convention, where information is stored as a relational schema of facts and dimensions tables.Fact tables are designed at a low level of detail (or granularity), which implies events can be recorded at a very fine granularity.Dimension tables contain attributes which describe the fact data.
A set of predefined routines and protocols called application programming interface (API) was used to access data from Canvas.Data collection spanned three main tables in Canvasnamely, the user (has attributes of the user/student), quiz submissions (contains details regarding the last submitted quiz), and submissions (has attributes related to the latest submission of a quiz).The data schema of the three tables, including the column names, description of stored data, and interrelationship between the tables can be found in Figure 3.

Data Preprocessing
LMSs log traces of data related to students' learning activities.However, since these platforms are not designed for data mining, data are not stored in structured and systematic ways.Also, not all LMS log data are stored in the same format.Therefore, educational data mining tasks may require time-consuming data preprocessing for cleaning up the data and modifying it to appropriate forms (Bienkowski, Feng, & Means, 2012;Krüger, Merceron, & Wolf, 2010;Merceron & Yacef, 2008).Additionally, when data collection occurs at granular levels related to the implemented learning design, analysis and interpretation of student interactions become increasingly complex (Kennedy et al., 2014).The following sections describe the data preprocessing tasks and extraction of variables related to learner behaviors from Canvas quiz logs.We provide this detail to emphasize the importance of choosing data structures and formats which represent the event under consideration while converting the available semistructured data into a structured format.
The responses to the API calls which were used for data collection were in JavaScript Object Notation (JSON) format.For each quiz, all events related to a single quiz submission were retrieved.Since a student could make multiple attempts for a given quiz, events related to each attempt were to be considered to have a complete overview of student quiz-taking behaviors.The summary of each attempt in the quiz-log data consisted of quiz submission events which had information including the quiz ID, Canvas ID (Canvas creates an alternate ID, labeled the "Canvas ID," for all students corresponding to their university enrolment ID) of the student, and quiz submission events, such as the current number of attempt of a given quiz, remaining number of possible attempts, day of the attempt, the start time and end time for the current attempt, the score for the attempt, and an indication of off-task activity (i.e., the number of times the student has left the active page during the attempt under consideration).
The retrieved events were available in a format where an event related to the submission of each attempt of a quiz corresponded to the respective quizzes.It was necessary to reorganize the raw data to a format wherein events corresponded to the individual student.Later, this would allow easy traversal through the data set and retrieval of the required information corresponding to each student.Data was modified to the desired format using Python scripts.The hierarchical, nested structure of the heterogeneous data after this reorganization is shown in Figure 4.

Design and Extraction of Variables Related to Learner Behaviors From Canvas Quiz Logs
There is no pre-identified standard set of variables in the literature which meaningfully capture learners' behaviors with respect to the varying pedagogical intent and design of the different learning activities planned (Macfadyen & Dawson, 2010;Saqr, Fors, & Tedre, 2017).Variable selection depends on the context of the course, learning design, learning environment, and purpose of the study (Agudo-Peregrina, Iglesias-Pradas, Conde-González, & Hernández-García, 2014; Gašević, Dawson, Rogers, & Gasevic, 2016;Rienties, Boroowa, Cross, Kubiak, & Mayles, 2016).Hence, mindful design and extraction of variables are required to translate the logged data into meaningful indicators of students' effort.The following paragraphs explain the process of variables identification from Canvas quiz logs, which could provide meaningful information related to students' learning behaviors as they interacted with the learning design implemented in the MIP course.
We explored the Canvas quiz-log data to design and build variables which reflect the selfregulated quiz-taking behaviors of students with the unsupervised online quizzing platform.Exploration of the quiz-log data in relation to learners' choice of study strategies, including spaced practice and massed practice (placing all study attempts close together instead of distributing them), was conducted to extract and build variables.Emphasis was given to the design of meaningful variables associated with productive learning behaviors, such as students' focus on the task, as indicated by the quality of time spent online, the spacing of study events, procrastination behavior, and the number of attempts of quizzes (indicative of more practice).As was explained in the background, the rationale behind the choice of these variables was the consistent findings from previous studies, which show that effective learning strategies, such as repeated practice, distributed practice, and quality time spent on learning, usually lead to higher performance (de Freitas et al., 2015;Hung & Zhang, 2008;Macfadyen & Dawson, 2010).We only considered actionable variables-that is, those that would potentially provide hints to the instructors on how to effect changes in learners' behaviors or make decisions related to future design and implementation of quizzes.Adding static, nonmalleable variables, like demographic information or prior performance of the participants, could limit the possibility of providing the students with personalized and targeted formative feedback (Hung et al., 2017;Tempelaar, Rienties, & Giesbers, 2015).

Participants
Quiz logs pertaining to each quiz submission were collected from Canvas for 133 students taking MIP during the fall semester of 2017.To protect student identity, we intentionally did not collect demographic identifiers on individual students, as differences among students were not a focus of our research questions.However, MIP students typically are in their third year (junior year) at the university.

Results
To answer RQ 1, an exploration of Canvas quiz-log data was done, which led to the identification of five variables related to student learning behaviors.A few of these variables are directly logged in quiz logs, while the rest of the relevant variables had to be derived by manipulating the available data.The details about the design and extraction of each of these variables from the Canvas quiz logs are discussed in detail below.
Total time spent.Carroll's Time-on-Task hypothesis, one of the pioneering works investigating the relationship between students' classroom behavior and learning, hypothesizes that a student who spends more time engaging with the study materials has greater opportunities to learn (Carroll, 1989).This hypothesis implies that off-task activities (behaviors not related to learning, including disengaging from the study material) reduce learning.Based on this hypothesis, it was important to track time spent on task and off task for each student.
The variable "total time spent" was calculated as a measure of the aggregate time a student spent on all attempts at an individual quiz.The attribute "time spent," which is recorded in the Canvas log data for each submission of a quiz attempt, was extracted.Total time spent was obtained by adding up the time a student spent across all quiz attempts.This measure of total time spent was approached with caution, as it may have included the time spent off task by the students as well.Hence, to have a more accurate measure of time spent on task, a new variable called "offtask behavior" was built, as explained below.
Off-task behavior.Canvas logs an event type related to the student's off-task behavior for each quiz submission event, logged when the current Web browser tab becomes inactive for a long duration during the quizzing activity.For example, this event could occur when a student leaves the online quizzing system within Canvas and engages in off-task activities, such as browsing other tabs or temporarily leaving the system.This feature logged by Canvas was considered appropriate as a measure of off-task behavior after preliminary exploration of quiz-log data and parallel experiments conducted to establish the validity of this event type as a measure of off-task behavior.We compared the off-task activity of students as they took tests in two conditions (one proctored and the other nonproctored).The proctored testing condition showed significantly less off-task behavior compared to the nonproctored condition, as indicated by the Wilcoxon signed-rank test (Z = -7.22,p < .01).

Closeness of the first attempt to the due date.
There is evidence to support the claim that the higher the number of test attempts, the more it potentiates subsequent learning among students (Soderstrom & Bjork, 2014).Therefore, attempting the quizzes early on was considered a productive learning behavior, as it may have given students the opportunity to practice tests multiple times.Procrastination in students' quiz taking was measured as the closeness of the day of the first attempt of a quiz to its due date.For example, if a quiz was due on a given date d, and the day of the first attempt by the student was d1, d1 -d was considered an indication of procrastination behavior.This attribute was termed the "closeness of the first attempt to the due date." Canvas logs the date and start time of each attempt by the student, labeled as "started at."The attribute "started at," which is logged in Canvas corresponding to the first submission of a quiz attempt, was extracted to calculate the day of the first attempt.The due date was a date set by the instructors as part of the learning design.

Number of attempts.
Studies show that retrieval attempts and practice, even if unsuccessful, enhance learning (Hays, Kornell, & Bjork, 2013;Kornell, Jensen Hays, & Bjork, 2009;Richland, Kornell, & Kao, 2009).This may be because unsuccessful attempts can initiate learning between attempts (McDaniel et al., 2011).Quiz designs where grades are awarded for the best attempt among multiple possible attempts encourage subsequent practice among students (Zimmerman, Moylan, Hudesman, White, & Flugman, 2011).In the learning design implemented in MIP, students could attempt a quiz up to 10 times to allow them to practice concepts until they felt confident in having mastered a concept.
In the Canvas quiz-log data for each student, each submission associated with quizzes has a feature labeled "attempt," which logs the ordinal number of the attempt the student makes for the quiz under consideration.The variable "number of attempts" was calculated as the highest number of attempt (maximum value among the logged number of attempts) a student makes for the given quiz.
Spacing the study sessions.Robust findings from psychology as well as studies conducted in classrooms support the claim that distributing or spacing the study time into multiple sessions is more productive for long-term retention than massing the study time into a few sessions (Kapler, Weston, & Wiseheart, 2015;Larsen, Butler, & Roediger, 2008;McDaniel et al., 2013;Nazari & Ebersbach, 2018;Roediger & Karpicke, 2006;Schutte et al., 2015).
In MIP, students could attempt a quiz up to 10 times and were free to choose how to space these attempts across time.The quizzes were open for a window of 9 days, and the students could distribute their attempts over that period or mass them all together on a single day.A score was assigned to the student depending on the number of days across which the attempts were distributed.For example, a student who spaced their practice across three different days would get a score of three and a student who massed their attempts in one day a score of one.

Correlations between variables related to productive learning behaviors and exam
scores.To answer RQ 2, we examined the relationship between the variables related to productive learning behaviors and grades in exams (exams which immediately followed a quiz as well as the final exam).Two of the variables identified from the quiz-log data, the off-task behavior and closeness to the due date, significantly correlated with the exam scores.For Quiz 1, correlation between scores in Exam 1 and off-task behavior was r(90) = -.51,p < .01; between scores in final exams and off-task behavior was r(90) = -.45,p < .01; between scores in Exam 1 and closeness to the due date was r(90) = -.22,p < .005;and between scores in final exams and closeness to the due date was r(90) = -.21,p < .005.For Quiz 2, correlation between scores in Exam 2 and off-task behavior was r(90) = -.30,p < .01;and between scores in final exams and off-task behavior was r(90) = -.30,p < .01.For Quiz 3, correlation between scores in Exam 3 and off-task behavior was r(90) = -.33,p < .01; between scores in final exams and off-task behavior was r(90) = -.34,p < .01; between scores in Exam 3 and closeness to the due date was r(90) = -.35,p < .01;and between scores in final exams and closeness to the due date was r(90) = -.40,p < .01.

Discussion
The present study is significant for the following reasons: Technology-enhanced learning platforms encourage metacognition among learners by supporting self-reflection and self-monitoring (McMahon, 2002).Despite the potential of such platforms to encourage effective learning behaviors and support learners' metacognitive awareness, use of these platforms remains limited in two main areas.First, changes in pedagogic practice to take advantage of the functionalities offered by LMSs are often not implemented (Collis & van Der Wende, 2002;Mitrovic, Suraweera, Martin, & Weerasinghe, 2004;Sinclair & Aho, 2018).For example, many instructors use these platforms mainly to deliver course materials electronically to the students (Campbell, 2007;Vovides, Mitropoulou, & Nickmans, 2007).Second, the integrated features, functionalities, and logged data that can be mined for understanding learners' interaction patterns are rarely explored (Milliner & Cote, 2018).In order to maximize the use of LA data available from technology-enhanced learning platforms, instructors may have to design and implement evidence-based reflective instructional activities (Hernández-Leo, Martinez-Maldonado, Pardo, Muñoz-Cristóbal, & Rodríguez-Triana, 2019).This implies that instructors may have to keep in mind the possibilities of meaningful data collection as early as the design stages of learning activities.The current work encourages instructors to consider the possibilities of implementing formative learning designs and exploring behavioral data with respect to the pedagogical design to refine the implemented design as well as encourage productive behaviors among students.
Previously, instructors have relied on qualitative methods like interviews or observations to understand students' learning behaviors (Mor, Ferguson, & Wasson, 2015).The present work encourages instructors to exploit the potential of using data-driven evidence to explore the actual behaviors of students collected from a technology-enhanced platform.Real-time access to students' actual learning strategies from quiz-log data analysis may help instructors understand patterns of learner behaviors in unsupervised platforms.In turn, instructors can provide meaningful feedback targeted to improve self-reflection among students who show less metacognitive awareness of their learning behaviors.Students' reflection of their choice of study strategies may encourage effective use of the quizzes as a learning tool, which promotes self-testing and spaced retrieval of information.
Many of the existing LA tools (e.g., analytics reported on LMS dashboards) to understand students' learning behaviors and their patterns of engagement rely on static data (like demographics and prior academic records) and/or simple metrics related to student engagement levels, like login frequency, the frequency of course materials accessed, number of discussions posted, and number of downloads of course materials.Criticism of the use of static variables is that these variables cannot be manipulated to implement specific interventions that target student learning and provide insight for improving teaching strategies.Simple metrics that track student engagement may lack the power to contribute to the understanding of student learning (Lodge & Lewis, 2012).Due to this limitation, existing tools may not support instructors in improving their learning activities.Variables considered in this study, which are related to student learning behaviors, are malleable and pedagogically meaningful and, hence, address this limitation.
Overall, this work offers instructors the chance to think about the design of interventions that have a direct and immediate impact on teaching and learning processes (Wise, 2014).Interventions planned with the goal of improving learning strategies that students employ within an unsupervised quiz will allow instructors to move beyond making mere predictions of exam scores.The focus of the current study is on improving the quality of learning of all students and is not limited to identifying at-risk students.The approach undertaken in this work may eventually aid in making the transition to a learner-centric approach (where the use of study strategies, involvement level, and performance of each student with the online platform is tracked and followed up with meaningful personalized feedback) from a variable-centric approach (comprising mere comparison of class averages on summative exams).
The future work will classify students based on their patterns in learning behaviors and examine the differences in exam scores among the identified groups of students.Further, the results of the quiz-log analysis will be shared with instructors to understand the pedagogical implications, such as the possibility of providing meaningful feedback to students and redesigning the quizzes.To meet this end, a qualitative study is planned wherein interviews with the MIP instructors will be conducted.

Figure 1 .
Figure 1.The cyclical process of design and refinement of the implemented learning designs based on data-driven evidence.

Figure 2 .
Figure 2. Timeline of quizzes and exams implemented in the MIP course.