Piloting Learning Analytics in a Multidisciplinary Online Program

Learning analytics is a recent innovation that holds promise for improving retention in fully online programs. However, only a few case studies exist to show models for and outcomes of the implementation of learning analytics systems. This paper reports on a learning analytics implementation in a fully online, multidisciplinary program designed for nontraditional students using a pilot planning group with stakeholders from various roles. The processes for selecting reports, creating communication structures, and evaluating outcomes are outlined. Overall, faculty and advisors were positive about the project and found the reports to be helpful. The results suggest that the actions most often triggered by learning analytics reports were emails to students. Evaluation results suggest that the implementation of the learning analytics program and the interventions enacted had a positive impact on student success, though we acknowledge that it is difficult to isolate the impact of the learning analytics tool itself. We also address several challenges that came along with the implementation of learning analytics including understanding the efficacy of interventions, data security, and ethics.

Online courses can attract students who might otherwise have not been able to attend traditional on-campus instruction because of work, family, or other obligations...if students are more likely to drop out of an online course because of work or family commitments, does that reflect on the nature of the course, or the nature of the student? (Allen & Seaman, 2014, p. 18).
With a growing focus on nontraditional students, institutions are looking for technological solutions to help boost retention (Legon & Garrett, n.d.)-hence an increased interest in the use of learning analytics in higher education. Defined as the "measurement, collection, analysis, and reporting of data about learners and their contexts" (1st International Conference on Learning Analytics and Knowledge 2011, n.d.), learning analytics has been seen not only as a way to improve student retention, but also a way to increase metacognition and improve the online classroom (Papamitsiou & Economides, 2014).
While there is a growing body of research in learning analytics, much of this work has focused on the "microlevel" of learning analytics-within individual courses and contexts (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). Arnold et al. (2014) comments, "despite the explosion of learning analytics research, most of what emerges in the field is course level, small scale, or tool-centric approaches" (p. 257). Less research exists for the utilization of learning analytics across entire programs, let alone online programs, and how individual stakeholders in those programs utilize learning analytics information. To address the lack of research in this area, we present this case study of the implementation of a learning analytics tool in a fully online multidisciplinary program geared toward nontraditional students. During the project, we worked to document processes as well as lessons learned during implementation. Our goal is to generate knowledge regarding the utilization of learning analytics in fully online academic programs.

Use of Learning Analytics in Online Education
While the field of learning analytics is relatively new, the body of work that has been produced in that time is sizable. Key literature reviews, like Papamitsiou and Economides (2014), seek to map the current state of the field by classifying learning analytics research according to the setting in which the research was done (e.g. learning management systems, cognitive tutors), the analysis method utilized (e.g. classification, regression, social network analysis), and the purpose of the research. They then categorized learning analytics research into six key objectives including 1) student behavior modeling, 2) prediction of performance, 3) increasing self-reflection & selfawareness, 4) prediction of dropout & retention, 5) improve assessment & feedback services, and 6) recommendation of resources.
Within that body of knowledge, several research studies have focused on utilizing learning analytics within online learning contexts. In looking at these studies, it is useful to differentiate between 1) projects that use learning analytics in an exploratory manner to better understand the behavior of their online students, and 2) those that are working to build institutional tools that can be directly applied to benefit student success and retention.
In the first category, we find several studies that use learning analytics in exploratory ways in the online environment. These look for predictive relationships between activity and outcomes but do not directly lead to a student intervention. Methods utilized include logistic regression (Carver, Mukherjee, & Lucio, 2017), structural equation modeling (Koç, 2017), and social network analysis (Saqr, Fors, & Nouri, 2018). Kim, Yoon, Jo, and Branch (2018) applied learning analytics methods to examine the relationship between student activities in an online environment and selfregulated learning behaviors as identified in a student questionnaire.
In the second category, the research is more applied-resulting in systems that can be utilized to enable interventions at the course or the program level. Hung and Zhang (2008) used several data mining techniques on student activity in an online course to predict whether or not a student would pass. Results were placed into an early warning system, which automatically sent students an email indicating that they should set up a time for additional support. Smith, Lange, and Huston (2012) describe the development of a predictive tool to identify at-risk students at the fully online Rio Salado College. Using data from the LMS, a risk score (high, medium, low) was created by using a naive Bayes model. This information was then passed onto academic programs which would work on developing an intervention strategy.

Learning Analytics and Institutional Impacts
While there has been high activity in the literature surrounding learning analytics in the last ten years, we are less certain whether this research activity has led to institutional impacts. Learning analytics has shown up for several years in the Horizon Report as a technology destined to impact higher education (Arnold et al., 2014). However, Sharkey and Harfield (2016) argue that learning analytics had entered the "trough of disillusionment" as described in Gartner's Hype Cycle, and what was needed were intentional institutional plans surrounding the adoption of learning analytics.
To that end, we looked for key studies that examined the existing evidence for the impact of learning analytics programs in an institutional setting. A prototypical example of this is Purdue University's Course Signals Project, a predictive learning analytics system which showed students' risk score as a green-yellow-red traffic light. The outcomes of research associated with the project suggested that students who were enrolled in courses that utilized Course Signals were retained at higher rates than those who were not (Arnold & Pistilli, 2012). Critics have argued, however, that the outcomes are the result of a "reverse-causality" problem, and that students who took additional Course Signals courses were retained better because they took more classes in general (Caulfield, 2013). Moreover, critics have remained skeptical of both the ethical use of this classification tool as well as potential negative impacts on student self-efficacy (Rubel & Jones, 2016).
Working in conjunction with vendor Civitas, Milliron, Malcolm, and Kil (2014) detail three institutional case studies that utilized learning analytics. Each case involved a predictive system making a recommendation for a student intervention (through email) to teaching and learning stakeholders (advisors and instructors). Although the research admits that the predictive system took several semesters to calibrate, by the latest semester, each institution found significant gains in persistence for those students who received an intervention. Dawson, Jovanovic, Gašević, and Pardo (2017) document another institutional campaign that involved the use of a predictive model which identified at-risk students and then fed them into a phone call campaign from advisors at the school. While initial analyses showed that students who were contacted were 31% more likely to continue in their studies when compared with those who were not contacted, the overall explanatory power was low. In fact, when looking at the full model, the researchers determined that the interventions did not have a significant effect on retention when student specific features were controlled for.

Higher Education Technology Implementation
To better understand the learning analytics implementation, it is important to discuss literature pertaining to higher education innovation and technology adoption in general. Higher education serves as an engine and driver for enriching the collective knowledge of humanity. It is critical that higher education institutions know how to sustainably stay innovative and lead the changes (Lundvall, 2008).
Scholars such as Zhou and Xu (2007) (2013) shared what barriers existed for technology adoption. Not surprisingly, faculty indicated the availability of technology and support. Other researchers such as Beggs (2000) highlighted what factors helped with faculty adoption of educational technology. Beggs (2000) administered a survey to 348 university faculty in the US. Results indicated that (1) improved student learning, (2) advantage over traditional teaching, (3) equipment availability, (4) student interests, and (5) ease-of-use are important.
Synthesizing relevant literature, researchers in online education recently proposed a framework which higher education should follow for efficient faculty adoption processes. The model provides three stages with detailed objectives in each stage. The process is composed of (1) awareness/exploration, (2) adoption/early implementation, and (3) mature implementation/growth (Porter et al., 2014). On a follow-up study that empirically tested the innovation adoption model, Porter and Graham (2016) found that sufficient infrastructure, technological and pedagogical support, evaluation data, and a solid purpose of adopting the innovation in the context of blended learning environments would influence faculty adoption. Learning analytics implementation could follow a similar adoption process of other emerging technologies; higher education staff and faculty could strategically prepare the adoption and implementation processes while minimizing unexpected barriers and mistakes.

Learning Analytics Ethics
The speed with which the field of learning analytics has grown has rapidly outpaced the development of ethical principles and practices surrounding its adoption. As the field develops, it becomes imperative to commit to a principled, reflexive, and sustained exploration of ethical considerations surrounding the use of learning data. The need for critical perspectives of learning analytics is particularly salient in the context of commercial and capital interests, reification of marginalizing university practices, social justice and equity, pedagogy and instructional quality, and benefit structures for stakeholders in the university. These ethical considerations must be addressed when (and before) implementing learning analytics systems (Ngqulu, 2018), even when (and especially when) the ethical boundaries of learning analytics are ambiguous or unclear (Jones, 2019). Without addressing these issues and designing policy and practice around them, institutions that adopt learning analytics may not only amplify the risk of harming students but amplify this risk at scale.
Despite claims to enhancing student success, retention, and graduation, the implementation of learning analytics systems is riddled with ethical questions that often go unanswered by their proponents. Most higher education analytics systems are purchased through LMS vendors, which, by their commercial nature, are designed as top-down solutions which administrators may purchase to address institutional interests. As such, the nature of learning analytics systems are often defined by their lack of participatory design. This lack of participation often precludes institutions from engaging in critical discourse with vendors regarding ethical boundaries and policy. While vendor transparency around ethics and limitations of learning analytics products is problematic, institutional actors who avoid a critical examination of these technologies are also implicated in the problem (Gregg, Wilson, & Parrish, 2018).
As such, institutions have the moral imperative to critically evaluate these tools and establish clear data governance on their use. Since learning analytics systems affect a large array of institutional actors with varying roles and interests, it is imperative to shift the design paradigm to include the perspectives of all stakeholders, most notably students (Ifenthaler & Schumacher, 2016;Jones, 2019;Swenson, 2015). Paramount concerns for this paradigm shift include: • Student rights and consent (Cormack, 2016;Howell, Roberts, & Mancini, 2018;Slade & Prinsloo, 2013;Swenson, 2015); • Little emphasis or clarity on potential harms to students, particularly those historically marginalized.
While the promise of research of learning analytics in online learning is helpful, we see a need for more research that helps us to understand the ethical implementation of learning analytics tools and reports in fully online courses and programs. As many online institutions are wrestling to understand how learning analytics might be utilized to improve student success and retention in their program, we feel that research in this vein is sorely needed. The research questions for this case study were guided by practical questions that the researchers asked about the process of implementing learning analytics in online education: RQ1: What are faculty and advisor perceptions regarding the utility of reports generated by the learning analytics system? RQ2: What interventions were most often triggered by the learning analytics process? RQ3: What evidence of impact exists for using the learning analytics system in an online program?

Learning Analytics System
A large, public western university implemented a learning analytics system in early 2017. This system, Blackboard Analytics for Learn (A4L), utilizes data from the Blackboard Learn Learning Management System and combines it with data from the Student Information System (Peoplesoft), making it available in a variety of formats. Program faculty and administrators are able to access data through visual reports.
Once the system was installed and the data validated, the online learning division of the university sought to partner with online programs to pilot the use of learning analytics reports. A pilot project was created with an online multidisciplinary program. This partnership was appealing for several reasons. First, the program is committed to student success and is interested in adopting new processes that would help improve student success in the program. Second, the program had strong student support built in, including faculty and advisors who are cognizant of the challenges that nontraditional learners face and who are committed to exploring innovative methods to help their students succeed. Lastly, the program leadership was open to the suggestions of new processes and tools that would help them better serve the needs of their students.

Participants
The program was a fully online undergraduate degree-completion program tailored to nontraditional learners-defining features of which include part-time enrollment, financial independence, full-time work status, or financial dependents (National Center for Educational Statistics, n.d.). At the time of this study, 88% of the students in the program were 25 years old or older, 62% were first generation college students, and 68% had returned to college after 10 or more years since high school. Since students in the program were often full-time working adults, the program used a "high-touch" approach, with a lower ratio of students to advisors than is found in traditional programs.
A pilot planning group began meeting in April of 2017, with a plan to launch the pilot in September of 2017. Pilot planning group participants represented several different roles associated with the program, and consisted of 1) researchers and administrators from the online learning center, 2) the program director, 3) the director of advising for the program, 4) the lead advisor for the program, and 5) a full-time faculty member from the program. This group represented the interests of other program stakeholders including additional faculty (full-time and adjunct) and advisors. In the initial meetings, the goals for the project were collaboratively created by the group and a plan for project deliverables was outlined. The pilot planning group met for four one-hour sessions during the summer of 2017 on a bi-monthly basis to plan the pilot activities. Throughout the semester, the group met six times to collect feedback and discuss the status of the pilot.
Upon launch in the fall, learning analytics were piloted in seven different courses in the program with 7 faculty members participating. The courses were all 15 weeks long and were taught completely online. Overall, a total of 338 students were enrolled in all of the classes. Because the focus of the study was on the implementation of the tool to support the teaching and advising process within the program, specific consent was not obtained from the students in the study. However, a student use of data statement was crafted (outlining how this data might be used) and was linked to in the learning management system.
Courses covered a variety of topics in the program related to communication and leadership. Instructors for the courses were experienced faculty who had taught the courses for several years, though none had previous experience using learning analytics. In addition to the faculty, there were four advisors in the program that were utilizing the learning analytics reports to support their students.

Report Selection and Communication Process Development
One of the goals of the pilot project was to determine what learning analytics reports should be made available to advisors and faculty members in the program. Out of the box, the university's chosen learning analytics system included over twenty different reports delivered in SQL Server Reporting Services (SSRS) designed for different uses. After reviewing the reports, we found that none of the reports delivered a single "risk" score for students, and instead a student's risk level would need to be determined by the faculty or advisor looking across an ensemble of metrics available on the reports.
To narrow the information used for the pilot, two or three reports were selected by each stakeholder group. The selection process included brainstorming and prioritizing metrics for each stakeholder group, and then matching the set of delivered reports with those priorities. Highly prioritized metrics from the program representatives in the pilot planning group included submitting assignments late and not logging in.
After completing the report selection process with the pilot planning group, 4 reports were prioritized to be used in the pilot-2 for advisors and 2 for faculty. The selected advisor reports included: 1. Advisor At-a-Glance Report (Figure 1). This report shows current course information about students on an advisor's caseload. This includes current course grades, the number of course logins, total time spent in course, course interactions, and course submissions (assignment submissions, discussion forum activity, tests). (Figure 2) -This report contains a list of advisees whose current course grade is below a certain level in the course. Selected faculty reports included:

Grade Center Exception Report
1. Course At-a-Glance Report (Figure 3). This report shows information about a single Blackboard Learn course. This information includes the number of assessments, content, and tools used in the course, as well as information regarding overall student activity in the course (logins, minutes, interactions, and submissions).
2. Activity Matrix Report (Figure 4). This report shows information about student participation in the course using a heat map visualization. It calculates a scaled score for the student's course activity and overall grade, and places them into a 4-quadrant matrix with the following classifications:

Figure 3
Screenshot of Learn Course At-a-Glance Report

Figure 4
Screenshot of Activity Matrix Report With the reports identified, the pilot planning group next worked to create a delivery method, communications plan, and intervention strategy. After weighing several options for delivery (having stakeholders log into a portal vs. delivering reports in their email), the group decided to send reports on a weekly basis through email. This push notification would act as a reminder that the faculty or advisor check on the progress of students.
The pilot planning group also felt that the creation of a communications plan and intervention strategy would be essential to the success of the program. To create this, the pilot planning group 1) documented their existing communications workflow and 2) augmented it with key points in the semester where data from the learning analytics reports could be leveraged. This resulted in a communications and action workflow between advisors and faculty members (Table  1).

Advisor and Faculty Workflow with Learning Analytics Reports
Prior to Term (Week 0) 1. Faculty: Email announcement is sent to students enrolled in the course (1 week before course is open) 2. Advisor: Contact students and encourage them to meet with their advisor and register for courses Beginning of Term (Weeks 1-2) 3. Faculty: Learning analytics report is sent to faculty during the first week (Thursday) of students who have not logged into the course yet. Faculty to directly contact student and CC advising coordinator 4. Advisor: Checks learning analytics reports to monitor student progress, reaches out to students identified as at-risk, and if necessary, will loop in the faculty 5. Faculty: Within the first two weeks, using learning analytics reports, faculty members find students who are not participating, reach out to those students, with the advisors cc'ed 6. Advisor: Performs outreach to students based on additional conversations from the faculty members Middle of Term (Weeks 3 -13) 1. Faculty: Utilizing learning analytics, checks student progress throughout the semester, contacts student when needed, while keeping advisors in the loop 2. Advisor: Checks learning analytics reports to monitor student progress, especially in relation to key dates in the semester (withdraw date, enrollment date, internship deadlines) 3. Advisor and Faculty: If there are concerns about a student's well-being, advisors submit a CARE report (Campus Assessment Resource and Education) 4. Advisor: If a student has an extraneous life circumstance, the advisor may contact the faculty to let them know there may be external factors affecting success 5. Advisor and Faculty: Maintains communication with faculty/advisor about possible student withdrawal End of Term (Week 14 and Beyond) 6. Faculty: Notifies advisors about incomplete grade requests received 7. Advisor: Help students through academic appeal By clearly outlining the flow of information and responsibilities associated with receiving information from the learning analytics system, the pilot planning group intended to reduce duplication of effort in the project.

Assessments and Measures
To assess the impact of the use of learning analytics reports on the online program, several assessments and measures were established to provide both formative and summative evaluation and to help us answer the research questions. These included: 1. Weekly surveys to track report usage and actions 2. A mid-term feedback survey for faculty 3. Interviews with program advisors 4. Pre-/post program outcome measures Weekly Surveys to Track Report Usage and Actions. In an effort to understand the actions of advisors and faculty based on information received from learning analytics reports, a link to a short survey was included on each report that was sent out during the week. This survey asked the respondent whether they took any action as a result of the report, what form of action they took (e.g. email, phone call, face-to-face conversation), and whether they had any feedback about how the report could be improved. In addition, faculty and advisors were asked to capture stories regarding how individual students were impacted by the enacted interventions.

Mid-term Feedback Survey for Faculty.
To understand the perceived value of the learning analytics reports for faculty, a survey was sent to them half-way through the semester (Appendix A). The survey asked questions regarding perceived usefulness, accuracy, metric importance, interventions, and intention to use learning analytics data in the future. Of the nine faculty that used the learning analytics reports during the semester, six (66%) replied to the survey.
Interview with Program Advisors. To gather feedback from program advisors regarding their use of learning analytics and how it impacted student success, several of the authors met with advising staff halfway through the semester and conducted an unstructured focus group. We chose to do a focus group instead of a survey for this group, because the advising team was co-located (while faculty worked mostly remote) and we felt that we would be able to obtain richer data through a focus group. The protocol used for this focus group is included in Appendix B.
Pre-/post-program Outcome Measures. As a final measure of impact, we compared retention rates for the program from before learning analytics were integrated with those after. While specific retention data of the 338 students in the courses that were utilizing learning analytics reports were not available, we felt that looking at the overall retention numbers of students in the program could help us understand how introducing learning analytics contributed to potential improvements in student success.

RQ1: What are faculty and advisor perceptions regarding the utility of reports generated by the learning analytics system? Faculty Survey.
Faculty were asked about their perceptions of the reports along several dimensions (Table 2). Overall, the perceptions of faculty towards the reports were positive, with the faculty most strongly agreeing that 1) the reports contained accurate information, 2) the reports encouraged them to take action, and 3) based on the report, they knew what plan of action to take. The weakest dimensions (though still averaging above the midpoint) centered on the usefulness of having reports emailed on a periodic basis and whether the reports helped them learn something new about their course. Faculty were also asked to rank the usefulness of individual learning analytics metrics in identifying students that were at risk. Overall, faculty (n = 7) found logins the most useful (M = 1.86), followed by minutes in course (M = 2.57), interactions (M = 3.14), submissions (M = 3.57), and current course grade (M = 3.86). Finally, they were asked "How likely would you be to continue use of [learning analytics] reports in the future?". Results here were very split with half the faculty indicating that they were either extremely or somewhat likely (n = 4, 57%) and the other half indicating that they were somewhat or extremely unlikely (n = 3, 47%). When asked why they would be unlikely, faculty responded that they thought there were more effective ways to get the needed information.
Advisor Focus Group. In addition to gathering feedback from faculty, we also held a small focus group session with the four advising staff of the online program. We first discussed the impact that learning analytics reports made to their workflow. Overall, advisors felt that the reports were a useful additional tool that they could add to their toolbox. They mentioned looking at the Advisor At-a-Glance report to check to see if a student was struggling across multiple classes. They also used the reports as a tool for corroboration. When needing additional information about a student, they found it helpful to be able to consult with the reports in order to look at what a student's course activity had been like.
Advisors were also asked to give additional feedback regarding their use of the tool. Most felt that the delivery of the report by email was not necessary. Instead, they preferred access to the tool on an as-needed basis. The advisors also felt that there could be better tools for supporting the workflow around student interventions. Currently, most documentation of interventions takes place inside of emails between students, faculty, and advisors, and certain information is placed into the advising notes feature of the student information system. The challenge with advising notes, however, is that the process of attaching email communications to the advising notes is rather cumbersome.

RQ2: What interventions were most often triggered by the learning analytics process?
As part of this pilot project, we sought to capture any interventions that resulted from the use of the learning analytics system. Because the system itself did not contain an internal messaging system to track interventions, we attached a survey link to each report email that was sent out, asking the faculty member or advisor to document any interventions. Over the course of the semester, we captured 60 reports from faculty members, and 9 reports from advisors. Faculty interventions are displayed in Figure 5 and advisor interventions are captured in Figure 6.

Figure 5
Faculty Documented Interventions Resulting from Learning Analytics

Figure 6
Advisor Documented Interventions Resulting from Learning Analytics The most frequent action recorded by faculty was no action. This is likely because the faculty were continuing to monitor the student's progress and did not feel a need to take action. The next most frequently cited intervention was an email to student (32.4%), other (13.2%), email to advisor (11.8%), phone call to student (2.9%), and revised assignment (1.5%). For advisors, the most frequently recorded action was email to student (46.2%), phone call to student (23.1%), other (15.4%), email to faculty (7.7%), and no action (7.7%).
We recognize that this is an imperfect method for capturing the types of interventions that resulted from the learning analytics system as it completely relies upon end-users to report the interventions. However, having this information gives us a sense of the variety of means that are being used to reach out to students who are deemed at-risk.

RQ3
: What evidence of impact exists for using the learning analytics system in an online program?
Next, we sought to understand whether these enacted interventions impacted students positively. While this project did not allow us to gather student level empirical evidence about the effect of using learning analytics, we used program outcome data as well as qualitative stories to give us a sense of the potential impact.

Program Outcomes
One data point used to evaluate the impact of the learning analytics system was program retention data. Here, we defined retention as students that either completed or were retained at the university (not just in the program) from one semester to another. We felt that including data regarding whether these students were retained at the university level was important because with nontraditional students, keeping them enrolled anywhere at the university is considered a success. We collected retention information for all students in the program as a whole (both face-to-face and online) as well as those students who were in the fully online program for the Fall 2017 to Spring 2018 semesters (as of the 10th day), and compared it to the same time frame from 2016-2017. This data is displayed in Tables 3 and 4 below. In the cases of all the students and the online-only students, we saw a marked jump from year-to-year in retention (3% for all students and 10% for online-only students). While we are assuming that the increase in retention cannot be solely ascribed to the use of the learning analytics system, it is nonetheless a good sign that the implementation of the system coincided with increased levels of student success in the program.

Student Stories
While retention numbers can give us an idea of the impact of the learning analytics system, forming relationships with stakeholders and discussing their individual experiences revealed richer insight on how the system impacted students. Here, the timing of the intervention and support has a high impact upon the student. In looking at documentation from faculty and advisors who participated in the pilot, we identified the following scenarios: • The intervention triggered a student to resolve financial aid issues and complete documentation to receive learning accommodations in their courses.
• A student was struggling financially. An advisor was able to refer them to the Dean of Students, where they received emergency funding for food.
• After receiving two interventions from advisors and faculty, a student was able to get back on track in the program.
• A student fell behind in a course with a major project on the horizon. Because of the intervention, they were able to hold onto a C.
When viewed through this lens, learning analytics helped expedite the process of getting students help when they were most in need, with some of these cases (such as getting emergency funding for food) helping to ensure the student's overall well-being.

Discussion
The goal of this study was to present a case study for the use of learning analytics in a fully online program geared toward nontraditional students. In this project, we found that it was important to deliver reports to key program stakeholders in an actionable way. This included identifying key information for supporting at-risk students and delivering reports to stakeholders in a convenient manner. Equally important was outlining a set of intervention and communications workflows which acted as a "playbook" for the program in supporting student success by: 1) allowing program stakeholders to understand their role in the intervention process; 2) defining key milestones where learning analytics information would be leveraged to trigger interventions; 3) defining intervention actions that would result from a trigger; and 4) defining how enacted interventions would be communicated to other stakeholders. We suspect that much of the benefit of any student success initiative (whether based on learning analytics or not) would come from the intentional coordination that a workflow like this provides. Therefore, we suggest that other online programs consider developing their own similarly structured workflows when engaging in a student success initiative.
Once the reports and communication plan were launched, we sought feedback from faculty and advisors regarding the reports. Feedback from faculty was positive in terms of utility and accuracy. However, in terms of future use, only half the faculty indicated that they would be likely to use reports in the future. Those faculty, who were not likely to use reports again in the future felt that there were other ways to get the same information. They felt that there was some overlap in terms of information that could be obtained through learning analytics reports and what could be obtained through other built-in tools in the LMS, like the gradebook. This indicates that the learning analytics report as currently constituted did not give a sufficient value proposition to some faculty to encourage future use.
Advisors were also mostly positive about their use of the learning analytics reports. They appreciated the ability to look at an advisee's performance across all of their enrolled courses to identify potential barriers for success. The advisors were quick to note some of the technological challenges that came along with tracking interventions that were made to students as a result of information that was in the learning analytics reports. This is something that should be considered in any implementation of a learning analytics system designed for student success, especially at a large university. To avoid burdening students with numerous intervention messages, higher education institutions should consider ways that they can track and coordinate interactions with students. We may see more attention to this as more colleges and universities adopt Customer Relationship Management (CRM) tools to track interactions with students.
Despite not having technological infrastructure to track all of the interventions, we were still able to get some information regarding the types of interventions that were enacted by both faculty and advisors. Interestingly for faculty, the most common intervention was no action, meaning that they chose not to reach out to the student based on information received. To be clear, these reports were simply showing metrics of student engagement for the course-not specifically high-risk students. While we don't have any empirical evidence to indicate why there was a lack of action from the faculty, we can infer that the faculty either didn't see a student who was at risk, or they were deciding to continue to monitor the student before they enacted any type of intervention. Regardless, the records indicated that faculty were frequently reaching out to students via email-much more often than via phone call. Although the documented interventions for advisors was low, they reached out to students most frequently through email and phone calls.

Challenges, Limitations, and Next Steps
As this project was a pilot, we sought to elicit some challenges and lessons learned in the process. We planned to use these to drive continuous improvement around the use of the learning analytics system at our university. These challenges were derived from conversations with administrators, lead faculty, and lead advisors that participated in the pilot group as well as data gathered via surveys and focus groups from faculty and advisors.
Understanding Interventions. As can be seen in the results above, the use of the learning analytics system triggered several interventions-a majority of which were emails for at-risk students. While the faculty members had some guidance regarding the emails, there needs to be additional investigation into what kinds of messaging is most effective for prompting students to action. Milliron, Malcolm, and Kil (2014) discuss the growing field of "Intervention Science," understanding what types of interventions are most well suited to a given situation. This study tracked the type of interventions but did not measure the effectiveness of each type. More research is needed to understand the effectiveness of individual interventions and how they can be utilized by stakeholders in student success initiatives, particularly how learning analytics might heighten or diverge student success equitably for historically marginalized student populations.
Another identified challenge was finding a way to internally track faculty and advisor communications with students. Though the faculty and advisor workflows (Table 1) were utilized by program members, there was no easy way to communicate across the program about which interventions were enacted for given students. Advisors commented that they tried to use several means of documenting student notes (including in a notes field of the Student Information System and a spreadsheet) but there was no uniform system that was used across all stakeholders in a program. A more robust system like an academic Customer Relationship Management (CRM) system may help to better coordinate intervention efforts across the program.
Data Governance and Ethics. During the pilot, we ran across several instances where there were questions about who should have access to A4L data, and how that data should be utilized. Some of these questions included (but were not limited to): • Should students have access to data captured in the learning analytics system? What should be their awareness regarding how their data is being used?
• How can faculty and advisors mitigate bias in interpreting and acting on data?
• What level of access should advisors have to course-level data?
• Should faculty be able to see other colleagues' course-level data?
While there were some data policies and procedures in place at the university, we found no policies specific to learning analytics. Instead, we used our best judgement, taking a cautious approach in allowing access to learning analytics data. In addition, we worked on the initial stages of a consent and data transparency model. We made concerted efforts to train faculty about what type of data was available regarding their own class; this would help them have an understanding of some of the metrics that advisors had access to. We wanted to ensure that there were no surprises or hidden metrics that could be weaponized against the instructor.
To increase transparency with students, we worked with the Provost's office and the department covering learning technology to develop a "student use of data statement." The goal of this statement was to make existing data policies explicit to the student as well as give parameters of usage around the live course data that was being utilized in learning analytics. These parameters included being "used by instructors, advisors, or other institutional officials to support student learning or improve administrative processes." This statement was posted on the university's website and a universal announcement was made in the learning management system at the beginning of each semester. However, we readily acknowledge that this is only an initial step into developing student awareness regarding the collection and use of learning data.
The goal of this research was to create a case study for the implementation of a learning analytics system within a completely online program. It is limited in its generalizability because the implementation of learning analytics is dependent on contextual factors-attributes of the institution, program, students, technology, and availability of institutional resources-which influence implementation decisions. This case study therefore serves as the product of the decisions made by our institution in the implementation of learning analytics. It is our hope that other institutions seeking to utilize learning analytics (especially in online programs) will use our decisions and lessons learned as a starting place for their own implementations, and that as they implement, will document their own decisions and feedback as we have done. Doing so will help to build growing evidence for the benefit of the implementation of learning analytics within online programs.
In the course of this research, we have identified several areas for future work and research including exploring more flexible delivery of data from the learning analytics system. During the pilot we received feedback from faculty that although learning analytics data was helpful, it wasn't particularly insightful. This is partly due to the canned reports delivered by the learning analytics system and their lack of flexibility. We are working on ways to take data directly from the data warehouse and port them into more flexible reporting platforms. This will allow us to shape the data into reports that are useful to additional stakeholders, including program directors and instructional designers.
We also have much work to do in terms of data governance, privacy, ethics, and equity. Additional work in this vein is ongoing at our institution and will be multi-pronged. Needed activities include: 1) better student awareness/consent regarding what data is collected and how it is being used; 2) more policies and procedures around access to and use of student data; 3) better practice in leveraging data in a care-centered approach, mitigating any potential harm from the use of data; and 4) community-driven research and feedback on the equities and inequities our learning analytics system may pose. We strongly encourage other institutions who are engaging in this work to develop ethical frameworks that are supported by literature to guide their utilization of learning analytics.