Understanding Faculty Use of the Learning Management System

The learning management system (LMS) has become a critical tool for nearly all institutions of higher education, and a driving force in online learning. According to a 2014 report by the Educause Center for Analysis and Research, 99% of higher education institutions have an LMS in place, and the LMS is used by 85% of faculty and 83% of students. This was not always the case, however. There was a time in the not-so-distant past when using an LMS was considered highly innovative. Understanding the growth and adoption of the LMS is a stepping stone to understanding how faculty may choose to adopt other technological and pedagogical innovations. This study was conducted at a large, research-intensive public university in the Midwest, which has used the same LMS for 15 years. From a small pilot, adoption has grown to nearly universal use. This study used system logs and database queries to examine how faculty used the LMS. The results identified the features that were used most frequently and how usage had changed over time. In addition, the study compared the usage data for face-to-face and online courses to determine if there are differences in LMS use due to course modality. Based on this, it is possible to better understand the role the LMS plays in higher education and online learning, to inform development of next generation learning systems or other innovative technologies.


Introduction
Educators have long sought to infuse contemporary innovations into teaching and learning.Through the years, various print, audio, video, and computer technologies have been incorporated into education and training (Lever-Duffy, McDonald, & Mizell, 2003;Picciano, 2001).With each wave of technological innovation, tools become more seamless and instructional practices evolve.As the Internet came of age, Learning Management System (LMS) technology became widely available, enabling faculty with little technical skill to deliver instruction to students at a distance.Most higher education institutions have since integrated the LMS with other institutional infrastructure systems, encouraged faculty adoption of the LMS, and provided the necessary user training and support.Understanding the growth and adoption of the LMS is a stepping stone to understanding how faculty may choose to adopt other technological and pedagogical innovations.This study at a large, research-intensive public university in the Midwest, which has used the same LMS for 15 years, examined which features were used most frequently and how usage has changed over time.

History of Learning Management Systems
The Learning Management System (LMS), also referred to as Course Management System (CMS) or Virtual Learning Environment (VLE), has evolved over decades of technological innovation to become a cornerstone of institutional instructional technology infrastructure.With roots dating back to the first computer-assisted instruction system, PLATO, developed in the 1960's (Bitzer, Braunfeld, & Lichtenberger, 1961), the LMS was popularized with the advent of the Internet, with many LMS platforms available today (Hill, 2017).LMS platforms commonly include a suite of integrated tools that enable online delivery of instructional content, interaction and collaboration, and tracking and reporting of student participation (Rouse, 2005).The LMS has been touted as having been instrumental in extending instruction and access to learning beyond the physical classroom (Harrington, Staffo, & Wright, 2006), enabling secure online collaborations among faculty and students (Barron, 2003;Liu, 2005), and promoting the integration of otherwise discrete content delivery, course management, and student assessment functions (Linder, Bruenjes, & Smith, 2017;Weaver, Spratt, & Nair, 2008).
The LMS remains a mainstay of online education infrastructure, with 85 percent of faculty confirming their use of the institution's LMS (Brooks, 2015) and 81 percent of chief online education officers reporting the LMS to be the technology that is most important to online programs (Legon & Garrett, 2017).Institutional leaders recognize the importance of supporting faculty in their use of instructional technologies including the LMS, with faculty development ranked as the number one key issue in teaching and learning in 2017 (Educause Learning Initiative, 2017).
Despite its widespread adoption, some have questioned whether the LMS is still needed.Education futurists call for LMS tools and platforms to be more agile to support emerging instructional practices, as some wish to unbundle the components of a learning experience to remix open content and educational apps in new ways (Adams Becker et al., 2017;Anshari et al., 2016;García-Peñalvo & Forment, 2014).Some thought leaders feel current learning management systems are too limited in functionality (Brown, Dehoney, & Millichap, 2015a) and have proposed a next-generation LMS, often referred to as a "next-generation digital learning environment" (NGDLE), to support more personalized and flexible learning experiences (Brown, 2015).Rather than being a single system, the NGDLE would encompass a "confederation of IT systems and application components that adhere to common standards...that would enable diversity while fostering coherence" (Educause Learning Initiative, 2015).
With the maturation of the LMS and critical mass adoption across higher education, there is a need to look deeper into how the LMS is being used, to see if it is still meeting the needs of faculty and students.Measurement of LMS usage, however, is challenging and often relies on estimates based on guesswork.This study addresses this problem by analyzing faculty usage of the LMS via empirical data gathered directly from the LMS database, and without the adoption of sophisticated yet expensive analytics systems.This provides a means to follow the digital footprints faculty leave in the LMS as a reflection of their course design.

Review of Related Literature
Previous studies have sought to understand the motivations and experiences of instructors as they adopt a learning management system (Samarawickrema & Stacey, 2007;West, Waddoups, & Graham, 2007) and their overall satisfaction with the tools available (Walker, Lindner, Murphrey, & Dooley, 2016;West, Waddoups, Kennedy, & Graham, 2007), focusing largely on the attitudes and observations shared by faculty themselves.Not surprisingly, faculty experiences have often varied widely from one institution to another, as enabling and impeding factors differ.Even so, research on the patterns of faculty adoption of the LMS and the degree to which the available capabilities meet instructional needs have been valuable.The data from such studies have provided interesting insights, but have inherent limitations because they often relied on instructor perceptions through self-reported usage.Malikowski (2008) took an alternative tactic in seeking to understand the factors related to breadth of LMS feature use, manually tallying tool usage frequency in LMS courses from across 842 course websites volunteered for study by 394 faculty members over a three-year period.He selected six features of the LMS and recorded the number of times each was utilized within the sample of courses.Through his descriptive and inferential statistical analysis looking at clustered use of tools, he identified that a stable state of LMS adoption at his institution had been reached.While this was an interesting approach, it was a monumental task to manually examine usage across courses and too time consuming to apply and replicate at scale.Conceptual studies have since posited models for a more holistic view of institution-wide LMS adoption, specifically examining analytics captured by the LMS and their potential use for institutional decision-making and instructional intervention.For example, Janossy and Hover (2008) proposed a 14-step model for analyzing user behavior within the system that could be applied to any LMS through content analysis of system database tables.Dawson, McWilliam, and Tan (2008) demonstrated how data captured from the LMS could be used to inform institutional decision-making processes and identify potential "at-risk" students.While presenting examples of data derived from an institution-wide LMS, they underscored the challenge of readily and accurately interpreting data and translating findings to practice.Whitmer (2012) explored the interplay of student LMS usage, persistence, and course achievement within a large hybrid-format course and found that data from the LMS could be used as a meaningful indicator of student effort.The rapidly expanding field of learning analytics continues to push the boundaries of what is known, and what can potentially be learned, from usage data stored by the LMS and other information technology applications.
Despite the conceptual interest, few studies have empirically analyzed LMS usage data at scale and over time.Fritz (2016) sought through an institution-wide analysis of LMS usage data to demonstrate how analytics focused on student use of the LMS could identify effective faculty LMS course designs, noting the potential for LMS usage to serve as a proxy for online teaching and learning engagement.Park and Jo (2016) analyzed system log data from 7,940 LMS course sites to evaluate learning activity, noting significant gaps in LMS usage levels across colleges and disciplines within the institution.More recently, research exploring student use and time on task within the LMS has provided an additional perspective to faculty use.Taking tool use as a proxy for course design, Whitmer, Nuñez, Harfield, and Forteza (2016) sought to discover archetypes of course design across institutions through examination of LMS usage.Analyzing an anonymized data sample that included 70,000 Blackboard courses from 927 institutions with 3,374,462 unique learners, they aggregated tool usage data at the course level based on the percentage of time students spent using each tool.As a result, Whitmer et al. identified five course patterns which generalized broad approaches to LMS usage which ranged from supplemental use to fully integrated holistic use.
Understanding the relevant factors affecting the adoption and use of the LMS can aid the design, development, and implementation of more effective support and training for faculty and learners (Kultur & Yazici, 2014).Given the vast and ever-expanding volume of data being generated and recorded regarding the use of instructional technology systems, there is a need for simple and automated methods for gathering and examining actual usage data.By understanding usage patterns of instructional technology tools such as the LMS by faculty members and students, institutional support personnel and administration can make better, data-informed decisions regarding future technology procurement and support prioritization to help ensure that instructional needs are being met.

Context for Study
The faculty at one Midwest, public, research-intensive university became interested in blended and online teaching in the late 1990s.This grassroot interest was supported through training and resources provided by the university's faculty development center.During that time, a committee of university faculty explored several LMS options and ultimately chose Blackboard (now known as Blackboard Learn).In 2001, a limited pilot was conducted, and Blackboard was adopted as an enterprise-wide system in 2002.
The university's faculty development center was charged at that time with providing support for faculty on all aspects of teaching with technology, including teaching using the LMS.Due to the university's culture of departmental autonomy, use of the LMS was not mandated and training was not required.Blackboard adoption and training have always been voluntary, yet usage has grown year after year.Today, nearly all faculty use the system.However, the term "use" can be defined in many ways and encompasses a variety of scenarios.
As adoption has become more saturated and more courses have been offered online, training and support have become more sophisticated and differentiated.This added complexity strained the capacity of the faculty development center to continue to offer high-quality support.Better data and evidence were necessary to prioritize the best level and types of support moving forward.

Research Questions
Analysis of the LMS adoption and tool usage has developed at the university since Blackboard was selected.Initially, the university only tracked the overall usage of the LMS, such as the number and percentage of faculty and students who used the LMS for at least one course.
As overall adoption increased, it became important to take a more nuanced approach to address individual tool usage as well as overall adoption.This study addressed three primary research questions: 1. What LMS tools do faculty include in their courses most often?2. How has LMS tool use changed over time? 3. Does LMS tool use differ based on course modality (face-to-face versus online)?
In particular, this study considered faculty course design in the use of individual tools within the LMS.

Methods
This study utilized automated methods to accurately identify how LMS tools were implemented across courses and used over time.Previous studies' manual methods for collecting this kind of data were not feasible due to the time and effort involved.Fortunately, data for Blackboard Learn is stored in a database and can be queried using custom-written scripts for either Oracle or Structured Query Language (SQL) servers.Because the university operates Blackboard Learn in a self-hosted environment (which means that the university owns and maintains the servers on which the LMS is installed), university staff also have direct access to the main database where the data are stored.When universities choose to have Blackboard host the system for them, there are still some data which can be extracted from the Open Database.Other LMSs have alternate methods for gathering similar system data, as well.
The SQL queries used in this study were adapted from the work of Kodai (2013).University staff customized the scripts to stay current with changes in the structure of the data due to upgrades to the LMS, as well as to track additional tools and features unique to the university.The SQL script revolved around the COURSE_CONTENTS table of the database, which tracked the tools and content added to courses.Each row of that table represented the use of a tool in a given course.Tools were identified by "handles" (CNTHNDLR_HANDLE) and courses were identified by unique IDs (CRSMAIN_PK1).From there, the script aggregated the data by course ID and tool name, and counted the number of records returned for each course-tool pair.
Occasionally, another table needed to be queried when a tool count could not be aggregated in this manner.For example, the Announcement tool was included in each course only once.Counting the single instance of the tool did not indicate whether or how much the Announcement tool was used.In this case, the query aggregated usage counts based on the ANNOUNCEMENTS table, which was where the content of any Announcements were stored.Because these additional tables also referenced the same unique identifiers for each course, the query was able to combine this with the results from the primary COURSE_CONTENTS table.
University staff ran the SQL script initially against all of the course data which existed in Blackboard (back to the spring 2011 semester), and again at the end of every semester.The results were compiled in a spreadsheet.Once tool use counts were successfully collected from the database, blank records were removed so that the statistics were not reduced by unused Blackboard course sections.Only courses with tool counts greater than zero were kept in the dataset.This included a few courses which did not use the LMS, but ensured all courses with minimal tool use were included.This process recognized that there was a wide range of legitimate use of Blackboard for teaching, and that it was not always possible to distinguish courses taught using Blackboard from those minimally used for other purposes.
Finally, the counts of tool usage were converted using a dummy coding method that only indicated whether or not a tool was used.For every course where a tool was used, the tool was coded as a 1, and coded as a 0 when the tool was not used.This binary approach was used as an initial analysis tool, to determine whether a tool was utilized within a course.There are interesting pedagogical and technology-integration implications for the number of times a tool was used (such as whether a course included 1 grade per student or 30 grades per student, or if 5 announcements were posted or 50), but such investigation is reserved for a future study.For the current research, the dummy coded values were used to compute the percentage of courses that used each tool in a given semester.
A significant aspect of using this approach was data verification and error-checking, at multiple steps in the process.Because the Blackboard LMS has evolved over time, upgrades and patches occasionally necessitated changes to the script.For example, the built-in plagiarism detection tool was originally a stand-alone feature, but then became an option enabled on the primary Assignment tool.This also included a change in the data-structure, which was detected during routine verification measures.Another example is the addition or replacement of tools that required new or updated scripts, such as the migration from Wimba Classroom to Blackboard Collaborate, and more recently to Blackboard Collaborate Ultra.Each of these changes required significant changes and extensive validation.
When the queries were run, the process was monitored to ensure that the operations completed successfully.In the event that one or more scripts failed, the queries were revised and retried.If the scripts ran successfully, the results were scanned visually for any anomalies, such as tools reporting no usage in any course.Finally, the tool usage percentages and trends were inspected for data consistency, including unusual spikes or steep changes.In difficult cases where the queries were particularly challenging to refine, staff built test courses for comparison against query results to identify the appropriate table entry for gathering usage data.To ensure adequate understanding of the results, staff also developed a detailed data dictionary that defined how each tool was measured.

Results
The results of the analysis show that the use of Blackboard for credit-bearing courses has grown substantially over the years at the university.Use of Blackboard by faculty rose from 65.5% of all instructional staff in the fall of 2008, which includes faculty, instructors, and teaching assistants who are instructor of record for a course, to a peak of 92.1% in fall of 2015, before decreasing slightly to 87.9% in fall of 2016 (Figure 1).Over the same time period, student use has remained consistently high, in the low to mid-90%.Since student use is both driven and limited by faculty use of Blackboard, the high percentage of students using it while faculty use was fairly low speaks to the widespread usage across campus.In addition, further analysis of the courses that used Blackboard showed that only 41.9% of all course sections used Blackboard during the fall 2010 semester (Figure 2).However, over time that has increased, and 63.8% of all courses sections used Blackboard during the fall 2016 semester.Interestingly, as with overall instructional staff use, the percentage of courses using Blackboard also peaked in the fall semester of 2015, at 67.0%.For this purpose, "use" is defined as any course section requested by faculty in Blackboard.The university requires faculty to submit an automated request for their Blackboard course to be created.This process also synchronizes the course with the student information system, so that the Blackboard course can be automatically populated with the students who were enrolled in the course.This definition of use indicates that faculty intended to use the Blackboard course, but it does not consider whether faculty did in fact make use of the course, or how they did so.In addition, although it is clear that the majority of faculty were using Blackboard, and adoption rate is an important indicator of the success of a technology implementation, it does not sufficiently answer questions regarding how the technology was used, and how its use had changed over time.The most frequently used tools in the fall 2016 semester were Announcements, Items, Grades, Folders, Files, Assignments, Web Links, Plagiarism Detection, Discussion Boards, and Tests, in that order.Descriptions of each tool and the usage in the fall 2016 semester are in Table 1.While these tools are specifically available in Blackboard Learn, they are common elements of most LMSs.After selecting tools based on prevalence in the fall 2016 semester, the same analysis technique was used to track the usage of those tools for every term in which data were available, back to the spring semester of 2011 (Figure 3).For individual tools, "use" was defined as the tool being deployed or activated at least once within a course.The percentages were calculated as the number of courses in which the tool was used compared with the number of courses in which at least one tool was used (excluding courses which may have been requested, but were not used).The analysis shows that use of most tools generally increased over the approximately fiveyear time period.While use of some tools increased more than others, all of the top ten tools were used in a higher percentage of courses in the fall 2016 semester than in the 2011 spring semester.There was a marked decrease in the use of Items from spring 2011 to fall 2011, while Files rose.That May, after the end of the spring 2011 semester, the university upgraded from Blackboard Learn 8 to Blackboard Learn 9.1, which introduced Files as a new content type.The simultaneous decrease in use of Items and increase in use of Files is evidence that some faculty adopted Files and stopped using Items.As Files became more prevalent, Items again increased in usage, demonstrating side-by-side use of the two tools.
There was a steeper increase in use of several tools in the fall semester of 2015, including Announcements, Items, Grades, Files, and Assignments.This was likely due to an ongoing campaign to reduce printing by both faculty and students.Students were responsible for paying for their printing as of the fall 2015 semester (Ervins, 2012), which would have encouraged faculty to use digital assessments more.As a result, this would have led to increased use of Grades as well, because the assessment tools feed into the Grade Center.
The visualization of the usage trends shows interesting peaked behavior for most tools during each summer semester, as a higher percentage of courses used tools during those semesters.This is particularly noticeable for Folders, Assignments, Discussion Boards, and Tests.There may be several contributing factors that led to this behavior.The primary reason for an increase in tool use in the summer seems to be increased online course offerings each summer.Figure 4 shows that online courses represented a much higher percentage of overall course offerings in the summer semester than in either fall or spring semester.In Figure 4, the online courses category includes those offered both fully online or online with face-to-face meetings, whereas the face-to-face courses category includes traditional face-to-face courses as well as independent study, internship, dissertation/thesis, and student teaching courses.This behavior closely mimics the peaks seen in the tool usage, so the next step was to examine tool usage based on the course modality.Interestingly, face-to-face courses showed some of the same peaked behavior for tool use seen in the initial combined data.Figure 5 shows that Files, Grades, and Assignments were used at somewhat higher rates during each summer semester in face-to-face courses than they were used during the fall and spring semesters.Because most courses have a compressed schedule when offered during the summer semester, at only 8 weeks long as opposed to 16 weeks long during fall and spring semesters, faculty may have relied on more Blackboard tools to support instruction or to reduce in-class time by blending instructional activities into the online environment.Overall, though, tool usage showed less variation and volatility in face-to-face courses than in the combined data.By comparison, online courses used nearly all of the LMS tools at higher rates than faceto-face courses (Figure 6).For example, eight of the top ten tools were used in at least 50% of online courses in the fall 2016 semester.In addition, six of the tools were used in over 75% of online courses in fall 2016 (Items, Announcements, Folders, Assignments, Grades, and Discussion Boards, in descending order of prevalence).This is to be expected, since online courses rely heavily on technology to support student learning.The two tools which were used less frequently in online courses compared with face-toface courses were plagiarism detection and files.Plagiarism detection was used at a rate consistent with face-to-face courses.In the fall 2016 semester, 23.55% of online courses and 22.26% of faceto-face courses used plagiarism detection.Usage was fairly stable from summer 2015 through fall 2016 for online courses, while face-to-face courses saw a substantial drop in usage in summer 2016.Files were used slightly less often in online courses in the fall 2016 semester with 49.17% of online courses and 53.76% of face-to-face courses posting content as Files.Given that Items were one of the most used tools for online courses, the lower usage of Files does not represent less content being posted in online courses, but rather a stronger preference for Items as the means for posting content in the LMS.This may be because the Item tool allows the faculty to provide additional context or instructions using the text editor, which is unavailable for Files.Further investigation is needed to determine the motivation for using one tool over the other.

Discussion
Initially, this study was undertaken to improve LMS support within the university.Understanding how the LMS is currently used has helped to inform the workshops and resources offered by the university faculty development center.For example, the center can prioritize topics for development of online tutorials based on the most-used tools, such as advanced techniques for grading Assignments using integrated rubrics.The results have also influenced the center's summer staffing models for LMS support.In the past, it was assumed that support requests would be lower in the summer because there are fewer courses offered.However, the increased usage rate of LMS tools in the summers challenged that assumption, forcing the center to rethink its strategy, and ensure adequate staff coverage for the summer months.
The data also helped to visualize a bigger picture of the use of the system.In general, a support unit interacts with a fairly biased sample of faculty, namely those that either struggle with the system and therefore experience problems or those who excel at using the system and use it in advanced ways, and thus experience problems.While data on support staff interactions can be used to inform the practices of a faculty development center (Krishnamurthi & Rhode, 2013), it provided a limited view due to this sample bias.Considering the actual use of the system provided more scope to inform decision-making.There were other sources of data to consider, as well, such as website analytics, that could provide insight into the demand or gaps of existing resources (Rhode, Richter, Gowen, & Krishnamurthi, 2015).
There are number of limitations to such a study, however.First, using do-it-yourself data collection using database queries required significant amounts of verification to demonstrate that the queries were gathering accurate results.Because of this, data gathering may be iterative and incremental as opposed to comprehensive, as the process was refined.In this study, there were a few LMS tools about which data could not be gathered, such as web conferencing sessions, because such data was not available to a database query.For other tools, usage data was held in another system, making it difficult or impossible to aggregate with the course-based tool usage data.Web conferencing was again an example of this; the institution's primary web conferencing system stored usage data in a separate server hosted by the provider, and records session and user data but not course, making it impossible to report web conferencing usage in the same format as the other LMS tools.As more technologies external to the LMS are used and integrated with the LMS, such as via a learning tools interoperability (LTI) standard or with publisher provided systems, data gathering will continue to become more difficult.This approach was best-suited for tools native to the LMS.
Another limitation to the data is that some information is simply not available to the researchers.For example, specialized courses, including independent study, internship, dissertation/thesis, and student teaching cannot also be designated as online in the university's student information system.It is possible that some sections of these specialized courses are actually considered to be online courses (such as when the section is dedicated for students in an online degree program), but that information is not currently recorded.Also, the researchers did not have access to other data sets that would be obvious to include in research on LMS usage, such as course grades, achievement of student learning outcomes, or faculty and course evaluations.It would be worthwhile to investigate whether specific tools or combinations of tools impact these types of outcomes, but requires data management structures or protocols not currently in place at the institution.

Conclusion
This is just the beginning of using LMS tool usage data to investigate patterns of teaching and learning.At this one institution, LMS adoption overall may not be increasing because use has reached the saturation point.Total (100%) adoption is not likely to occur, and is not even a desirable outcome.However, it is clear that individual tool use is still increasing, with obvious differences between online and face-to-face course modalities.
There are several future directions that study of LMS usage should consider.For example, this study only considered descriptive statistics of usage trends, and further statistical analysis is needed.In addition to considering individual tools, there is much to be learned about clusters or combinations of tools.Which tools are used together and create synergies of practice?In addition to the differences by course modality, are there also differences to be seen by academic discipline or by pedagogical approach?While using LMS database logs provides a comprehensive and unbiased view of the use of LMS tools, it cannot interpret intention or motivation for use.Future research should also consider why faculty use the LMS in general, or what their pedagogical intention is for adopting individual or clusters of tools within the LMS.
Despite the growing popularity of Next Generation Digital Learning Environments (NGDLE), which are loosely coupled collections of tools (Brown, Dehoney, & Millichap, 2015a), studies such as this show that there is still a need for formal learning management systems to provide structure for the learning process.While the NGDLE is an attractive application of the more advanced cloud-based tools and emerging protocols such as LTI and xAPI, the extensive use of the LMS and of the variety of tools used in each course indicate that the more flexible NGDLE environment may also require too much time and knowledge for individual faculty to implement.

Figure 1 .
Figure 1.Percent of students and instructional staff who used Blackboard in the fall semester of each year.

Figure 2 .
Figure 2. Percent of course sections, which used Blackboard in the spring, summer, and fall semesters of each year.

Figure 3 .
Figure 3. Percent of all Blackboard courses using each of the ten most-used tools.

Figure 4 .
Figure 4. Percent of courses offered face-to-face or online for each semester.

Figure 5 .
Figure 5. Percent of face-to-face courses using Blackboard which use each of the ten most-used tools.

Figure 6 .
Figure 6.Percent of online courses using Blackboard which use each of the ten most-used tools.

Table 1 .
Ten Most Commonly Used Tools in the Fall 2016 Semester