Enrollment , Engagement , and Satisfaction in the BlendKit Faculty Development Open , Online Course

BlendKit is a 5-week course designed by the University of Central Florida in an open, online format specifically for the professional development of higher education faculty and designers preparing to design and teach blended learning courses. The evaluation of this course provides us with interesting and valuable information on the success of using an open approach of a MOOC compared to more traditional, highly structured professional development offerings typically seen in higher education. This article will focus on the lessons learned from the third iteration of this open, online course.


Introduction
In 2011, the University of Central Florida (UCF) and the American Association of State Colleges and Universities (AASCU), in collaboration with twenty AASCU member institutions, received a Next Generation Learning Challenges grant for the Wave 1 blended learning challenge area (About Next Generation Learning, 2014).Our project sought to expand adoption of blended learning practices by sharing UCF's 15 plus years of blended learning knowledge with AASCU's twenty participating institutions.
The Blended Learning Toolkit website was created to disseminate information on blended learning based on proven practices and successful implementation of the modality at UCF.The website includes successful strategies for blended course design and delivery, assessment and data collection protocols, and examples of blended courses.In addition, two model courses for Algebra and Composition were provided to fulfil grant requirements.All materials are available through a Creative Commons license and are open to everyone (Moskal & Cavanagh, 2013).
As part of the NGLC grant, the developers of the Algebra and Composition courses provided "train-the-trainer" materials and workshops to participating schools.However, there were a number of faculty who were developing blended courses in other disciplines and were eager for guidance and resources to help with the process.To fill this gap, the BlendKit Course was created to help faculty think through issues involved with designing a blended course.
The first BlendKit Course was delivered during the summer of 2011-open, but advertised specifically to the 20 AASCU institutions.However, we found that participants from all over the world joined the course.In total, there were over 200 participants from seven countries.The course included: • Reading materials around weekly topics of (1) understanding blended learning, (2) interactions, (3) assessment, (4) content and assignments, and (5) quality assurance • Weekly focus questions • Do-It-Yourself projects to help participants step through the design of a blended course • Weekly webinars with successful blended faculty members.This first course was modeled after the connectivist MOOC (cMOOC) format.Participants could engage in part or all of the course, as desired.They shared their thoughts through blog postings, tweets, Diigo and Flickr.
The course was delivered again in fall 2012 to 1,230 participants from 29 countries, and ten K-12 districts.In this offering, the course's content remained essentially the same; however a learning management system (LMS) was incorporated to provide structure that some participants felt the 2011 section lacked.Participants still had the option to use blogs and other social networking avenues for course interaction.Badges were introduced in this iteration, providing a structure for completing course activities that previous participants had requested.
In spring 2014, the BlendKit Course was offered once again.In this third offering, there were 2,840 participants from 50 countries and 22 K-12 districts.The course was delivered through the Canvas Network learning management system (LMS).This iteration included a partnership between UCF and EDUCAUSE to provide a credential certificate and badge.Participants opting for the certificate had to pay a nominal fee, complete the BlendKit Course and submit a course portfolio.In order to receive the credential certificate, the participant's course portfolio needed to demonstrate successful application of blended learning concepts as discussed in the BlendKit Course.
Badging was introduced into the course in 2012 and remained a course component in the 2014 offering, partially in an attempt to increase engagement as participants from earlier iterations requested more structure and guidance in what they needed to "do" for the course.Figure 1 illustrates the BlendKit badging structure with five content areas and five activities in each.Participants had to complete an activity to receive a badge for that activity (Webinar, DIY, Reading, Blog post/comment, Info/stream contributor).With five possible activities and five weeks of BlendKit, participants had the ability to receive 25 total badges.Those earning a badge in each of the weekly Blended Learning content areas (Design, Interaction, Assessment, Content, and Quality) were classified as completers.UCF and EDUCAUSE plan to continue to deliver the BlendKit Course and the certificate on a periodic basis.Between deliveries, the content is still open and available for anyone to work through the materials on their own.Also, institutions may adapt the content to meet the needs for their institution.

The History of Participation and Engagement in Open, Online Courses
As early as 2009 in studying the first MOOC, "CCK08: Connectivism and Connected Knowledge" facilitated by George Siemens and Stephen Downes (Moe, 2015), Fini (2009) noted the challenge of engaging with informal learners who "in the absence of a stronger motivation, attend only partially" (Section 01: The CCK08 Course, para 5).Years later, in an analysis of over 100 open access sources on MOOCs spanning the years 2008-2014, Koutropoulos and Panagiotis (2015) identified "learner motivation" and "learner engagement and participation" as two of 15 pervasive issues affecting MOOCs.
One particular metric of engagement, MOOC completion rate, has received considerable attention.Bruff (2013, February 19) observed that learner completion rates of MOOCs were notoriously low (e.g., 5%), leading many in higher education to question the value of open, online courses.However, he suggested that perhaps the low completion rates were an indicator of a more diverse set of learner needs than educators see in traditional university courses, noting "[m]aybe all those 'dropouts' got just what they wanted out of the course" (para 4).Similarly, in a summary of research findings released by Harvard University and the Massachusetts Institute of Technology about 17 MOOCs in which the completion rates also averaged 5%, Kolowich (2014, January 22) noted that many students undertake MOOCs with no intention of completing the course and observed that in such an environment "[s]uccess and failure take many forms" (para 9).Indeed, Hill (2013, March 2) proposed four distinct types of MOOC learners as classified by their intentions toward and behaviors associated with MOOC enrollment: lurkers, passive participants, active participants, and drop-ins.Thompson and Moskal (2015) noted that the goal in the design of the first version of the BlendKit MOOC (i.e., "BlendKit2011") was to "remove obstacles that might lead a participant to disengage in the course, while still providing a wide range of possibilities for interaction and course engagement for those who had the time and motivation" (BlendKit2011, para 2).As a result the evaluation of BlendKit2011 was guided by a principle Thompson and Moskal (2015) described as "gathering all data possible" in order to "read… [the] tea leaves" and, thus, "find the accurate picture in the myriad patterns" (Evaluating the BlendKit2011 Experience, para 2).They noted that while a "saturation point" (Thompson and Moskal, 2015, Conclusion, para 2) can be reached with such an approach, these rich data are worth revisiting "again and again" (Conclusion, para 2) to inform future cohort iterations.

Data Collection Methods
In evaluating the BlendKit2014 offering, we not only wanted to examine who participated, but also why they enrolled; their participation patterns with the open, online course; and their opinions and comments on the course content and delivery.We were also interested in gathering data that could help inform and improve the future offerings of BlendKit.Knowing the past history of challenges when measuring engagement in MOOCs, we used a variety of sources to capture as many pieces of the puzzle as we could.Naturally, we relied on participants to provide us with information on their backgrounds as well as feedback on the course.
When registering for the course, participants were asked for their contact information including name, organization and email.Optional information requested included a phone number, Twitter username and blog URL(s).This data allowed us to create multiple interaction streams with which participants could interact as they desired.In addition, email addresses of registrants provided a glimpse-albeit a challenging and limited glimpse-into the countries, school districts, and universities or colleges represented by the attendees.
A welcome survey was sent to participants who registered, requesting some general demographic information including country, gender, ethnicity, language, education, etc.We also asked questions regarding why each participant registered for the course and previous experience with online and blended learning.
Weekly feedback was requested from the participants on their views of the course material and their interaction with the course moderators and other participants.In addition, questions regarding their participation for that week as well as expected participation in future weeks of the course were another source of information.Knowing that open online courses often have very different participation patterns from "traditional" courses (in any modality), we sought to determine how much of the difference was due to personal constraints versus lack of relevance or interest around course topics.Again, this data would help us plan for future iterations.
An end-of-course survey to all registered participants provided data on their overall view of the course and its usefulness to their goals and future direction in blended learning.Because the participants who purchased the credential option were required to provide more information about themselves, an additional survey was sent to them which was directly related to why they chose to obtain a credential as well as requesting feedback on their experience.
Finally, we captured qualitative-and often anecdotal-data from the various methods of interaction available in BlendKit.For instance, tweets and postings often provided a notion of the "pulse" of how the course was progressing and provided hints at participants' interactions and attitudes toward the course itself.While not "significant" in a statistical sense, these were practically significant to course facilitators and to keeping participants engaged as the course progressed.
The results presented below focus on who registered for BlendKit2014, their engagement with the course, and their overall perceptions.Certainly, we have a limited view of the many data points we have collected, but we report lessons learned that may inform others who are attempting to develop or evaluation similar courses.

Enrollment
BlendKit2014 had 2,840 participants who completed the registration process to participate.Figure 1 illustrates the worldwide dispersion of participants, with size of the dot indicating crudely the number of people from each location.This international participation points to the demand and need for blended learning faculty development around the world.This data was captured from an evaluation of email addresses, distinguishing 50 unique country codes.It is likely that the number of countries represented is larger as many participants registered using emails such as Gmail, Yahoo, or Hotmail which do not have the ability to determine country.
The welcome survey was distributed via Canvas Network (https://www.canvas.net/)and was completed by 1,195 participants.In examining the majority of participant demographics, 63% were female, 75% had at least a master's degree, 81% indicated that English was their primary spoken language and 80% were aged 35 or older.In examining where participants lived, 73% were from North America, 9% lived in Western Europe and 3% lived in Australia or the South Pacific.(It is interesting to note that the survey data from participants was limited to regions presented as survey items on the CanvasNet survey, while sifting through registrants' email addresses provided more detail on actual countries represented).Table 1 indicates participant motivation for enrolling in BlendKit2014 with the majority (84%) expressing an interest in blended learning or a desire to find resources related to blended learning (52%).Many indicated they were developing (30%) or teaching (16%) a blended course, or would be in the next year (39%).Others were curious about MOOCs (38%) or Canvas Network (26%).In addition, a separate question asked whether they had taught a blended course and 54% had not.In planning for the course and future iterations of BlendKit, we wanted to know what topics were most interesting or important to the participants.As Table 2 indicates, the majority (77%) were interested in content and assignment, with the least interesting topic to participants being quality assurance (57%).Clearly, however, the topics were all important to the majority of those who had registered for this course, indicating a need for blended learning resources and guidance for those involved with blended learning.Quality assurance 57

Engagement
Because we were interested in examining participation patterns of a unique open, online course, we polled participants regarding their expected engagement (Table 3).The majority expected to engage sequentially through all 5 weeks-a common pattern in "traditional" coursework, but not always the pattern in open, online courses.Anticipating that some would not participate in the entire course, we were curious as to whether time constraints or topics would be the discouraging factor.At least in this early time period of the course, participants indicated (23%) that time constraints would keep some from participating in all five course weekly topics as opposed to a lack of interest in the topics being presented.A small percentage (6%) indicated they were just there to check it out-perhaps these are the "lurkers" familiar in MOOC research.
Table 3. Participants' expected engagement in BlendKit2014?(n=1,195) Response Percent I hope to engage sequentially in all 5 weeks 64 I know I will not engage all 5 weeks due to personal time constraints 23 I know I will not engage all 5 weeks due to lack of interest in some topics 0 I'm really here just to check it out, not complete any topics 6 Other 2 The open nature of the course allowed participants to float in and out at their convenience-we did not expect participants to necessarily participate in a sequential, week-by-week manner.The BlendKit materials were available online including the readings and activities.The weekly, topic-specific seminars featuring guest speakers who were experienced in the weekly topics were broadcast to participants via Adobe Connect (guest presentations could also be accessed via the course's archive).Those who were unable to participate in "real time" could participate fully in the course except for the synchronous discussion during the webinar.Interaction was possible via social media or through the LMS tools within the week of a specific topic, or even after the week that topic was covered in BlendKit.
The synchronous nature of the weekly webinars did allow another glimpse into engagement by participants in examining how many actually participated in the webinars in real-time.Figure 2 illustrates that the first week Understanding Blended Learning was the highest attended webinar and attendance generally tapered off during the next four weeks, bouncing back for the final week of content on Quality Assurance.In addition, weekly surveys allowed us to poll participants about their engagement, namely which of the course components had they used that week? Figure 3 illustrates the distribution of responses for the five weeks.The responses clearly indicated the resources which were most frequently used by participants-readings and webinars topped the list.Note that the number of responses to the weekly feedback also dropped each week, signifying a challenging aspect of research in an open, online course in dips and drops for those interacting with the course and its correspondence-including research surveys.

Figure 3. Weekly participant self-reported engagement with course components.
In the last week of BlendKit2014, participants were asked to complete a survey evaluating their overall experience with the course and to provide feedback on what worked and what didn't.Anticipating that we had lost some participants during the course's duration, we chose to use the original emails provided during registration to request survey feedback as opposed to distributing through the CanvasNet LMS.This proved to be valuable at increasing the number of responses that were completed and allowed us to survey some registrants who had not engaged in the course.Of the 401 people who completed the survey, 69% said that they had originally registered via Canvas Network and also engaged in the course in some capacity.However, 26% of those who completed the survey admitted they had never engaged at all in the course.Table 4 illustrates that many of the participants were either developing a blended learning course (36%) or would be developing a course within the future (44%).Thirteen percent indicated they were not planning to develop a blended course.Only 20% of the final respondents indicated that they had engaged in all 5 weeks of BlendKit2014 (Table 5).However, for those who did not engage in the course, the majority (43%) indicated that it was due to personal time constraints or that they registered to merely check out the course (16%) -reiterating the "lurker" persona that appears frequently in open, online courses.The freedom to lurk in the course, uninhibited by requirements and pressures is definitely appealing to those who may be interested in the topic, but not in participating in a course per se.
Table 5. Participants' reported engagement in BlendKit2014 (n=386) Response % I engaged in topics all 5 weeks 20 I did not engage in all 5 topics due to personal time constraints 43 I did not engage in all 5 topics due to lack of interest in some topics 3 I was just here to check BlendKit2014 out, not complete any topics 16 Other (please explain) 9

Engagement with Badges and Credentialing
Badges were added to the BlendKit Course in response to participant complaints that they needed more structure and guidance on what was "expected" of them in the course.Ironically, the nature of the open, online course is to dismantle structure in some capacity, so that participants are free to interact as they choose with the course on their own timeline.However, we have seen with each iteration of BlendKit that for many this notion is foreign or discomforting.
In BlendKit2014, many participants (Table 6) indicated they did not receive a badge.Even more interesting was that 15% did not have any idea of whether they had received a badge during the course.When examining the activities completed by participants against badging criteria we found that 3,810 badges were issued (duplicate headcount), but only 59% of those had claimed their badge through Credly.This does raise the question of who are those who choose or want to "collect" badges?
Recall that we defined completers as those who completed at least one activity in each of the five content weeks.At course end, 174 participants had met that criteria.In addition, 84 people had paid for the credential certificate and 77 of those passed the portfolio review process to receive the credential from EDUCAUSE.

Satisfaction and Feedback
Looking at the data gathered on enrollment and engagement as BlendKit2014 progressed provided a glimpse of who had enrolled and how much and in what manner they interacted with the course itself.Given this course was designed to provide resources and guidance for those involved with blended learning, an important part of the feedback we sought was how well we achieved that goal and what we might change in the future.
Table 7 illustrates the feedback we obtained from participants at the conclusion of BlendKit2014 with regard to how helpful they felt the course had been both in helping them develop a blended learning (BL) course and at making them comfortable with blended learning in general.Overall, participants were extremely positive about the course and its helpfulness.and what they did not like.We classified the comments as being related to course content, resources, or materials; related to the management of the course itself-organization, structure, etc; or related to the components that were designed to allow participants to interact with each other as the course progressed.Participants were fairly evenly distributed in commenting on course content and management issues that they liked regarding BlendKit2014.Many mentioned the relevance of resources to their own course and the course reminders and structure.We continue to find it interesting that participants prefer an open, online course that is not "too open!"There is a need for structure for many participants.Perhaps this has been conditioned from our own educational course experiences.It seems that while participants could certainly choose to impart their own structure to interacting with an open, online course, many are uncomfortable doing so and crave the structure provided by an LMS, clear timelines and due dates, and even badges or credentialing.
Participants were thrilled with the free and usable resources.Comments such as "Creative commons resources-that is fantastic!Plus continuing access to the materials-that's wonderful!"reinforced their desire to accumulate usable and relevant materials that they could apply to their individual settings.
In addition, many were positive and appreciative of the effort put into the organization and interaction possibilities in BlendKit2014.Anecdotal comments included: "Even though I didn't engage all that much, I appreciated how organized you seem.The constant reminders and engagement from your end was nice."And, as a positive on interaction, "Guest speakers [and announcements and interaction with peers via social network (Twitter)]."We thought it positive, that in interacting in a blended fashion, participants were also able to better visualize how to apply this in their courses: "How everything is archived so participants can explore resources at their own pace and on their own schedules.As a faculty developer who works with faculty teaching hybrid courses, I got new ideas from how the facilitators interacted with participants."Certainly, many of the respondents embraced the open nature: "How well it was organized, and how we were encouraged to commit to as little or as much as we could.That made it easier to stay until the end." However, there were those who had a disconnect with the format of the course as can be seen by the largest distribution of negative comments related to the course organization and structure.Some had negative comments that carried over from past MOOC experiences: "bit confusing trying to find resources…just another issue of having to negotiate different platforms for MOOCs provided across the world ;-)."Participants had to figure out how to pace themselves within the course structure: "I found the pace of the course to be too fast for the amount of material presented.The first week I tried to complete everything and found that the work required approximately 15-20 hours.This was beyond what I could commit so began cutting back."While some craved structure, others found it cumbersome: "Deadlines; I realize they're necessary, but I wasn't able to meet both work and personal deadlines and those for obtaining credit for this course."And, there were those who failed to see how pedagogy related to blended learning -or somehow felt blended learning should be different than other courses: "I felt like the first weeks, in which I attended the webinar session and read the reader selections, were helpful for very general information about blended learning but there was more that I thought applied to general teaching than specifically to blended learning."

Future Directions and Thoughts
The open educational resource (OER) courseware that has undergirded past BlendKit MOOC cohorts will remain online publicly for use, reuse, and remixing by individuals and institutions pursuing blended learning designs under the terms of a Creative Commons license (https://blended.online.ucf.edu/blendkit-course/).Future cohorts may continue to be offered by the University of Central Florida based on these materials.However, the decision to offer such cohorts continues to be reviewed on a case-by-case basis in which the costs and benefits are weighed.At the time of this writing, a fourth BlendKit MOOC cohort ("BlendKit2015") was in production with a new set of facilitators.(The experience of facilitating such a cohort and the time commitment it takes to do so represent both a benefit and a cost.).A fifth iteration is tentatively planned for spring 2016.
For as long as such cohorts are offered, we anticipate continuing to collect a variety of data to gauge effectiveness and guide modifications to the course design.Of particular interest are data related to various types of participant engagement.The following questions guide this ongoing inquiry: • What motivates registrants to pursue "completion" of the open course?• What design structures will maximize and validate learner choices--independent of course "completion?"• What types of facilitation are most effective at engaging registrants?
We have made a point of gathering as much data as we can.However, it has become obvious with multiple iterations of data collection, that while many data points are possible, each has its issues and challenges.Being an open, online course allows for some analytics data to be collected.However, gathering these data meant relying on Instructure Canvas to both provide us with that data and the data format and items were under their control.Surveys always present the challenge of having to rely on the goodness of participants to both complete the survey and respond honestly and openly to questions posed.
In addition, we found it interesting that we also were monitoring and collecting "unorthodox" data points in terms of comments made during webinars, discussions, tweets, and other social media or interaction tools.Certainly, the argument that these are anecdotal at best holds true.However, in some cases they provided insights we might not have known otherwise.As an example, one tweet-"Hanging out watching #BlendKit2014 #LWTech (With some poll subversion going on.)" was accompanied by a picture of a room full of participants interacting with the course at Lake Washington Institute of Technology.This made us realize that individual campuses might have multiple people engaging with the course, which alludes to a potentially larger scale than can be captured through analytics or surveys.It also points to a promising and powerful method of bringing campus faculty and educators together around the theme of blended learning within their own context and institution.This is a great and encouraging look at engagement, but definitely unorthodox and difficult to capture in an evaluative sense.
We still stand by our mantra of gathering all the data you can in evaluating open, online courses.We have dropped the "massive" moniker as we feel the BlendKit Course was not intended to be in the likes of Coursera, EdX or Udacity offerings and our goal was to make open content that participants could freely take and use to improve blended learning in their world.Certainly, the benefit of having too much data outweighs the challenge of missed opportunities or lessons learned from uncollected data.
Overall, we feel that the BlendKit Course achieved what we set out for it to do.And, perhaps the best measure of engagement and satisfaction is the continued growth in enrollment, which essentially nearly doubled with each BlendKit iteration.Whether that trend will continue remains to be seen.But for the near future, it appears that the BlendKit Course fills a niche that is lacking in satisfying a need for resources and guidance for the growth of blended learning courses.

Table 2 .
Topics most interesting to participants (n=1,195)

Table 4 .
Time of participants' development of next blended course.(n=396)

Table 7 .
Overall, how helpful was BlendKit2014 in… In addition, we asked participants for more specifics on what they liked most and what they liked least about BlendKit2014 and their experience with the course.Many people were open about what they felt worked well in the course

Table 8 .
Aspects of BlendKit2014 Most Liked and Least Liked (Multiple Responses).