Customizable Modalities for Individualized Learning: Examining Patterns of Engagement in Dual-Layer MOOCs

Dual-layer MOOCs are an educational framework designed to create customizable modality pathways through a learning experience. The basic premise is to design two framework choices through a course: one that is instructor centered and the other that is student determined and open. Learners have the option to create their own customized pathway by choosing or combining both modalities as they see fit at any given time in the course. This exploratory mixed-methods study sought to understand the patterns that learners engaged in during a course designed with this pathway framework. The results of the quantitative examination of the course activity are presented, as well as the categories and themes that arose from the qualitative research. The results of the analysis indicate that learners value the ability to choose the pathway that they engage the course in. Additional research is needed to improve the technical and design aspects of the framework. participants to navigate the course in a way that best suits their particular learning interests by utilizing one modality, both modalities, or a custom combination of either modality at different time frames in the course. Following any pathway through the course would count as completing the course. A dual-layer MOOC might have an instructor-centered modality focused on traditional content delivery and discussion paired with a student-driven modality focused on networked and social learning. This study investigates the experiences of participants in the Data, Analytics, and Learning MOOC (DALMOOC), a course delivered with edX in fall 2014. Using a mixed-methods approach, shown to be preferred in MOOC research (Ga š evi ć , Joksimovi ć , Kovanovi ć , & Siemens, 2014), course participant patterns of engagement were analyzed to investigate the differences between participation strategies, as well as to identify participants that utilized different pathways through the course. After initial quantitative analysis of course participation activities, a subset of participants were invited to answer a set of follow-up open-ended survey questions in order to provide more depth to the analysis of their patterns of engagement. Additionally, online discussion postings and social media activity during DALMOOC were analyzed. The main goal of this study is to examine differences in participation strategies across both course modalities as well as to utilize study findings to refine, improve, and focus future research and design of the dual-layer model of MOOCs.

participants to navigate the course in a way that best suits their particular learning interests by utilizing one modality, both modalities, or a custom combination of either modality at different time frames in the course. Following any pathway through the course would count as completing the course. A dual-layer MOOC might have an instructor-centered modality focused on traditional content delivery and discussion paired with a student-driven modality focused on networked and social learning. This study investigates the experiences of participants in the Data, Analytics, and Learning MOOC (DALMOOC), a course delivered with edX in fall 2014. Using a mixed-methods approach, shown to be preferred in MOOC research (Gašević, Joksimović, Kovanović, & Siemens, 2014), course participant patterns of engagement were analyzed to investigate the differences between participation strategies, as well as to identify participants that utilized different pathways through the course. After initial quantitative analysis of course participation activities, a subset of participants were invited to answer a set of follow-up open-ended survey questions in order to provide more depth to the analysis of their patterns of engagement. Additionally, online discussion postings and social media activity during DALMOOC were analyzed. The main goal of this study is to examine differences in participation strategies across both course modalities as well as to utilize study findings to refine, improve, and focus future research and design of the dual-layer model of MOOCs.

Background
Dual-layer MOOCs have their genesis in the two main frameworks that arose from the different delivery approaches that have shaped MOOC growth over the past 10 years. The first framework is connectivism, which arose at about the same time as the first MOOC. In fact, the original course (to later be labeled as a MOOC) was initially designed to be offered online covering the emerging theory of connectivism (Fini, 2009).
Connectivism is a theory of learning proposed by George Siemens and Stephen Downes to address complexities in the learning process that existing theories did not fully address. According to Siemens (2005), connectivism is the integration of principles explored by chaos, network, and complexity and self-organization theories. Learning is a process that occurs within nebulous environments of shifting core elements-not entirely under the control of the individual. Learning (defined as actionable knowledge) can reside outside of ourselves (within an organization or a database), is focused on connecting specialized information sets, and the connections that enable us to learn more are more important than our current state of knowing. (p. 6) Connectivism is a student-driven learning framework in that it views learners as a node in a network of learning connections, with each connection and node being of equal value and power in the overall design of learning (Siemens, 2005).
In order to explore the possibilities of connectivism, Siemens and Downes offered an open online course called Connectivism and Connective Knowledge (CCK08). Since this course was offered to anyone that wanted to register, the total number of participants swelled to well over 2,300 (Rodriguez, 2012). The success of CCK08 led Dave Cormier to coin the term Massive Open Online Course in order to later describe the course (Dabbagh et al., 2016). For the following few years, most MOOCs were connectivist in nature (Daniel, 2012).
In 2011, Stanford University offered a different form of MOOC. A group of popular courses were to be offered online with open registration. One of the most popular ones was the Artificial Intelligence course with over 160,000 learners registered (Cabiria, 2012). Soon, hundreds of courses were offered based on the model that Stanford had implemented. Initially, Siemens viewed these courses with optimism, saying, "I love this! Stanford University Artificial Intelligence is being offered as an open online course" (Siemens, 2011, para. 1). However, it was soon noted that a distinct power dynamic informed the design of these courses (Anders, 2015;Daniel, 2012;Rodriguez, 2012). These new forms of MOOC were more focused on the instructor rather than the networked learner and involved watching videos and taking computer-graded tests over social interactions and collaborative knowledge building.
These new MOOCs were based on a broad framework of learning design called instructivism. According to Porcaro (2011), "instructivists, whether behaviorist or cognitivist, are ontologically objectivist and realist, and epistemologically empiricist…. They see learning as simply mapping the real, external world onto the minds or behaviors of the student" (p. 41). Instructivism can cover a wide range of instructional strategies, from information transfer to class discussions. The main focus of instructivism is that the teacher is in control of the course, including content to be covered, learning activities to be engaged in, and the final assessment of learner performance. These two frameworks, instructivist and connectivist, informed the design and delivery of the dual-layer DALMOOC.
In order to differentiate between the original form of connectivist MOOC and the newer form of instructivist MOOC, Downes coined the terms cMOOC and xMOOC (Kovanović, Joksimović, Gašević, Siemens, & Hatala, 2015). Connectivist MOOCs were labeled cMOOCs, while the more instructivist MOOCs were labeled xMOOCs, meaning "MOOC as eXtension of something else" (Downes, 2013a, para. 5). Stephen Downes differentiates between the two by stating that a "cMOOC is designed as a network…while an xMOOC is based on a central course site and content that will be followed by all students" (Downes, 2013b, para. 25).
In 2014, a MOOC on learning analytics was offered in the edX learning management system. As its name implies, edX is closely aligned with the MOOCs that some label as xMOOCs. A team meeting of over 20 people was held at the University of Texas at Arlington to design the Data Analytics and Learning MOOC (DALMOOC). At the meeting for this team (dubbed a DesignJam), the decision was made to mix cMOOC and xMOOC modalities in a manner that would support both frameworks. The dual-layer model emerged out of the work of this team.
The dual-layer model is based on several structural considerations. The main consideration is that learners can choose their preferred framework, instructivist or connectivist, at any point in the course (Crosslin & Dellinger, 2015). This is an expansion of the recent work in personalized learning that allows learners to choose their learning pathway rather than the instructor (Reddy et al., 2015) or technology (Lin, Yeh, Hung, & Chang, 2013). The second consideration was that the transition between these frameworks would be well described in a visual syllabus and supported through various centralized and distributed technologies and digital spaces (Crosslin & Dellinger, 2015). The visual syllabus is based on the idea of using a graphic representation of the course in order to assist learners with grasping the course concepts, similar to work by Nilson (2009) and Biktimirov and Nilson (2003). Finally, any personalized pathway that learners chose during the course would count as a valid approach to completing the course (Crosslin & Dellinger, 2015). This is an expansion of the work by Bell (2011) and Calvani (2009), which examines how different theories of learning have different uses at different times for different learners in different contexts.
In support of these considerations, several tools and design strategies were utilized to support learners in the course. The first strategy was to introduce a course metaphor to help learners understand the modality choices available to them in the course. The metaphor chosen for DALMOOC was based on the movie The Matrix, with a "blue pill" visual cue assigned to the traditional instructivist modality and a "red pill" visual cue assigned to the nontraditional connectivist modality. The second strategy was to create a visual syllabus that included these visual cues as well as graphic maps and charts to explain the course tools and structure. The third was to focus the course content on a set of open-ended competencies in order to give learners greater flexibility in navigating course structure.
Several different technologies were utilized to support both frameworks in DALMOOC. The instructivist modality was supported in the edX learning management system, with videos, content, and discussion forums specifically designed to guide learners through the recommended instructor content. The connectivist modality was supported by ProSolo, a competency-based networking tool that connects learners with each other and social networks like Twitter, enabling learners to create their own connections and competencies. Weekly Hangouts were organized by the course instructors, who also actively engaged with social media (mostly on Twitter, but some activity occurred on Facebook and a Google Group) and the edX discussion forums. Likewise, the course instructors encouraged the course participants to write blog posts about what they learned in or about the course. Additionally, instructors added three tools in the edX learning management system to help learners navigate the course content. First, QuickHelper is a discussion tool that analyzes forum-post content in order to connect learners with questions to learners who would have the answers. Second, Bazaar (https://www.cs.cmu.edu/~cprose/Bazaar.html) is a real-time communication tool that pairs learners into small groups and then leads them through a guided discussion script written to help them learn the content for that week. Finally, instructors embedded an intelligent tutoring system, authored with Cognitive Tutor Authoring Tools (CTAT) , into the edX platform through the use of the IMS Learning Tools Interoperability (LTI) specification. This CTAT-based tutor supported activities related to predictive modeling in Weeks 5 and 6 of the course.

Problem Statement
While there have been a few studies on DALMOOC (Dawson, Joksimović, Kovanović, Gašević, & Siemens, 2015;Rose et al, 2015;Crosslin & Dellinger, 2015) looking at the different aspects of student engagement, the researchers identified a need for more research into the participant experiences in DALMOOC through a lens of learner engagement. Previous studies examined data that the learners left behind in the course, but none of these studies questioned learners after the conclusion of the course in order to determine their opinions about how they engaged with the course or why they chose those patterns. Therefore, the gap in knowledge that was identified was the lack of insight into learners' opinions about why they chose the patterns that they engaged with in the course content and activities.

Methods
This study analyzes the activity of the learners in the tools that they utilized in DALMOOC. Additionally, several DALMOOC participants were selected to provide additional data via openended survey questions in order to gain deeper insight into their course activities and motivations. To accomplish these research objectives, this study examines the patterns of engagement and participant experiences in the 2014 DALMOOC. The study focused on participants that utilized both edX and ProSolo platforms. The research questions for this study are the following: 1. How long did participants engage in DALMOOC? a. How much time did they engage with the content through the completion of assignments/competencies? b. How much time did they spend watching videos? 2. Of the MOOC participants who logged into the edX platform, how many also utilized ProSolo? a. How much time did they spend in one platform versus the other? b. How many chose to utilize both platforms? 3. What were participants' experiences with and thoughts on the customizable pathways through the DALMOOC? 4. In what ways would participants refine, improve, and change the design of dual-layer MOOC models?

Participants
DALMOOC attracted more than 23,000 participants from around the world with various professional and cultural backgrounds. There was no cost to participate in the MOOC. Participants in this study certified that they were all over the age of 18 and were informed that their participation would be part of a research project. The course design was open to all registered participants, meaning that anyone who registered could see posted work and communication. Participants did not earn formal credit for completion of the course, and grades were not assigned. To receive a certificate of completion, participants were required to complete 70% of the 26 competencies available in the course.

Instrumentation and Data Sources
This mixed-methods study had two phases. Since the first phase focuses on quantitative data while the second phase collects qualitative data, this study is categorized as an exploratory mixed-methods research design (Creswell, 2011). The first phase analyzed stored data from the MOOC to quantitatively identify weekly (Monday-Sunday) patterns of engagement by users that participated in both pathways, as well as their social media engagement patterns. This analysis included general course and participant counts of completed assignments/competencies, logins, videos watched (split into quartiles), mean/average time active in the edX and ProSolo platforms, and counting and examining participant social media posts. ProSolo allowed course participants to link their Twitter account with their ProSolo account, which then aggregated their tweets in a manner that allowed other participants to read them. Likewise, through the use of syndicated web feeds (also known as RSS), ProSolo aggregated the participants' blogs, which were then shared with others through daily digests. These daily digests mirrored the functionality of an RSS aggregator called gRSShopper, which was developed by Stephen Downes and used in early cMOOCs. Python, R, and the igraph library in R were used to complete the analysis. The second phase identified 10-30 potential qualitative opportunities from disaggregated data by identifying MOOC participants that had varying participation patterns. Then the phase concluded by asking structured qualitative questions of the chosen participants about their use of social media and overall course engagement. Open-ended questions were administered through an online form in Qualtrics. A total of 12 MOOC participants completed the online questions. Survey participants were assigned pseudonyms to insure anonymity. None of the responses from the 12 participants were discarded.

Procedure
The first phase also consisted of descriptive analyses of selected statistics from the DALMOOC. Due to the open nature of the course, this study utilized convenience sampling by analyzing data from all registered learners in the first stage and then questioned all participants that self-selected to participate in the second phase. Data gathered from the open-ended questions in the second stage were coded for emerging patterns and themes using a content analysis approach. Content analysis was selected because it is a controlled analysis of text within the context that the text was originally communicated in, thereby preserving some of the advantages of quantitative content analysis from communication science (Mayring, 2000). Additionally, because theory and knowledge of the relationship between engagement and individualized learning in MOOCs is very limited, classical (or conventional) content analysis was conducted with the goal of inductive category development (Hsieh & Shannon, 2005;Mayring, 2000). For content analysis in this study, researchers utilized etic coding in order to identify categories and themes, which were then used to answer the research questions. Etic coding was selected because the researchers wanted to connect the new setting of a dual-layer MOOC to the existing theoretical frameworks of connectivism and instructivism. According to Lett (1990), "Etic constructs are accounts, descriptions, and analyses expressed in terms of the conceptual schemes and categories regarded as meaningful and appropriate by the community of scientific observers" (p. 130). Two of the researchers coded the categories and themes separately and then compared their codings for agreement. Finally, the questions also included constant-comparative analysis (Miles & Huberman, 1994;Strauss & Corbin, 1990) while developing a more detailed discourse analysis to examine trends and themes within and across the data set.

Results
The data for this study comes from DALMOOC, offered from October 20, 2014, through December 22, 2014. This 9-week course focused on an introduction to learning analytics methods and approaches, including the general learning analytics data cycle, social network analysis, text mining, and predictive modeling. The different pathways offered in the course were not completely separated. Competencies completed in ProSolo were submitted to edX. Similarly, in Weeks 5 and 6, ProSolo users logged in with their edX accounts for learning activities with the CTAT-based tutor, as the version of ProSolo at the time did not support IMS LTI integration, and access to the CTAT-based tutor was only possible through the edX platform.

Course Participation
The course attracted 23,330 participants, with 13,535 showing any sign of activity in the course (e.g., watching a video or submitting a competence). In the first week, 6,993 and 1,025 participants were active in edX and ProSolo, respectively (see Figure 1 and Figure 2). Participants were considered active when they performed any activity (e.g., viewed a video, posted a message, created a competency) within a given week. As typical in MOOCs, the participation rate dropped significantly in the first few of weeks, with around 1,400 participants being active in edX from Week 6 onward ( Figure 1). A similar pattern was observed in ProSolo, with around 60 participants being active from Week 7 onward (Figure 2).  One of the most interesting groups that emerged in this study were those who were active in both platforms over the whole or some segment of the course. Figure 3 shows the number of participants per week who used both platforms. As Figure 3 demonstrates, most of the ProSolo participants also used the edX platform, which was in large part due to how the course was structured. Also of interest is the percentage of active participants in both of the learning platforms (as represented in Figure 4 and 5). Originally, ProSolo had a higher percentage of active participants than edX (Figure 4). However, ProSolo weekly attrition rates were higher than in edX, which can be explained, at least to some extent, by the availability of the CTAT intelligent tutor exclusively in edX. However, the percentage of active participants from the previous week ( Figure 5) after the initial drop increases steadily from Week 3 onward. This suggests that attrition rates got lower and lower, and that there were populations of participants who regularly accessed both edX and ProSolo platforms.    (Kizilcec, Piech, & Schneider, 2013), participants tended to be less active as the course progressed. However, although the total number of active participants constantly dropped, it is indicative that the general pattern of participation remained the same. Specifically, the ratio between the number of participants who were engaged for one, two, or more days within a week remained mostly the same throughout the course. Within the first few weeks of the offering, the majority of participants engaged only during a single day in a week, with very few participants engaged for more than four days. Here, participants were considered active if they performed any activity (e.g., watching a video, posting to a discussion forum).
Toward the end of the course, there were no participants actively engaged with the course content for more than three days in a week.  Besides the general description of participant engagement patterns, this study also sought to discover different modalities of participation for those who engaged with both platforms ( Figure  7). The largest number of participants (18) were active for only one or two weeks, or for the first four weeks of the course (16). Only four participants were active on both platforms every week until the end of the course, while eight were active more sporadically until the end of the course. Finally, two participants were active on both platforms since the middle of the course (start of Week 5) until the course end. From these numbers, it is likely that largest two groups of participants active within both platforms represent course participants who aimed at investigating the advantages of different course modes (i.e., cMOOC vs. xMOOC). Specifically, it is likely that these two groups of participants were primarily interested in investigating features of both platforms before choosing one or the other for the remainder of the course (or dropping the course).

Open-Ended Question Coding Results
The qualitative research phase involved asking a series of 13 structured questions. The responses to these questions were coded for categories and themes. The categories that emerged are explained in Table 1. Some of the more important categories to note are that participants generally noted a positive experience in the course, that many participants recognized the benefits of the choices that the dual-layer model afforded, and that many diverse pathways were noted for a large number of reasons. However, many participants were also forthcoming with how the course or design could be improved. Many of the categories fell into these areas, which led to the themes that emerged during analysis. These themes are explained in the next section.

Categories
Supporting statements

Question 1: Can you tell me about your overall experience in the DALMOOC?
Interest in the dual-layer design "the novel combination of the two kinds of MOOCs that thrilled me" (Taylor) "I was very interested in the dual-layer approach at first" (Landry) Importance of instructor/peer interaction in course "I also loved the hangouts and when instructors interacted through social media" (Charlie) "The greatly motivating engagement of the facilitators enabled me to gain a deep impression of how advanced this field is already" Majority expressed some type of positive experience "This MOOC ranks at the top among the very best experiences" (Parker) "It was a great experience that I enjoyed very much -it was one of my favourite MOOCs" (Jordan) "To sum up, DALMOOC course was a great experience" (Charlie) Problems or frustrations with design/functionality of course design and tools "I do like the approach, but it simply felt like too many hoops at times" (Landry) "At first, I felt overwhelmed and I also found ProSolo difficult to manage, but the video tutorials [for the use of ProSolo] helped me a lot" (Charlie)

Question 2: What was your impression of the red-pill/blue-pill metaphor?
Most found metaphor helpful "It was a fine pop-culture metaphor" (Landry) "The metaphor made sense to me" (Casey) "The metaphor helped me to understand the course structure" Those that didn't find it helpful were confused over meaning of metaphor "Not being a movie-goer, I had to ask colleagues, and then I thought it fits" (Taylor) "I don't recall it being used elsewhere, so it was lost to me" (Parker)

Question 3: Can you describe your pattern of engagement in this course (how much you used the red-pill ProSolo pathway versus the blue-pill edX pathway)?
Most participants attempted both pathways "I would start with the blue pill to simply see the content.... After that I would go learn on my own and tinker" (Landry) "I used both pathways" (Charlie) "50/50" (Emery) Preference for blue-pill layer due mostly to not understanding ProSolo "I must admit, I never really found out how to fully utilize the feedback system of ProSolo" (Morgan) "My first impression was that ProSolo wasn't very intuitive when I tried to find the same contents of a week as in edX" (Jordan) Technical problems with ProSolo and integrated tools "I found the interface somewhat limiting" (Parker) "there were some bugs and so I had to use the edX platform to use discussion forum" (Charlie) Some preferred to start in edX and progress to ProSolo "after a while I've got to use more ProSolo because I didn't understand how to operate on it at first.... after a while I became more enthusiast of ProSolo" (Hayden) "I would start with the blue pill to simply see the content, to check my own understanding, and then see what I could do in ProSolo to demonstrate competencies/etc." (Landry)

Question 4: What was your rationale for using the pathway that you choose?
Likes safe, familiar, manageable linear pathway "the edX pathway was more familiar and offered more privacy" (Jessie); "I took the pathway I knew best (edX) and with which I already had good experiences before" (Jordan) "sometimes it was due to time, where I would simply do what was outlined in edX to move through the week quickly" (Landry) Tended to start in edX but ventured into ProSolo if time allowed "when I saw something that didn't interest me in edX, I would see what I could do instead in ProSolo…. if I had more time, ProSolo and self-directed activities" (Landry) "Get experience with both and take full advantage of the course"(Emery) Liked self-guided, social, exploratory connectivist pathway "I also like to interact with other students and search for people that share the same interests" (Charlie) "As a self-driven learner/researcher, I find the idea of creating peer-groups to spur on my studies fantastic" (Parker)

Question 5: What was your rationale for not utilizing a different pathway?
Unclear expectations and structure in the course "I did not understand much of the educational terminology used at the beginning of the course" (Jessie) "It was confusing and difficult for me to keep track of my progress" (Casey) "Time and unclear expectations for how my own work would actually be viewed as contributing to the 'grades' of the course" (Landry) Took too much time or effort to figure out new tools or structure "This was far beyond my ability at the time" (Jessie); "using multiple websites can be time-consuming" (Hayden) "It's difficult to filter what is valuable from what is not" (Hayden)

Question 6: What might be a reason or reasons a course designer/instructor would want to utilize multiple pathways?
Modality choice is helpful to address different learner preferences "different types of students might learn more in different settings" (Jessie) "some learners might be experienced self learners, others might need the feel of an active (small?) community -in our MOOC, the community in ProSolo was much smaller" (Jordan) "I think it was a good way to create my own experience" (Charlie)

Question 7: What might be a reason or reasons a course designer/instructor wouldn't want to utilize multiple pathways?
A theoretical reason might be that teachers prefer linear modality for control "the instructors prefer that students can be guided through a linear path of material" (Charlie) "if the instructor has a standardized, orderly, old-fashioned comprehension of what teaching should be, then only one path for everyone might be useful" (Hayden) A theoretical reason might be that due to context, other paths could be better in specific cases "it could be bring the students out of their comfort zone or simply if the instructors have a specific view on optimum learning" (Morgan) "learners who just want to take the course in order to learn the topic might be extra stressed when they don't understand the pathway logic (and don't care about it) and spend a lot of time searching" (Jordan) Question 8: What learning strategies, personal feedback, or motivational techniques did you utilize while participating in this course?
Instructor and social presence helped, especially on social media "I was very impressed with the excellent engagement of all course facilitators -that (and fellow students) made the difference for me to stay on course even in difficult weeks" (Jordan) "Twitter helped, and the hangouts were good when I could view them" (Landry) Personal learning goals of higher achieving students increased motivation "My initial motivation was intrinsic (interest in the topic Learning Analytics).... After some weeks when I had the impression that the course level of difficulty was o.k. for me, I decided to do the assignments in order to get the edX Honor Code certificate" (Jordan) "the main motivation probably is to think how learning this is feeding my identity as a lifelong learner" (Hayden) Question 9: Do you feel that you have a higher or lower level of control over your learning strategies, personal feedback, and motivational techniques? What rationale do you have for your rating?
Self-regulation improves learning experience due to background, interest, and desire for autonomy "in all MOOCs I feel that I have a lot of control over my learning strategies, personal feedback and motivational techniques" (Jessie) "I have the impression that I have a high level of control over my learning strategies etc. due to my profession" (Jordan) Other participants wanted more guidance, feedback, motivation, etc. from instructor/course "I do sorely lack in motivational techniques for additional learning, such as MOOCs" (Landry); "I feel I have a higher level of control but at the same time I was looking for some feedback of my understanding" (Morgan) "It seems to me that ProSolo helps to relieve the learning from this, as I perceive them, limiting factors" (Hayden)

Question 10:
In what ways did the dual-layer design of the course (including videos, tools, flow, activities, etc.) influence your learning strategies, personal feedback, or motivational techniques?
Participants were able to pick most meaningful path "because of this structure, the MOOC became deeper in meaning (not just another MOOC), the live feeling was more powerful, the hard work and excellent engagement of the facilitators was very visible" (Jordan) "although I preferred to use the edX environment as my primary pathway, I liked exploring the ProSolo environment and sometimes found helpful content there" (Jessie) Layer choice allowed for flexibility, selfregulation, and modality opportunities "I really enjoyed the way that I could choose the path and tools to enhance my learning" (Charlie); "it was like a bouquet of opportunities" (Morgan) "I was allowed to pick as much as I wanted, so my strategy was supported" (Taylor)

Question 11: In what ways did your learning strategies, personal feedback, or motivational techniques affect the layer of the course that you choose?
Interested in collaborating with others that share interests "I like to interact with other students and search for people that share the same interests" (Charlie) "I loved it and enjoyed finding the work of others who participated in a similar manner" (Parker) Wanted learning pathway tailored to personal preferences "I prefer a more structured approach so that is why I was most comfortable with the edX pathway" (Casey) "I love to read and search for other references in sites, blogs, MOOCs and so on" (Charlie) "I want to reduce the cognitive workload, the multitasking, and the information overload, no matter if I lose understanding in the process of the topics.... I believe that the traditional path provides this" (Hayden)

Question 12: In what ways did the design of the course (including videos, tools, flow, activities, etc.) affect which layer of the course that you choose?
Course design worked better for some more than others "some social activities were too much skewed towards the rapid end of the spectrum, and so I adapted my engagement to reduce this activity" (Taylor) "since I did both, I got the most of these materials available to my learning" (Emery) External or technical issues affected coursework "unfortunately (mainly due to time reasons) I didn't use the ProSolo functions to connect with others and only wrote some status updates" (Jordan) "the videos were too demanding, so I was able to skip it" (Taylor) Question 13: What else would you like to share that would be useful?
Interesting, great course experience "DALMOOC was a very interesting learning experience" (Jessie) "it was a great learning experience" (Charlie) "the idea of a final concept map project was inspired" (Jessie) Technical or design difficulties with ProSolo, Google Hangouts, and social connections "unfortunately (mainly due to time reasons) I didn't use the ProSolo functions to connect with others and only wrote some status updates" (Jordan) "the videos were too demanding, so I was able to skip it" (Taylor) Suggestions for improvements "the daily digest didn't seem to appear very often" (Jessie) "better documentation and stability of ProSolo" (Landry) "ProSolo is a project I would like to use in the future, but more work needs to be done to bring the interaction to a level capable of meeting the needs of diverse learners" (Parker)

Emerging Qualitative Themes
The preceding categories were analyzed through content analysis for similarities and synthesized into emerging themes. The two themes that were identified were "learners like customizable pathways that allow personal preferences, but this introduces time-consuming complexity" and "in order to support personal preferences, learners need scaffolding, as well as a course that is free from design or technical difficulties." A few categories did not fit into either of these themes but did not form a cohesive theme of their own.
The first theme includes participants that preferred the instructivist modality, the connectivist modality, or a mixture of both. Participants in general expressed appreciation for the ability to utilize their preferred modality, even if they seemed confused about the general structure of modality choice. In general, most were aware of the choices available to them and were thankful for the ability to make that choice. However, most participants were also aware of the additional time required to understand and make modality choices. This is an important learning design consideration that should be taken into account in customizable modality pathway design.
The second theme is closely connected to the first theme. The additional time that is introduced into modality choice is connected to dealing with design or technical limitations, or to participants scaffolding themselves by learning about considerations specific to each modality. However, these time issues do not have to be left completely up to the participants to overcome. As the discussion section will explore, there are many ways that course designers and instructors can work to leverage both of the themes uncovered in this analysis to improve the design of customizable modality pathway courses in a manner that improves the overall course experience for learners.

Discussion
At the time of its offering, DALMOOC provided participants a unique experience through the use of a dual-layer framework. Setting up the modules in a way that allows participants to choose their path through a course raises interesting implications for learning theory, design, and analytics. Some implications that need examination include 1. how instructivist and connectivist approaches coexist in the same learning space (e.g., an online course); 2. how learners interact with peers, content, and instructors/facilitators in a space that utilizes both approaches; 3. how learners choose to navigate a relatively free space that also includes a traditional, instructor-led pathway; 4. how to best scaffold courses using a dual-layer framework to elucidate the notion of customizable modalities without simultaneously overwhelming learners; 5. how different personal backgrounds inform selected pathways; 6. how to encourage learners to take ownership of how they learn; 7. how to integrate different approaches and tools in one space without creating excessive limitations; 8. how to develop the process of collecting, measuring, analyzing, and reporting data in this framework; and 9. how to use data to provide real-time and/or targeted, personalized feedback. A number of these implications are addressed by the results of this study, which provide valuable insight into the four research questions.
Regarding the first and second research questions (the length of engagement and the number of learners who spent time in both platforms), DALMOOC followed similar trends of MOOCs regarding the decline in participation over the length of the course (Onah, Sinclair, & Boyatt, 2014). This is not surprising given that increasing retention was not a core objective of the course, which was designed as a research project from the beginning of the development process. The decline in participation in edX, ProSolo, and simultaneous use of both platforms is similar to the instructivist pathway attaining and finishing with the greatest number of active users. The decline in the total number of connectivist pathway users from Week 1 to Week 9 was approximately 94.5%, whereas the instructivist pathway was approximately 79%. It is interesting to note that the decline in the total number of those who participated in both was almost identical to the connectivist path, at approximately 94.5%, even though the number of active participants using both pathways tried ProSolo more in Week 1. This may be reflective of the novelty of the connectivist approach and the ProSolo tool, as people tried it and either moved more into what they were more comfortable with or experienced a barrier, such as technology issues.
Additionally, the second question asked why someone might choose one path over another. The quantitative data illustrates greater use of the instructivist path overall, but there were a significant number of participants that tried the connectivist path or utilized both. The qualitative data suggests that there was initial interest in the idea of multiple pathways given their novelty and that the blue-pill/red-pill metaphor helped them visualize the innovative course structure. Participants believed that modality choice addressed different learner preferences, and some liked being able to pick their own path. Some participants used edX as the starting place because it followed a more traditional learning pathway (and was therefore perceived as being safer, familiar, and more manageable) and later moved into ProSolo. Those that tended to remain in the instructivist platform suggested that technical issues and unfamiliarity with the connectivist approach and new tools introduced a barrier. The time that it took to learn and complete tasks, coupled with new content, also reduced participant engagement in both pathways. The implications of this for practitioners is that any attempt to allow modality choices in educational settings should be well designed, explained, and supported in order to help learners feel comfortable with making modality changes. Additionally, practitioners should look at an instructivist modality as one that provides a safer, more familiar pathway for learners to either start with before making a new modality choice or restart with after choosing.
It is important to note that the participants took DALMOOC on their own time and not as a required course. It is anticipated that for-credit online courses utilizing the dual-layer framework would reveal different patterns. MOOCs typically have a very low completion rate, and the literature consistently notes that most people taking MOOCs are educated adults who are interested in learning certain skills or enrichment (Dillahunt, Wang, & Teasley, 2014;Ho et al., 2015;Jordan, 2015;Reich, 2014). Therefore, participants who receive no credit can pick and choose what they want to learn and ignore things that they perceive are not interesting or are too difficult to reasonably master. Given the differences in motivation and other driving factors, it will be interesting to see how dual-layer frameworks could operate in for-credit university courses.
Finally, the results do offer insight into the third and fourth research questions (pertaining to participant experiences as well as refining, improving, and focusing future research and design of dual-layer MOOC models). These results yield several implications for practitioners interested in flexible learning models. First, scaffolding is essential for learners (Hmelo-Silver, Duncan, & Chinn, 2007;Kirschner, Sweller, & Clark, 2006). This scaffolding must start at the beginning but also needs to be subtle enough to keep participants focused without distracting them from progressing, especially for those who are used to traditional, instructor-led courses. The qualitative data suggests some confusion with an explicit dual-layer explanation, as participants may be unfamiliar with terminology and theory, and might be resistant to trying new things. Second, there needs to be a central, integrated space for both layers to coexist. Crosslin (2015Crosslin ( , 2016a has mentioned the idea of creating a neutral zone, being a place (1) for MOOC participants to start; (2) that exposes biases, epistemologies, and ontologies behind pathways to help inform learners when they should choose one modality over another; and (3) allows learners to mix and match tools, services, and even courses through a learning map. This place could allow learners to fully understand what choices mean and provide a truly personalized experience (Dabbagh & Kitsantas, 2012). Additionally, having one space like this would help collect data more effectively and efficiently, eliminating the need to mine multiple sources. The challenge is creating the infrastructure in a way that is free of limitations, like many traditional learning management systems have. A third consideration is the improvement of tools to connect participants over the first iteration of DALMOOC. Participants mentioned that QuickHelper and Bazaar were limited, as few people were successfully paired in the course. The social and feedback features of ProSolo also proved problematic in terms of usability and number of users. The creators of these tools are aware of this feedback and have been improving their tools accordingly. Finally, learning analytics could help inform automated and personalized feedback in the course, as some participants wanted, but the open nature of the connectivist side challenges notions of uniform assessment.

Conclusion
The relatively new nature of the dual-layer framework creates several intriguing avenues for investigation into the learning process. Many learners are not accustomed to a level of choice and self-determination or the regulation that these choices require. This is highlighted by the emerging themes that spoke to how the dual-layer framework added complexity that introduced the need for higher levels of scaffolding and technical support to aid learners in adjusting to the new design. Therefore, many of the discussion points and themes uncovered in this study are subject to change as the idea of learning pathways gains acceptance in educational circles. Additionally, as this study highlights, new technology and design methodologies are needed to fully support true learner choice. The quantitative data indicated that many learners were interested in exploring different pathways to course completion, while the qualitative results revealed several areas for improvement in dual-layer design and technology in the future. Research into other duallayer courses, such as the Humanizing Online Learning MOOC (HumanMOOC), have yielded similar themes and recommendations (Crosslin, 2016b). As technology and design limitations are diminished, new insights into learner-centered design could possibly be gained through the customizable modality pathway design framework. Therefore, further research and experimentation with this framework is warranted in the future.