Scaffolding a Culminating Assignment Within a Community and Task-based MOOC

We aim to understand the impact of scaffolds within a digital workbook to facilitate self-directed learning for learners completing a final project within a community and task-based MOOC. Optional reflection and articulation prompts were embedded in the tool support assignment development. Workbook use was prevalent, with 65% of learners using it to some extent. Our qualitative analysis revealed that assignment responses associated with substantial workbook use were A) informally written and loosely connected to assignment objectives (36%), or B) well-articulated and connected to assignment objectives (29%). Responses associated with little to no workbook use were C) superficial or uncontextualized (29%), or D) consistent with type “B” responses (6%). We discuss implications for instructors and learning designers in scaffolding complex projects in MOOCs.

Massive Open Online Courses (MOOCs) offer the possibility of "anytime, anywhere" learning, an appealing option for working professionals, lifelong learners, and even full-time students (edX, 2017).Based on an open-learning model, mainstream MOOCs provide learners with open access to learning materials, including instructional content from highly ranked universities from across the globe (Najafi et al., 2015;Hollands & Tirthali, 2014;Downes, 2008).Although critics have pointed to limiting factors of the model such as few opportunities for social interaction and basic assessment options (c.f., Reich et al., 2019), others have postulated that MOOCs hold promise to be a "rich landscape of learning" (Fischer, 2014) through pedagogical innovations such as social learning platforms (Ferguson & Sharples, 2014) and project-based learning designs (Pinto et al., 2020;Verstegen et al., 2015).
During the COVID-19 pandemic, interest in MOOCs increased, with providers seeing drastic growth and dramatically increased enrollments (Shah, 2020).Coursera, a prominent MOOC provider, witnessed the largest increase, with 35 million new enrollments from mid-March to the end of July 2020 (Lohr, 2020).It is apparent that learners have found flexible, new ways of learning that allow them to sharpen their professional skills and be responsive to the evolving workplace (Zhu et al., 2022).With this influx of MOOC learners, it is important to recognize that although MOOC platforms are built to accommodate large numbers of learners, instructors have limited opportunities to provide direct support to learners (Bali, 2014;Rohs & Ganz, 2015).To enable assessment at scale, MOOC platforms use auto-graded assessments (e.g., multiple choice quizzes) and peer-graded assignments to allow for personalized feedback (Kasch et al., 2021).Thus, with most MOOCs available on demand, learners progress at their own pace and must engage in self-directed learning to be successful and meet their goals (Zhu & Bonk, 2019).
Instructors can play an important role in facilitating self-directed learning through design choices that they implement in their MOOCs (Zhu & Bonk, 2019;Zhu, 2022).Although not prominently featured in the literature on self-directed learning in MOOCs, one such design choice is the intentional use of scaffolds that leverage learning technologies and software embedded in the learning experience (Saye & Brush, 2002).Scaffolds are conceptualized as any process by which an instructor or more knowledgeable peer provides assistance enabling less experienced learners to succeed in challenges that would otherwise be too difficult if attempted on their own (Wood et al., 1976).Studies that examine technology scaffolds in MOOCs have largely focused on fostering self-regulation and time management skills broadly for learners to successfully complete MOOCs (c.f., Gutiérrez-Rojas et al., 2014;Milikić et al., 2018;Pérez-Sanagustín et al., 2020;Sambe et al., 2018), rather than scaffolds created to foster reflection and task completion within course projects (e.g., peer reviewed assessments).This study considers how scaffolding prompts-embedded within a digital workbook tool in a resilient teaching MOOC-can facilitate self-directed learning within the context of a culminating, peer-reviewed assignment.Thus, this study offers an expanded view of instructor-led strategies for fostering self-directed learning using articulation and reflection technology scaffolds, filling a gap in the current literature on self-directed learning in MOOCs.

Objectives
Our overarching goal is to contribute to literature that identifies the "unique contributions of MOOCs to a rich landscape of learning" (Fischer, 2014, p. 7).We do this through our study of a MOOC on resilient designs for learning (Quintana et al., 2020), henceforth known as the Resilient Teaching MOOC.The course was offered at the onset of the COVID-19 pandemic to assist instructors and learning professionals who were planning for a year filled with challenges and uncertainty.In keeping with Fischer's (2014) call for learning scientists to explore innovative, multi-dimensional aspects of learning in MOOCs, the design of the Resilient Teaching MOOC aimed to bridge two design trade-offs that exist between cognitive and social dimensions of learning, and instructivist and problem-based, self-directed learning.
We characterized the pedagogical design of the Resilient Teaching MOOC as community and task-based, following Anders' (2015) model of MOOC learning designs.This model is typified by a combination of social and instructional support mechanisms and were instantiated in the Resilient Teaching MOOC in the following ways:

Community-based:
The instructor stated the objective of supportive social engagement through instructional videos and presented multiple opportunities for social interaction across discussion forums.
Task-based: The pedagogical design followed a task-based structure, with scaffolds contained within the course's digital workbook (e.g., prompts) indexed to relevant course content, leading up to a culminating assignment that required learners to synthesize and apply concepts from the entire course.
1.The specific objective of this study is to understand the efficacy of the taskbased aspect of the design by examining the impact of a digital workbook that contained prompts designed to foster reflection on course content and enable learners to articulate a resilient teaching plan (i.e., through a culminating assignment) in a stepwise fashion throughout the course.The course was designed to facilitate self-management and self-monitoring on the part of learners by employing articulation and reflection scaffolds.To understand the effects of this highly structured design, we pursued the following research questions.What happens when digital workbook prompts are used to scaffold a culminating assignment within the Resilient Teaching MOOC?
2. To what extent are assignment objectives met when learners choose to (or choose not to) adopt the digital workbook?
3. What are the characteristics of assignment submissions when learners choose to (or choose not to) adopt the digital workbook?

Literature Review
Massive Open Online Courses have long been associated with transfer-oriented pedagogies and self-paced learning approaches (Eisenberg & Fischer, 2014).Although some early MOOC designs (i.e., cMOOCs) promoted connectivism (c.f., Downes, 2009), the xMOOC model (i.e., cognitive-behaviorist approach) has largely eclipsed the cMOOC model in recent years.Fischer et al. (2014) offered a perspective for weighing the design tradeoffs that exist between these contrasting MOOC designs.The "rich landscape of learning" approach offers a range of antinomies-pairs of complementary truths, each of which is worth pursuing in different contexts all while presenting contradictions and tensions for learners and instructors (Bruner, 1996;Fischer et al., 2014).This rich design space centers on the following multidimensional aspects of learning: who, why, what, how, where, when, and with whom.Each dimension can be conceptualized in a "connectivist" or "instructivist design" and that choice offers certain affordances while coming at specific costs.Fischer et al. (2014) posited that the challenge then becomes to find ways to bridge these design tradeoffs to enrich learning designs.

Creating a Rich Landscape for Learning Through Hybrid MOOC Models
The work of forging a middle path as prompted by Fischer et al. (2014) in MOOC design is underway, with scholars thinking about integrating disparate typologies into hybrid models to allow for a more integrated and flexible approach.Such hybrid models are more consistent with existing MOOC designs, countering the narrative that MOOCs are "monolithic entities"' (Major & Blackmon, 2016).Lane (2012) outlined three MOOC typologies: 1) network-based, which are exemplified in connectivist designs where the focus is on socially-constructed knowledge through exploration of open educational resources and discussion; 2) task-based, whose designs center on skill acquisition and demonstration, with a secondary emphasis on community and social interaction; and 3) content-based, which are exemplified in extended MOOC designs (i.e., xMOOCs) where content acquisition is the primary objective, followed by networking and task completion.To acknowledge the integrated approach that already exists within many MOOC designs (Major & Blackmon, 2016), Anders (2015) built on Lane's (2012) typology of three MOOC types (network-based, task-based, and content-based) and proposed three hybrid models that could better account for the diversity of theories and applications that exist beyond the dichotomous categories of cMOOC and xMOOC.According to Anders (2015), hybrid models have the potential to "balance the strengths and weaknesses of the xMOOC and cMOOC models" (p.46) by attending to the needs of specific audiences and instructional goals.
All of the hybrid models expounded by Anders (2015) contained an elaboration of the basic typologies by Lane (2012).Anders' (2015) network-based hybrid model included a higher level of technological support and scaffolding.The original connectivist MOOCs (i.e., cMOOCs) were enacted through participant-initiated technology integrations, with a variety of technological tools and supports serving to structure the course in a somewhat ad hoc manner.The network-based hybrid model retained an emphasis on community-directed learning and inquiry, with the inclusion of a higher level of scaffolding and technological support.Networkbased hybrid models may be particularly valuable for professionals and lifelong learners, since they embody a learning environment that closely mirrors workplace environments, placing a high importance on personalized, professionalized, and situated learning (Milligan & Littlejohn, 2014).The introduction of scaffolding into "connectivist" MOOCs may encourage retention and progress-addressing the "drop off" phenomenon observed by Clow (2013)-potentially "unlocking uniquely valuable learning opportunities" (Anders, 2015, p. 55) for participants.
Content-based hybrids, as described by Anders (2015), use didactic content from MOOCs as the basis of a blended, interactive, and customized experience with a small group of learners.Content-based hybrids may leverage blended learning opportunities, by supplementing cohort-based, face-to-face instruction with digital content that was originally intended for a large audience of MOOC learners.In this way, they can be considered an expansion of xMOOC designs.Within higher education contexts, content-based hybrids can leverage high-quality instructional materials produced for at-scale learning environments and social learning experiences that occur within localized environments.In one example of a content-based hybrid, Ibrahim et al. (2021) described a "choose your topic" MOOC for a global audience of learners that was used as the basis of a small private online course (SPOC) enacted in a university setting.The MOOC included a wide range of topics, with lectures provided by over 25 nationally recognized faculty experts.The course was used as the basis of a two-week elective for secondor third-year pediatric residents.Students within the SPOC were required to complete all elements of the MOOC as well as additional in-depth readings assigned by the local course instructor.Students in the SPOC were also expected to participate in whole group discussions.SPOC instructors were able to create a differentiated learning experience focusing on a particular aspect of medical education, while using the MOOC materials as a foundation.
Finally, the community and task-based hybrid model described by Anders (2015) used project and artifact creation as a means of advancing skill development within a supportive learning community.The community aspect of this hybrid model was strongly rooted in sociocultural theories of learning that emphasized fostering dialogue and discussion amongst members of the learning community.Diversity of ideas were prized in an effort to advance socially negotiated forms of knowledge construction, similar to that of knowledge building models described by Scardamalia and Bereiter (2014).The model relied on extensive scaffolding and support structures to enable task completion as well as enabling social interaction.Mackness et al. (2013) described a community and task-based hybrid MOOC that was designed to support professionals in higher education transition into a non-academic career in industry.In their design, more active and experienced learning communities were instrumental in "creating the emergent spaces supporting connectedness and interactivity" (Mackness et al., 2013, p. 156).In a MOOC series focused on educational leadership, Quintana et al. (2020) advanced a related pedagogical model called self-directed/community-supported learning that enabled learners to develop professional competencies through applied work structured around an activity structure called "team practice."In the enactment of this pedagogical design, course designers and instructors aimed to draw diverse learners around the world into a community of discourse and practice through coordinated video content presentations, web-based enrichment activities, scenario-based team practice exercises, and community-wide discussion.Quintana et al. (2020) observed that more experienced and active members of the community acted as role models and guides, providing necessary support for learners who may have had less experience in educational leadership and policy.Similarly, in a MOOC focused on teacher professional development, Håklev and Slotta (2017) combined small-scale intense collaboration with largescale knowledge-building efforts through a set of learning activities and projects that were indexed to a community knowledge base.Other MOOCs exemplify the community and taskbased model, including those that lean toward the community aspect (c.f., FemTechNet White Paper Committee, 2013;Levine, 2013) and those that focus on the task-based aspect (c.f., Beaven et al., 2014;Mackness et al., 2013).In the present study, we elaborated on MOOC designs that embody the integrated community and task-based model, which are consistent with the design of the Resilient Teaching MOOC detailed in this study.

Scaffolding for Hybrid MOOC Models
Both the networked-based hybrid model and the community and task-based model require the implementation of additional scaffolds to support learners participating in these ambitious designs for learning.While the foundational scholarship on educational scaffolds was not describing support for at-scale learning environments, it is still relevant when considering the utility, type, and effectiveness of the scaffolds.Wood et al. (1976) established a key definition of scaffolding as a temporary instructional process where a more knowledgeable teacher or peer can control elements of a complex task in ways that allow the learner to focus on activity that is within their ability and ultimately engage in problems that would otherwise be beyond the learner's reach.The MOOC design context demands a modified approach given that course designs do not require that instructors take an active role in course enactment (Bonk et al., 2018).
Similarly, while learners may function as "more knowledgeable peers," this is not always a given, considering the range of experiences that learners may bring to a learning situation (Gregori et al., 2018).In addition, low participation rates in MOOCs (c.f., Clow, 2013) could inhibit the impact of peer support.
Thus, in the open, online space, course designers and instructors may opt to rely on what some scholars have called "hard scaffolds," which are static, anticipated, and planned supports based on known difficulties and challenges that learners are likely to encounter (Brush & Saye, 2002).Hard scaffolds can be introduced into an at-scale, online learning environment through course delivery platforms and integrated technologies, providing scaffolds to learners.Designers and instructors can make use of technological affordances to provide "hard" scaffolds that impact learners' understanding of not only new content areas, but also how they should think about completing a given task.Quintana et al. (2004) advanced a framework for technology-enabled scaffolds that were based on three processes of inquiry: sense-making, process management, and articulation and reflection.While these three scaffolding categories were initially conceptualized in a science education context, these categories can be more broadly applicable to other contexts and research areas.Process management scaffolds involve mechanisms that guide knowledge construction and strategies to steer investigation (Quintana et al., 2004, p. 358).Scaffolds of this sort are necessary in spaces where learners lack the insight and experience of a more experienced practitioner that would aid them in navigating complex processes and challenges.To this end, Quintana et al. (2004) posited that scaffolds should provide structure for learners' tasks while illuminating "what steps are possible, relevant, and productive" (p.359).Articulation and reflection scaffolds are necessary for learners to communicate inquiry findings and reflect on those findings to better understand one's strengths and weaknesses in terms of conceptual mastery (p.369).To support this process, instructional designs should encourage learners to articulate and reflect on their ideas in ways that are productive in the context of their respective fields of study (p. 370-371).Finally, sense-making scaffolding could be broadly construed as necessary for learners to reason about new ideas and concepts, to engage with representations that are part of a discipline, and to build on their intuitive ideas as they engage with new material.Each of these processes and their corresponding scaffolds involves engaging learners in tasks that are "cognitively complex and are often implemented in a social activity such as discussion, negotiation, and consensus-building" (Quintana et al., 2004, p. 341).While these scaffolding approaches can be conceptualized more generally to apply to different contexts, the work by Quintana et al. (2004) was more focused in exploring how scaffolding features can be developed for technology-situated learning tools and environments.This provides a perspective to consider how scaffolding features can be applied in online learning contexts.
Much of the work on scaffolds in MOOCs has been focused on supporting more metacognitive awareness by learners.For example, Sambe et al. (2008) used scaffolds in MOOCs to address known challenges of self-regulation to promote strategic planning and encourage consistent study habits.In other MOOC designs, scaffolds were provided to show feedback to learners about activity in the course and examine how these scaffolds affect performance and outcomes (Milikić et al., 2018;Pérez-Sanagustín et al., 2020).In another example, MyLearningMentor aimed to guide learners towards course completion by offering timely and helpful tips to help learners monitor their own work in productive ways (Gutiérrez-Rojas et al., 2014).There have been fewer published studies that explicitly make connections between scaffolds designed to support learners' reflections on course content and subsequent application to a new context (e.g., their own work settings).
In this study, we focus on the use of articulation and reflection scaffolds that are enabled through an LTI (i.e., hard scaffolds) in a MOOC on resilient teaching.As we will describe, these scaffolds were situated within a MOOC design that embodied a community and task-based hybrid model.

Methods and Theoretical Frameworks
Our theoretical frameworks are defined by the two components of the hybrid model that the MOOC design embodied: 1) community-based approaches to instruction and 2) task-based learning.
A long-held view espoused within the learning sciences is that learners play an important and active role in their own learning (Roschelle, 1997) and that learners learn best when activity is situated within a rich social context, which includes collaboration and exchanges with peers (Vygotsky, 1978).This view is complementary to the concept of a "community of practice" advanced by Lave (1991) and Wenger (1998;2011), in which learners engage in sustained and distributed learning in authentic contexts alongside more knowledgeable peers and mentors.A practical instantiation of this idea was realized by Brown and Campione (2002) in their model called Fostering a Community of Learners.In this form of pedagogy, an entire classroom community is engaged collectively, with well-defined learning goals for both content and practice, with each member responsible for contributing diverse perspectives and expertise to the advancement of a common goal.Although these theoretical frameworks did not originate in open, large-scale learning environments, they serve as inspiration for advancing social learning opportunities and productive peer-to-peer interactions within highly structured course designs.To enable such rich, social interactions within MOOCs, recent research in the learning sciences has explored the efficacy of designs that push on platform affordances (c.f., Quintana et al., 2020;Håklev & Slotta;2017;Slotta & Najafi, 2013).
In keeping with the "rich landscapes for learning" vision presented by Fischer (2004), MOOC instructors and designers are experimenting with more flexible open-ended tasks such as project-based approaches (c.f., Pinto et al., 2020;Verstegen et al., 2015).Such complex designs require elevated levels of support, but without the possibility of direct instruction, self-directed learning models should be considered.Garrison (1997) characterized three interrelated elements of self-directed learning: motivation (entering the task); self-monitoring (cognition and metacognition), and self-management (task control).Thus, to be successful within the context of self-directed, project-focused learning opportunities in MOOCs, learners must cultivate selfdirected learning skills, including self-management and self-monitoring strategies (Zhu, 2021).A vital consideration is the role that instructors can play in facilitating self-management and selfmonitoring skills in MOOCs through design choices.Instructors can create opportunities for learners to set their own learning goals, provide time frames and progress indicators, and offer flexible learning resources and peer assessments (Zhu & Bonk, 2019;Zhu, 2022).The present study examines the impact of carefully designed articulation and reflection scaffolds to support self-management (completion of the culminating assignment) and self-monitoring (reflection on course concepts and connection to relevant contexts).

Research Context
The Resilient Teaching MOOC is a four-week course, designed to support instructors at all levels who grappled with the realities of changing and evolving instructional contexts, brought on by the onset of the COVID-19 pandemic.The MOOC was situated as a "communityoriented" open, online learning experience (DeVaney & Quintana, 2020), where learners and instructors could come together, share experiences, and develop implementable teaching plans to address some of the difficulties encountered during the period known as "emergency remote teaching" (Hodges et al., 2020).The MOOC consisted of lecture videos, readings, discussion prompts, quizzes, reflection prompts, and a culminating, peer-reviewed assignment.The first part of the course presented a resilient design for learning framework, consisting of three principles: extensibility, flexibility, and redundancy (Quintana et al., 2021).Following lectures, quizzes, and reflection opportunities that delved into the principles, learners viewed a worked example that demonstrated all the principles in action.The remainder of the course focused on the development of learners' own resilient teaching plans, which were intended to be crafted and tailored to their own instructional contexts and to be used as a guide for both planning and implementation.

Digital Workbook
The digital workbook was integrated into the Resilient Teaching MOOC using learning technology interoperability (LTI) protocols at several points throughout the course.Each workbook prompt was indexed to specific course topics and activities within the instructional sequence, allowing learners the opportunity to pause and reflect on new information in small, related chunks which served as the foundation of the culminating assignment prompt (see Figure 1).The reflections drafted by each learner were saved to their own private instance of the workbook, and learners could review these entries at any point during the course.Additionally, learners had the ability to download selected entries or their complete collection of workbook entries to refer to once they completed the course.If learners opted to do so, they could share their workbook entries to a public gallery space in which peers could view and comment on one another's entries.The commenting functionality within the shared-response gallery space resembles a comment section that enables learners to utilize a text field to share reactions, offer feedback, and ask questions.The original entry author as well as other peers can reply to comments, creating conversation threads that serve to guide the original author's reflection on their understanding and application of course concepts.
Reflection prompts were embedded in a digital workbook and indexed to course topics (Appendix A).The course's instructor made the goals of the culminating project known from the outset, and the reflective prompts were designed to feed into the peer-reviewed assignment, aiding learners to construct a draft of their final project.The reflective prompts encouraged learners to carefully reflect on how each design principle could be applied in their work context.The reflection prompts were optional and ungraded.
Immediately preceding the culminating assignment, a textual description summarized course activities that led up to the final assignment and reminded learners that they could draw on their existing workbook entries.The passage also encouraged learners to take some additional time to refine their writing and to prepare a shareable resilient teaching plan.
The instructions for the final project asked learners to describe their context of teaching and learning, the interactions they desire to facilitate, and then to "explain how the principles of extensibility, flexibility, and redundancy are informing how you are thinking about facilitating those interactions."Learners were directed to review earlier reflection prompts that were most closely associated with project requirements (Appendix B).

Research Design
Our research design consisted of two phases (see Figure 2).In phase one, we gathered learner submissions to the culminating course assignment and analyzed the quality of those responses using our evaluation rubric (discussed in the Approach to Analysis section below).In phase two, we adopted and modified an explanatory sequential mixed methods research design developed by Plano Clark & Creswell (2011).This approach begins with "the collection and analysis of quantitative data" intended to address a research question, followed by the "collection and analysis of qualitative data" that builds on the results of the quantitative analysis (Plano Clark & Creswell, 2011, p. 71).We used quantitative methods to group assignments into a 2 x 2 grid, based on their rubric scores and number of workbook prompts completed.We used qualitative coding methods to analyze the characteristics of assignments in each of the four groups.

Data Sources
We collected assignments (n=80) submitted during the first four months that the MOOC was offered on the Coursera platform (n.d.), between June and September 2020.We eliminated one duplicate assignment, one plagiarized assignment, and one advertisement and arrived at our final dataset (n=77).We chose to analyze the first four months of learner data from the course because this timeframe represents a critical time early in the pandemic when instructors were still determining how they would implement online learning design plans intentionally (as opposed to reactionary measures, such as emergency remote teaching) and in preparation for the start of a new academic year.

Participant Backgrounds and Professional Contexts
Through a review of learners' assignment submissions, we were able to identify a range of learner professions, professional contexts, and subject areas.Most learners represented in our data set were educators (i.e., instructors teaching in a formal educational setting).Other professional experiences were represented as well, including professional training facilitator, physician, executive director, student, and instructional designer.A strong majority of the MOOC learners operated in higher education contexts, followed by several learners who worked in K-12 settings, and only a couple who were employed in the private sector.A wide range of subject areas were represented, from social science, language arts, education, and medicine to law, engineering, and computer science.Refer to Appendix C for more details on the backgrounds of the learners in this study.

Approach to Analysis
We analyzed responses to the final prompt (Prompt 5) from the resilient teaching plan: Explain how the principles of extensibility, flexibility, and redundancy inform how you are thinking about facilitating interactions in your course.We chose to analyze this prompt because it encapsulated the key learning goals of the course, which were to develop a nuanced understanding of the resilient design for learning framework and apply it to an authentic instructional context.To adequately respond to this prompt, learners needed to demonstrate a competent understanding of the principles of resilient designs for learning and the ability to apply those principles in their specific context.Other prompts available for analysis offered a much less holistic perspective of learners' level of content mastery.
Although the checklist style of rubric is an appropriate choice for peer assessment because it is easy to use and results in consistent evaluation, it was not sufficiently nuanced for our research objectives.Hence, we developed two analytic rubrics, which were more detailed than the instructor-developed rubric used for peer review.Both rubrics consisted of three categories (not addressed, addressed, nuanced reflection) and focused on the following aspects: assignment objectives (rubric one) and teaching context (rubric two).Rubric one assessed the extent to which learners met assignment objectives, making specific reference to the three principles of resilient design for learning.Nuanced responses also showed evidence of the application of resilient teaching principles within a specific context and provided specific examples of how targeted interactions were supported (Table 1).Rubric two focused on situational factors (e.g., points of failure, unknown situations) in teaching contexts, with reflections focusing on contextual factors that can be addressed by a learning design (Table 2).Using these rubrics, two coders independently coded the first 15 assignments in the dataset and achieved an interrater reliability (IRR) score of 0.67, a "moderate" level of agreement (McHugh, 2012).Through discussion, they came to a consensus and refined the two rubrics for greater clarity.They independently coded the next 15 assignments, reaching an IRR score of 0.85, a "strong" level of agreement (McHugh, 2012).The two coders each independently coded one-half of the remaining submissions.
We grouped scored responses into a two-by-two matrix, with rubric scores from low to high on the x-axis and workbook use from low to high on the y-axis.In other words, the responses were divided into four categories (see Figure 3) that represent a function of learners' use of the digital workbook and their overall ability to meet assignment objectives.

Figure 3 Four Types of Responses to Culminating Assignment Question Prompt
Responses that were associated with low rubric scores (i.e., left quadrants) were coded as having "addressed" or "not addressed" the assignment objectives.Responses associated with high rubric scores were coded as "nuanced" and positioned on the right quadrants.Since our coding focused on completed assignments and not the workbook entries themselves, we also wanted to consider whether learners whose assignments had received low or high scores had made significant use of the workbook.We considered "low use" of the digital workbook to be the completion of nine or fewer (of twelve) workbook prompts and positioned these instances in the lower two quadrants.We considered "high use" of the digital workbook to be the completion of ten or more (of twelve) workbook prompts and these responses were positioned in the upper two quadrants.We chose ten or more responses (of twelve) to represent "high" workbook use as this meant that a learner responded to 80% or more of the workbook prompts, and that they completed most of the workbook prompts presented in each course week.We reread assignment responses for each of these groupings and used conventional content analysis to analyze these groupings (Hsieh & Shannon, 2005), with codes derived directly from the assignment text.Our analysis resulted in a description of each quadrant or grouping of responses, as we will describe in the Results section.

Findings
Although use of the digital workbook was optional (i.e., not required for assignment submission or grading), 65% of learners who submitted a culminating assignment for peer review elected to use it to some extent.The following observations relate to our first subresearch question: To what extent are assignment objectives met when learners choose to (or choose not to) adopt the digital workbook?First, we will describe groupings of responses that were associated with high workbook use.We characterized 50 learners' (n=50) use of the digital workbook tool as "high."Of these learners, over half did not fully meet the stated assignment objectives (n=28) while 22 learners met the stated assignment objectives to a high degree through "nuanced reflection."Despite the majority of these responses not meeting assignment objectives directly, many of these responses showed evidence of reflection, as we will describe below in our content analysis.These groupings are presented in Figure 2 as Type A and Type B respectively.
Second, we will describe groupings of responses that were associated with low workbook use.Of learners who responded to nine or fewer workbook prompts (n=27), roughly three-quarters did not meet the assignment objectives to a full extent (n=22).A small minority of submissions exhibited nuanced reflection (meeting assignment objectives) but were not associated with workbook use (n=5).These groupings are presented in Figure 2 as Type C and Type D respectively.
While we cannot correlate workbook use and submission scores, we can make observations about the characteristics of the four response types, addressing our second research question: What are the characteristics of assignment submissions when learners choose to (or choose not to) adopt the digital workbook?
Type A responses (high workbook use, met assignment objectives to a low degree) were typified by informality and indirectness.Generally, they made a loose connection to assignment objectives but were incomplete in their articulation of how the three resilient design for learning principles were considered in the creation of a coherent teaching plan.Some focused on descriptions of specific situational factors of the instructional environments and others provided theoretical or "textbook" descriptions of the three resilient teaching principles.But all responses lacked an integration of context and design principles.From a structural and stylistic perspective, these responses tended to be disorganized in their composition, containing incomplete thoughts, akin to a draft or personal journal.
Type B responses (high workbook use, met assignment objectives to a high degree) were typified by completeness, coherence, and relevant detail.They contained a clear articulation and holistic view of how the three resilient design principles could work together to create a coherent teaching plan.They provided a complete view of the instructional context and carefully integrated theoretical perspectives with contextual factors.From a structural and stylistic perspective, these responses tended to be well-organized, thoughtfully composed, and written in a formal writing style.
Type C responses (low workbook use, met assignment objectives to a low degree) were typified by superficiality and lacking context.These responses often paraphrased course content and offered uncontextualized explanations, sometimes focusing on specific situational factors.The responses did not provide a clear articulation of how the three resilient principles work together to create a coherent teaching plan.Structurally, these responses tended to be brief.
Type D responses (low workbook use, met assignment objectives to a high degree) were very similar in their characteristics, structure, and writing style to Type B responses.The significant difference is that learners in this group made limited use of the reflection prompts available in the workbook.Our findings are summarized below in Table 3.

RQ1:
To what extent are assignment objectives met when learners choose to (or choose not to) adopt the digital workbook?
Of learners with high degree of workbook use (n=50): • 22 met stated objectives though "nuanced reflection" • 28 did not fully meet stated objectives Of learners with low degree of workbook use (n=27): • 5 met stated objectives through "nuanced reflection" • 22 did not fully meet stated objectives RQ2: What are the characteristics of assignment submissions when learners choose to (or choose not to) adopt the digital workbook?3 types of responses observed: • Type A (high workbook use, met objectives to a low degree)-informal tone, indirect, lacking coherence • Type B (high workbook use, met objectives to a high degree) & Type D (low workbook use, met objectives to a high degree)-complete, coherent, relevant detail, holistic view of course content • Type C (low workbook use, met objectives to a low degree) -superficial and uncontextualized, lack of cohesiveness with respect to course principles One unexpected finding was that many responses associated with high level workbook use seemed more like drafts than polished writing.Although the framing instructions for the assignment asked learners to "take some additional time to refine your thinking and prepare an initial resilient teaching draft," it appeared that many learners did not complete this additional step.Given the prevalence of "Type A" responses (i.e., high workbook use, with assignments meeting objectives to a low degree), we investigated the connection between workbook responses and assignment components.In doing this, we observed that several submissions simply were copied-and-pasted from the digital workbook into the assignment submission area and submitted for peer review.While still capturing the basics of a resilient teaching plan, the structure and quality of these submissions was incongruent with our requirements for the culminating assignment.As we have described, the writing styles of these submissions could be characterized as informal and incomplete.It appeared as if learners neglected to translate their initial ideas into a teaching plan that could be easily understood by their peers.Although the reasons for this oversight are not apparent through the analysis of our dataset, we speculate that this could have occurred for a variety of reasons, including time constraints, lack of awareness that final assignments were lacking in rigor and quality, and the "checklist" style rubric used for peer assessment.It may also be that learners were aware that the assignment rubric did not specifically address matters of structure and style (for reasons we have already articulated) and thus did not focus on these elements in their responses.

Discussion
Our review of learners' culminating assignment submissions provided evidence that many learners took advantage of the digital workbook prompts to incrementally develop their final resilient teaching plan over the duration of the course (self-management).Our findings show that a high level of workbook use (i.e., completion of reflection prompts) corresponded to high quality written assignment responses for some learners.There was a small group of learners who did not engage in workbook use and still submitted high quality responses, but our findings suggest that this activity pattern was an outlier, given that most learners who submitted high quality responses used the workbook (self-monitoring).The majority of learners who opted not to participate in reflection and articulation through the workbook activity submitted assignments that met project requirements to a low degree.In other words, it appeared to be beneficial for learners to engage with these types of scaffolded prompts, despite the number of assignments that corresponded with high workbook use and failure to meet assignment objectives to a high degree, as we will describe in Additional Findings.Workbook activities contributed to the learning process and served as a resource to support fulfillment of culminating assignment objectives.
These results offer evidence that within complex MOOC designs articulation and reflection prompts (i.e., hard scaffolds) (Brush & Saye, 2002) can effectively support learners' 1) self-management toward completion of a culminating assignment and 2) self-monitoring to connect course concepts to their respective contexts.In this way, the embedded prompts facilitated self-directed learning (Garrison, 1997), enabling self-management (i.e., task control) and self-monitoring (i.e., cognition and metacognition).Workbook activities contributed to the learning process and served as a resource to support achievement of culminating assignment objectives.
We would like to note that our characterization of Type A responses as meeting assignment standards to "a low degree" relates to the requirements of the rubrics we developed for this research where we set the bar high for explicit connection to course concepts and application to an instructional context.The checklist-style rubric developed by the instructor for peer-assessment in the course led to a successful assignment outcome for most learners (i.e., a passing grade).While the reflective prompts attempted to spotlight what learners should be thinking about and articulating during the development of their teaching plan, it seems that some learners may have needed models (e.g., worked examples) to help them to see what a more complete response should look like, or reminders to help them see how they may need to iterate to develop more polished work.In other words, the reflective prompts may have helped some learners see what directions to go in, but further support may be needed to help them continue to work productively.Providing worked examples or other types of model artifacts was a capability of the digital workbook tool used.However, the project team did not anticipate how necessary making use of this capability would be given that this was a novel endeavor and therefore lacked any precedent to inform this design decision.Additionally, the project team was constrained by time limitations and bandwidth issues brought on by issues surrounding the COVID-19 pandemic.

Study Significance
This study examined the impact of carefully designed articulation and reflection scaffolds (Quintana et al., 2004) to support self-management (completion of the culminating assignment) and self-monitoring (reflection on course concepts and connection to relevant contexts).Although prior research has examined instructors' strategies for facilitating self-directed learning in MOOCs (c.f., Zhu & Bonk, 2019;Zhu, 2021), less attention has been paid to the use of articulation and reflection scaffolds to directly support self-directed learning in MOOCs.We have shown that articulation and reflection scaffolds can be effectively integrated into learning sequences through technology tools, opening opportunities for instructors to embed reflection and articulation prompts directly within a course.This possibility allows for instructors to include complex, open-ended projects, such as those that develop professional skills and competencies.While earlier work has relied on "companion" resources in the form of websites or fillable PDFs (c.f., Lambert, 2015;Quintana et al., 2021), our study shows that tighter integration of reflection prompts within a learning sequence can benefit the learning process.We have highlighted the utility of such prompts being tightly integrated into a learning design to support assignment development and completion, and reflection on course concepts and relevant contexts.
Our study also contributes to the larger conversation about hybrid MOOC models, specifically community and task-based designs (Anders, 2015).As part of the ongoing pursuit to better understand effective means for integrating flexible, open-ended tasks that support projectbased pedagogies (Quintana et al., 2020;Håklev & Slotta, 2017), this study forges a path for instructors and designers seeking to develop more rigorous and relevant MOOCs, responding to the demand for high quality instruction that serves the demands of today's evolving workforce (Zhu et al., 2022).Future instructors and designers will be able to draw on this approach to further refine the practice of facilitating self-management and self-monitoring that promote selfdirected learning in a MOOC setting.
While not the focus of the research questions and analysis of the current study, the findings are situated within the broader context of community-based approaches to instruction in open online learning environments.The stated goal of the Resilient Teaching MOOC was to foster supportive social engagement, which included the opportunity for learners to share their own workbook entries and to provide early feedback on teaching plans to peers.In this way, course design embodied social learning and community-oriented pedagogies (Lave, 1991;Vygotsky, 1978;Wenger, 1998) and contributes to the recent body of learning sciences research that explores productive peer-to-peer interactions within complex, technology-enabled course designs (c.f., Quintana et al., 2020;Håklev & Slotta;2017;Slotta & Najafi, 2013).

Limitations
As this work is situated within the MOOC space, our dataset is limited by learner autonomy in interacting with content and corresponding low learner completion rates (Khalil & Ebner, 2014).Because the course was developed to be a resource for educators as they prepared to teach in the COVID-19 pandemic, learners could pick and choose parts of the course that were relevant to their needs and gain valuable insights without necessarily completing the entire course.This challenge exists across all massive open online courses, as one of their primary affordances is self-paced, self-directed learning (Eisenberg & Fischer, 2014, Zhu & Bonk, 2019).As a result, the actual number of MOOC learners who interact with all course lessons and earn a course certificate is relatively low in comparison to the large number of active learners present in a course (Khalil & Ebner, 2014).
Our data analysis was further hindered by the fact that learners in the Resilient Teaching MOOC were only presented with the opportunity to complete reflective digital workbook entries throughout the course.In other words, the completion of workbook entries was not a project requirement for the final assignment.Additionally, given the burden of preparing instructional content for an uncertain academic year, learners may have prioritized other planning efforts over responding to the work of peers and completing a time-consuming, peer-reviewed assignment.
After reflecting on the size of our data set and the possible rationales explaining the smaller than expected sample, we revisited the design of the Resilient Teaching MOOC to survey what scaffolds and directions were present that served to guide learners' use of the digital workbook tool.Our review revealed initial references to the community-oriented nature of the course that were not sustained throughout the course and a basic overview of the digital workbook tool that mentioned the capabilities of learners to share their entries and comment on their peers' shared entries.While these statements could certainly be interpreted by learners that they should make use of the digital workbook and embrace the community ethos by sharing their work and offering input on other's entries, these efforts did not result in the desired outcome conceived of in the provision of the digital workbook as a scaffolding tool.

Implications for Future Research
Our findings lead to a question for instructors and learning designers about what additional support may be needed to help learners realize levels of completeness and quality required to meet assignment objectives, particularly in an open, online learning environment.Future research could focus on the design of the prompts themselves, encouraging instructional teams to pay close attention to the wording of the prompts, with particular attention to aspects of articulation and reflection (Quintana, 2004).Furthermore, consideration of additional kinds of scaffolds (i.e., process-oriented scaffolds) and frequency of use is needed to move closer to realizing the desired learner use of the digital workbook as a learning tool (i.e., consistent use and additional efforts to refine and polish workbook entries before submission).Drawing on user-experience design methods (Schmidt et al., 2020), course designers could implement learner-testing approaches that would elucidate the clarity and effectiveness of the prompts, before the introduction of these "hard scaffolds" into a MOOC.In future research we plan to explore techniques to create a tighter coupling between reflection opportunities and assessments, including using scaffolds to guide learners through formalizing their workbook entries into more complete, formal drafts and how to leverage peer feedback to refine their work.These efforts would allow us to deepen our understanding of how instructors and learning designers can play a role in facilitating self-directed learning in MOOCs.
Identifying Potential Points of Failure (Week 2) We have defined designing for redundancy as identifying and/or creating interchangeable elements that could function if one or more aspects of the course plan fails due to perturbations in the learning environment.
• How are you considering this idea of redundancy as you plan your course?
When designing for flexibility we can think about the following kinds of questions: • The guiding principles of resilient design for learning are intended to be a tool for thinking about your course design.Like most design tools, they are not necessarily meant to be worked through in a linear order.One principle informs another and it may be necessary to revisit one or more multiple times as you work through your course design process.
In previous journal entries, you have considered each principle individually.
Now as you begin to think about putting your course plan together for the peer-graded assignment, describe how you might be thinking about the principles working together: • What new questions emerged as you worked through each principle?What ideas might you need to revisit? • How is one principle informing another?
• How are you capturing your design ideas and decisions?What forms of representation might be useful to share with your peers?
If you have created useful representations of your course design plans (e.g., tables, flowcharts), please consider publishing this journal entry to the gallery.

Figure C2
Plot depicting the various subject areas the focus participants of the present study work in.

Figure C3
Plot depicting the different professional contexts of the focus participants of the present study.

Figure 1
Figure 1 Digital Workbook Alignment with Culminating Assignment Prompts

Table 1
Rubric Used to Assess the Extent to Which Learners Met Assignment Objectives

Table 2
Rubric Used to Assess the Extent to Which Learners Addressed Situational Factors in Teaching Contexts When you consider your design plan, can you identify areas that are "brittle" or particularly vulnerable if one or more elements failed?• How can you minimize dependence on certain tools or activities so that if those features are lost due to a disruption, your class will still largely work?• How might you identify alternative ways of facilitating desired interactions?