Development and Testing of a Roleplaying Gamification Module to Enhance Deeper Learning of Case Studies in an Accelerated Online Management Theory Course

Research has established that “cognitive rehearsal,” (CR) or the visualization of application of a behavioral response to a situation, can increase self-efficacy through vicarious experience, but is challenging to induce online. Online higher education curricula can include collaborative gamebased learning (GBL) in the form or roleplaying, which can facilitate CR. The purpose of this study was to develop an online GBL module to facilitate CR in online business graduate students learning management theory case studies, and to evaluate its ability to induce CR. A convenience sample of students voluntarily participated in a proof-of-conduct study using a mixed-methods design to gauge whether use of the module facilitated CR, a descriptive and thematic analysis was conducted. Data were collected from 106 students within two course sections in an online Master of Business Administration (MBA) program. The module was used as an optional supplement to a management case study assignment and approximately 50% of students used it. Scores on the assignment were compared between students who used the module and students who did not, and those who did scored statistically significantly higher (p = 0.0003). Students reported in a survey that the module induced CR about applying management theory. Students also expressed satisfaction with the module and provided feedback for its improvement in the areas of format and content. Our findings suggest that collaborative roleplaying GBL and other methods to induce CR in online higher education should be explored and formally studied.

The aim of this study was to develop an online gamified roleplaying module to facilitate CR for improving DL in online Master of Business Administration (MBA) students learning management theory case studies and conduct a proof-of-concept study of this module to evaluate how well it succeeded in encouraging DL and CR, and how the module can be improved. The specific research questions were the following: 1.
Does using a GBL online roleplaying module along with a case study result in higher levels of learning compared to students using the case study without the module? 2.
Do students who use the module report high levels of engagement with the module? 3.
Do students who use the module report high levels of satisfaction with the module?
This paper starts with a literature review, which begins by covering how deeper learning principles (DLPs) have been implemented in higher education curricula in both face-to-face (F2F) and online settings. This is followed by a focus on roleplaying as part of game-based learning (GBL) and a description of how the concept of cognitive rehearsal (CR) has guided the use of roleplay in teaching soft skills in higher education in clinical occupations. The literature review ends with a description of study designs used to assess the efficacy of GBL implementations.
Next, research questions are presented, then the methodology for the study is delineated. This includes a description of the online, asynchronous GBL roleplaying module designed around DLPs and intended to induce CR that we tested in an online MBA management theory course. It also includes a description of our proof-of-concept study design and mixed-methods data collection and analysis. Finally, our results are presented, and our findings discussed.

Review of Related Literature
Online learning has been increasing in higher education settings, and instructors are faced with the challenge of gaining skills at curriculum development and delivery in an online setting (Pereira & Wahi, 2017. One theory that has been used to guide the development of educational curricula both in F2F classrooms and online is deeper learning (DL) theory (Wickersham & McGee, 2008). Wickersham and McGee (2008) proposed ways to incorporate DL or learning that "engages the learner who actively explores, reflects, and produces knowledge," into online teaching (p. 74). To incorporate deeper learning principles (DLPs), the authors encourage designers of online curricula to induce "active learning" by using real-world problems as examples, and "engaged learning" and "engaged activities" by providing the user choices about how to use the online functions, and different ways to interact with the curriculum (Wickersham & McGee, 2008). In addition, the online curriculum should consider "learner context" and "learner ownership," providing independence to the learner to self-set goals and self-manage utilization of the online module for learning (Wickersham & McGee, 2008).

Deeper Learning Principles in Higher Education
We recently conducted a review of articles reporting the results of testing of DL educational modules used in both online and F2F higher education settings over the recent two decades and found that the modules tested were most directed at undergraduates (63%) (Pereira & Wahi, 2019). When we stratified by type of profession for which the students using the modules were in training, we found undergraduates who were studying a clinical profession were less likely to be exposed to DL educational modules compared to those studying an education profession (Pereira & Wahi, 2019). We also found that among the DL methods tested, only 9% included "interactive, game-based and simulation" activities, even though those are seen to reflect many DLPs, such as active learning, engaged learning, learner context, and learner ownership (Pereira & Wahi, 2019). Simulation specifically taps into the DLP of solving realworld problems.
There are reasons why different types of educational modules reflecting DLPs are implemented disproportionately across higher education, especially in an online setting. First, some topics are easier to teach online in such a way that students excel at learning the material. We conducted a different study comparing the uptake of didactic knowledge, technical skills, and role modeling in students in a F2F undergraduate computer information systems course to the online version of the same course (Pereira & Wahi, 2018). We found that while uptake was comparable between F2F and online, students overall excelled at different tasks in the F2F compared to online setting (Pereira & Wahi, 2018).
Next, certain DLPs are more intuitive to emphasize in higher education curricula than others. As an example, in our review of articles reporting research on modules incorporating DL in higher education, we found that modules emphasizing "collaborative peer learning," which were the most popular type of DL modules in the articles we reviewed, were implemented in similar rates in both F2F and online settings (Pereira & Wahi, 2019). We found that this bias towards certain types of DL modules over others influences how topics are taught using DL modules in higher education, in those topics where collaborative peer learning was easy to incorporate were more likely to have DL modules developed about them (Pereira & Wahi, 2019). As an illustration of this, in many countries, the COVID-19 pandemic provided a natural experiment in comparing F2F learning prepandemic to learning in hastily assembled online settings that emerged during the pandemic. In an article written about the medical student experience in Saudi Arabia, students found that collaborative peer learning increased during the pandemic out of necessity as courses moved from F2F to online (Hamad et al., 2020). While the authors anecdotally felt that this transition improved stress management and social learning (which is a DLP), they point out that scientific investigations need to be done about the true impact, as the peer learning strategy may not have been effective for all topics, and they could tell that not all students could adequately adjust (Hamad et al., 2020).
A third reason why different types of online educational modules reflecting DLPs are represented disproportionately across higher education is that some types of DLP-associated modules are easier to design and implement than others. GBL modules can be especially effective at reflecting DLPs but can be a challenge to incorporate into a curriculum successfully. One study of the implementation of a GBL module for teaching geography education called Farmtasia found that it appealed only to students who were interested in competition (Jong, 2015). While the Farmtasia module described was quite extensive, the article did not cover the actual development of the module in terms of cost, effort, and expertise needed (Jong, 2015). Another article describes very explicitly the design of a collaborative GBL module designed to encourage multi-stakeholder dialogue in rural communities in Zimbabwe and how challenging the actual design can be, even if the GBL module is not meant to be a competitive but a collaborative one (Perrotton et al., 2017).

Roleplaying as Part of Game-based Learning in Education
As a result of the diverse nature of these various challenges, certain types of DLPassociated modules are less commonly developed for online higher education settings. As alluded to earlier, a particular challenge has been encountered in developing roleplaying modules as a GBL implementation associated with DLPs in the online space in higher education, although new approaches are being seen, especially in online settings. A few recent examples can be provided here. In one report, as a response to COVID-19, a F2F patient roleplaying simulation in an undergraduate nursing program was changed to simulate a telehealth setting so it could be conducted online, and potentially be more authentic to a real-world situation (Whited et al., 2021). In a different study, pre-service music teachers were placed in the setting of an alternative reality game (ARG), where they learned social and managerial skills through interaction with live actors via telephone, email, and social media (Overland, 2017). Both types of roleplaying modules could be seen as collaborative GBL, although it could be argued that the module with pre-service music teachers had a greater gamified component than with the nursing students. Still, in both cases, there are actors involved providing a simulated experience, and the student and the actors worked together to use collaborative GBL skills to arrive at common goals (Wang & Huang, 2021).
Although collaborative GBL roleplaying educational modules are less likely to be developed for a higher education student audience, when they are, they are more likely to be developed around certain learning domains. Wang and Huang (2021) recently conducted a comprehensive review of articles reporting on collaborative roleplaying GBL modules intended for education, many of which were implemented in the higher education space. They included 22 papers in their systematic review and in their summary of these papers, certain learning domains were favored over others as the subject of these GBL modules (Wang & Huang, 2021). The most common learning domains included language, programming, scientific fields, and general collaborative problem solving (Wang & Huang, 2021). None of the articles reviewed reflected GBL modules aimed at developing social and behavioral skills in a specific domain, the way the nursing program simulation was intended to develop patient interaction skills, and the preservice music teacher simulation was intended to develop student interaction skills (Overland, 2017;Wang & Huang, 2021;Whited et al., 2021). Another observation that can be made about the articles reviewed is that many were missing basic information, preventing the authors from comprehensively appraising the module being described in the article (Wang & Huang, 2021). In a table summarizing the articles included in the review, the learning domain was listed as "null" on over a third of the articles reviewed (Wang & Huang, 2021). Further, some articles did not characterize the type of study participants, and therefore "null" was entered under the column "participants" in the table (Wang & Huang, 2021).
An additional observation that can be made about the GBL articles reviewed is that there is currently no standard way of assessing and comparing GBL educational modules tested with students (Wang & Huang, 2021). One of the criteria the articles needed to meet to be included in the systematic review is they needed to "provide empirical assessment of descriptions of the effectiveness of the game" (Wang & Huang, 2021, p. 90). While the article focused on summarizing the frameworks, components, and mechanics of various modules tested in the articles reviewed and not on the empirical assessments, it pointed out that there are various ingame metrics that can be gathered to measure the level of collaboration being engendered by the module (Wang & Huang, 2021). Further, the authors encouraged conducting analytics on elements of the modules that may include game logs, chat logs, and behavior logs, as well as conducting prospective data collection to gather data from users about their experience with collaboration, social learning, and knowledge uptake with respect to the module (Wang & Huang, 2021). This recommendation synthesized the different ways the authors observed that these items were assessed in learners using the modules in the articles reviewed (Wang & Huang, 2021).

Teaching Soft Skills Using Roleplay: Cognitive Rehearsal
A general focus of higher education is to develop "soft skills" in students, with soft skills being defined as the interpersonal, socio-behavioral skills that are required to apply technical skills and knowledge in a real workplace setting (Sousa & Rocha, 2017). To that end, several studies have focused on gamified roleplay in higher education students for the purpose of teaching soft skills. Examples include class roleplaying scenarios intended to improve project management skills (Sousa & Rocha, 2017) and skills interacting on a business team (Jabbarova, 2020). Consistent with previous findings, both examples were designed for F2F courses, and not for the online setting (Jabbarova, 2020;Sousa & Rocha, 2017).
A term from the clinical education literature is helpful to introduce here, and that is "cognitive rehearsal" (CR). CR is the mental visualization of one's application of a skill or one's behavioral response to a situation which can increase self-efficacy in applying the skill or handling the situation through vicarious experience (Longo, 2017). Roleplaying modules in higher education using CR have been aimed at nursing students to teach optimal ways to cope with interpersonal stress on-the-job, and as an effort to prevent bullying and incivility (Clark & Gorton, 2019;Iheduru-Anderson & Wahi, 2017). CR has also been the framework used in roleplaying educational modules used by social workers and surgical trainees (Delich, 2017;Finnesgard et al., 2016). As with the previous examples, these higher education modules with roleplaying to induce CR were developed for an F2F rather than online setting (Clark & Gorton, 2019;Delich, 2017;Finnesgard et al., 2016;Iheduru-Anderson & Wahi, 2017).
CR is a useful concept to apply to gamified roleplay in non-clinical fields, such as management, where the goals of the roleplay are collaborative rather than competitive in terms of GBL, and the intention of the activity is to develop soft skills (Wang & Huang, 2021). The concept of CR could be said to apply to gamified roleplay for developing skills for collaborative problem-solving, as was seen in some of the GBL modules described in two different proof-ofconcept studies of gamification in teaching various subjects to undergraduates in an F2F setting (Crocco et al., 2016;Dias, 2017). CR could also be used to characterize the student experience of participating in the virtual, online roleplaying module to teach software project management skills described in an article by Maratou and colleagues (2016).

Study Designs in Game-based Learning and Roleplaying Research
Studies on roleplaying as part of GBL or CR implementations are greatly limited by the educational context for which they were developed. As an example, in a study of a CR module to teach lateral violence prevention in nursing students, only eight students participated (Iheduru-Anderson & Wahi, 2017). The study was mixed methods in design, in that authors reported descriptive results from quantitative analysis, as well as themes from qualitative analysis (Iheduru-Anderson & Wahi, 2017). In GBL implementations, a study design like this is often called a "proof-of-concept study" (Crocco et al., 2016). Proof-of-concept study designs of educational modules are not intended to test whether there is superiority of the module in terms of teaching and learning compared traditional educational methods. Rather, they are intended to gauge whether the application of the educational module had any impact on the learning experience, positive or negative. This applies not only to the students, but the educators as well (Crocco et al., 2016). It is intended to provide initial evidence of utility, such that if utility is not realized, the module can be redesigned before being tested again.
Proof-of-concept study designs are admittedly biased, but efforts can be made in the study design to reduce bias for the purposes of getting a clearer picture of whether the novel educational module is effective. For example, in their research on incorporating GBL into undergraduate math, English, and science courses, Crocco et al. (2016) designed their study so that there were control sections of students that were not exposed to the GBL so that some comparisons could be made. By contrast, Dias (2017), who was testing a competitive GBL module on an undergraduate management class, used a pre-and post-test study design as well as comparisons with historical classes to assess the impact of the GBL implementation. As recommended by Wang and Huang (2021), proof-of-concept studies on GBL should propose research questions related to the desired learning outcomes and learning experience the GBL module is meant to induce and gather both quantitative and qualitative data to determine whether these outcomes were met.

Methods
We developed an online DL gamified roleplaying simulation module to induce CR in higher education students to support the learning in one unit of an online management theory course that is part of an accelerated online MBA degree program. We conducted a proof-ofconcept observational study with a convenience sample of students using a mixed-methods design to gauge whether use of the module facilitated learning, engagement, and satisfaction in the students who chose to use it. Students were offered to use the online module as a supplement to opportunities to learn the regular material in exchange for extra credit on an essay associated with the material.
Characteristics of those who chose to use the module were compared to those of the rest of the class to assess selection bias. Essay and course grades were compared between those who chose to use the module and those who did not (with extra credit points for using the module subtracted to ensure a fair comparison). Those who used the module were given a survey about their opinions of the module, which included both closed-and open-ended questions (see Appendix A for a complete list of survey questions). We analyzed the responses from the survey to answer the research questions and to assess whether students reported engaging in DL and CR as a result of using the module. This section describes these procedures in detail.

Study Population/Sampling
The study sample consisted of students enrolled in an online accelerated MBA program offered at a state university in the northeastern United States (Fitchburg State University, 2019). Each student took one of two consecutive online classes in Management Theory and Process (MGMT 9080). MGMT 9080 is one of ten courses required for MBA students concentrating in management. The course examines management theories in relation to how they may influence managerial practices. Each class was seven weeks long, and took place in Fall 2019, with the first (Fall A) taking place over September and October 2019, and the second (Fall B) taking place over November and December 2019. Table 1 summarizes the course plan for each section of the class. As shown in Table 1, there were three case studies assigned throughout the course (in Units 1, 3, and 5), and the online GBL roleplaying module was designed as a supplement to the Unit 5 topic, Total Quality Management (TQM). We chose to add the module to Unit 5, the last unit in the course where a case study was assigned, to allow students who used the module to compare their experience in Unit 5 with their experience in prior case study units that did not include a supplemental module. The Unit 5 case study assignment was based on a published case study about how British Airways (BA) implemented TQM (Madar, 2015). Briefly, TQM theory holds that to ensure quality throughout the business processes of design, manufacturing, and delivering services, every level of employee should be empowered to contribute to the improvement of quality, although the nature of their contribution will be different depending upon their role in the organization (Kiran, 2017). The case study described how BA trained their employees to engage in TQM at all levels and presented data tables reporting the impact of those efforts over time (Madar, 2015).
For all case study assignments, learning objectives were developed. The learning objectives for the Unit 5 TQM assignment were that at the end of this unit, students will be able to: 1. State at least two performance metrics that could be measured about an organization and explain how they could be changed through a management intervention. 2. Identify a TQM principle that could guide a management intervention at an organization designed to improve quality and state your rationale as to how the principle could guide the intervention. 3. Describe how two different departments in the same global organization might implement the same TQM principle differently to improve quality, and 4. State one of the steps to implementing TQM and describe challenges that may be associated with this step.
Each case study was associated with an essay assignment consisting of three essay questions designed to reveal the extent to which learning objectives were met. The essay questions for the Unit 5 assignment were: Question 1. In the case study, [a table in the case study article] presents metrics about BA for March 2008. Imagine you were on the BA executive team, which was meeting to set goals for these metrics for the next fiscal year in 2009. Imagine the team wanted to use TQM principles as a guide setting the new goals.
State two (2) metrics from [the case study table] that you think you could improve through a management intervention. Describe the management intervention, and how you think it would improve the stated metrics. Explain how at least two (2) principles of TQM relate to your management intervention.
Question 2. In 2010, although BA attributed their success at increasing inventory turnover and reducing number of employees as resulting from their TQM, unfortunately, they were suffering profit losses. Reflecting on how British Airways used TQM in the past, how might you shift the focus of the TQM if you were leading BA in 2010?
State one (1) TQM principle you would de-emphasize, and one (1) TQM principle you would emphasize and explain your rationale. Based on your choice of a priority TQM principle, suggest a management intervention that you think would bring a turnaround in profit and emphasize the TQM principle you stated. What metrics (aside from profit) would show you were successful at implementing the new TQM goals? [The table in] the case study can provide ideas for metrics, but you can come up with your own, as wellso long as they are measurable, and you can explain how they would be measured.
Question 3. In 2013, BA was doing well making a profit and having a high passenger load without many employees (see [case study table]). Part of their success was that they were able to keep their inventory turnover high, which they found was closely associated with a high passenger load factor and a larger number of aircraft. What TQM strategy would you recommend using to keep inventory turnover high at BA? What is your rationale for choosing this strategy? Which TQM steps do you think would be most impacted by this strategy, and why? Students uploaded their papers to the learning management system used in the course, Blackboard (Blackboard, Inc., 2021a). Each essay assignment was graded according to a scoring guide (see Appendix B for the Unit 5 assignment scoring guide). In addition, the Blackboard anonymous grading option "Grade with Usernames Hidden" was used to help eliminate grading bias (Blackboard, Inc., 2021b). Specifically, because the grader did not know which student she was grading, she was not influenced by factors such as the student's race, gender, previous performance, or perceived aptitude (Blackboard, Inc., 2021b).
The essay assignment for Unit 5 had a total of 25 points (see Appendix B). For the participating classes, students were told that they could earn 2.5 extra credit points (10% of total) for participating in the online deeper simulation module. The 2.5 extra credit points would be added to their grand total calculated from their scoring guide.

Module Format
The online deeper learning roleplaying module consisted of five web pages: an introduction page, three scenario pages, and a module completion page. The introduction page included text explaining how to use the module, the case study and background resources for the module, and module learning objectives. Additionally, the introduction page described five different departments that were the subject of the simulation: Human Resources, Finance, Operations, Engineering, and Customer Service. Each department was represented by a fictional executive leader with a back story (see Appendix C, Figure C1 for an example of an introduction page description of department and executive leader from the online learning module.) Each of the three scenario pages started by relating the events in a meeting of the executives of the five departments described in the introduction where metrics from the case study were presented. In addition, a challenge was given to the group by one of the executives (see Appendix C, Figure C2 for an example of a scenario introduction from the online learning module). On each scenario page, below the scenario description, each character provided both their thoughts about the scenario, as well as the dialogue they say in the meeting (see Appendix C, Figure C3 for an example of the thoughts and dialogue by a character in a scenario in the online learning module). In addition, below each character's thoughts and dialogue were audio versions, allowing users to listen to voice actors reading scripts (see Appendix C, Figure C3).
Finally, on the module completion page, there was a short debriefing video covering the learning objectives of the module and reviewing the scenarios. For students to get extra credit for completing the module, they were assigned to click on a link on the module completion page and fill out a survey in the online survey software SurveyMonkey. The survey collected their college email addresses to identify which students completed the module.

Module Elements Reflecting DLPs and CR
As described earlier, some DLPs that could be reflected in online curricula include active and engaged learning, learner context and ownership, and working with real-world problems (Wickersham & McGee, 2008). To inspire active and engaged learning, different types of media were presented involving text, a variety of images including data visualizations, video, and audio. To inspire the students to own their learning experience, the module was presented in a simple manner with navigable web pages that could be accessed in the order preferred by the learner. Also, the learner was not required to use any part of the module; as an example, learners could entirely skip choosing to listen to the audio or watch the debriefing video and could choose not to complete all three scenarios. Finally, to inspire authenticity, each scenario was built around a problem that was imagined from visualizing actual data presented in the published case study (see Appendix C, Figure C2).
The presentation of thoughts and dialogue in a meeting for each character about each scenario was designed to induce CR in the students. By explaining their dialogue through using their thoughts, the characters were intended to role model how CR may be used so that the students could understand this strategy and personally apply it if they wanted.

Survey Design and Data Collection
Two sets of data were collected in this study: data from all students in participating classes, and data from the survey completed by students who used the module. The following data were collected for all students in the participating classes: class membership (Fall A or Fall B), point score on Unit 5 essay assignment (before adding extra credit if earned), final percentage score in course (minus extra credit), and whether or not student used the online deeper learning roleplaying module. Consistent with other proof-of-concept study designs, demographics of the students were not collected, as demographic classifications did not relate to our research questions (Crocco et al., 2016;Dias, 2017).
For the second dataset, those who participated in the online module were able to earn extra credit points if they completed an online survey and included their email address for identification purposes (see Appendix A for survey). The survey link was presented at the completion of the online module, as described earlier. The survey included a quantitative and qualitative portion, and the quantitative part included two sections. The first section asked the respondent to rate statements about how the module helped the student achieve the assignment learning objectives on a Likert scale, where 1 = did not help me learn at all and 5 = helped me learn a lot. One statement per learning objective was included (total = 4) and rated on this scale. The purpose of these items was to gauge whether the respondent believed knowledge uptake was enhanced by the module.
In the next quantitative section of the survey, using a scale of 1 = strongly disagree and 5 = strongly agree, respondents were asked to rate twelve statements about how well the module performed for them technically. These included eight statements that were positive if answered in the affirmative (e.g., "The layout of the module was clear"), and four that were positive if answered in the negative (reverse-coded, e.g., "The module was too complicated"). In addition, respondents were asked if they thought the use of the module in future classes should be required (with possible answers: "Yes, it should be required," "No, it should not be required, but it should be extra credit," and "No, I do not think the module should be used at all"). These items were included to provide practical guidance as to ways to improve the module, and whether the respondents felt it was of any value to implement above and beyond what was already being taught in the classroom.
Finally, respondents were asked four open-ended questions about the module: comments about how the module could be improved to help them achieve learning objectives, any specific complaints about the module, any parts of the module that stood out to them as excellent, and what improvements they would recommend to the module. The purpose of this qualitative portion was to elicit evidence of DLPs and evidence of CR. We considered expressions of active engagement with the module, learner context and ownership, and practicing real-world problems as evidence of DLPs. We considered expressions of rehearsing future behavior as evidence of CR.

Data Analysis
R was used for analysis (R Core Team, 2014). To characterize the sample, a descriptive analysis was conducted on the data from all students in participating classes. Proportions of students in Fall A and Fall B choosing to use the online DL GBL roleplaying module were compared via chi-squared analysis. To attempt to answer the first research question as to whether use of the module was associated with higher knowledge uptake, using a t-test, student grades on the assignment were compared between those using the online module (before adding extra credit) and those who did not use it (with alpha set at 0.05). However, we recognize that this is a biased analysis in those students who have higher grades in general may have chosen to use the module. To assess whether use of the module was biased toward higher performing students, a correlation analysis was done between assignment score (without extra credit included) and final course score (without extra credit included). We used this analysis as a way to rule out the module, in that if a higher-performing group of students who chose to use the module did not demonstrate a higher knowledge uptake than the lower-performing group who did not, the value of the module would be seriously in question.
For the quantitative data from the survey questions, a descriptive analysis was done. Responses to Likert scale questions were visualized using the Likert package (Bryer & Speerschneider, 2016). The results of this analysis were used in part to answer the second and third research questions, which are whether students report a high level of engagement and satisfaction with the module. The results of this analysis were also used to troubleshoot and improve the content and implementation of the module.
Finally, qualitative data from the open-ended survey questions were analyzed using an approach recommended by Burnard and colleagues (2008) in which text from the answers is first coded into an initial coding framework, and from these results, themes are then assembled into a final coding framework. The results from this thematic analysis were also used to answer the second and third research questions. As part of answering these research questions, as stated before, we looked for evidence of DLPs and CR in the qualitative responses. As with the quantitative analysis, we also used the qualitative results as guidance as to how to improve the content and implementation of the module.

Overall Results
A total of 113 students participated in the MGMT 9080 classes taking place in Fall A (n = 59, 55%) and Fall B (n = 49, 45%). Of those, 108 (96%) completed the Unit 5 assignment and were therefore included in this analysis (see Table 2). Of the 108 who completed the assignment, 60 (56%) chose to additionally use the module and submitted a survey providing feedback. The proportion choosing to use the module did not significantly differ between classes (chi-squared p = 0.9140).
As shown in Table 2, the average score on the 25-point essay assignment (before adding extra credit) for Unit 5 was statistically significantly higher for those who used the module compared to those who did not (23.4 vs. 21.5, t-test p = 0.0003). However, because students were given the option of using the module, these results may be biased, as students who generally have better grades may have been the ones to choose the module.
To assess this potential bias, an analysis was done comparing final percentage score in the course to Unit 5 essay scores (with extra credit subtracted from both values). As shown in Table 2, those who used the module also had a statistically significantly average higher score in the entire course compared to those who did not use the module (96.5 vs. 91.6, p < 0.0001), lending support for the existence of bias. A scatter plot and correlation analysis support this theory (see Figure 1).

Figure 1 Scatter Plot and Correlation Analysis: Assignment Score vs. Score in Course
As shown in Figure 1, assignment and course grades were highly correlated (Pearson r = 0.68, p < 0.0001), and students who used the module were more likely to be higher scorers in general, as seen by the fact that they group into the upper right quadrant on the scatter plot. Figure 2 shows the Likert results for how the respondents rated how well the module helped them achieve the four learning objectives, and Table 3 decodes the labels used in Figure  2.

Figure 2 Likert Scale Plot for Course Objective Statements
Note. Response codes: 1 = Did not help me learn at all, 2 = Did not help me learn very much, 3 = Neither helped nor did not help me learn, 4 = Helped me learn a little, and 5 = Helped me learn a lot. Labels and their learning objectives described in Table 3 below.

Table 3
Labels and Items for Figure 2 Label Learning Objective

Two_Departments
Describe how two different departments in the same global organization might implement the same TQM element differently to improve quality.

TQM_Element
Identify a TQM element that could guide a management intervention at an organization designed to improve quality and state your rationale as to how the element could guide the intervention.

State_Metrics
State at least two performance metrics that could be measured about an organization and explain how they could be changed through a management intervention.
Steps_Challenges State one of the steps to implementing TQM and describe challenges that may be associated with this step.
In Figure 2, each horizontal bar represents a quantitative item from the survey, with labels on the y-axis on the left side which are decoded in Table 3. The items are arranged top to bottom starting with the largest percentage of students saying answer four or five (helped learn a little or a lot). These percentages are listed on the y-axis on the right side of the figure. The percentages responding four are visualized in the bar with light grey, and the percentages saying five are visualized in very light grey. Correspondingly, on the y-axis on the left side of the figure are the percentages of respondents answering two or one (did not help learn very much or at all), and these percentages are visualized in the bar in dark grey and very dark grey, respectively. The percentages of respondents answering three (neither helped nor did not help learn) are listed in the vertical center of the figure and visualized in a neutral grey.
In Figure 2, each item corresponded to a learning objective for the unit (presented earlier). As shown in Figure 2, users of the module overwhelmingly felt that the module helped them achieve the learning objectives, with between 87% and 90% indicating that it helped them learn a little or a lot for each objective. Figure 3 shows Likert scale results for the technical performance statements in the survey, and Table 4 decodes the labels.

Figure 3 Likert Scale Plot for Technical Performance Statements
Note. Response codes: 1 = Strongly disagree, 2 = Somewhat disagree, 3 = Neither agree nor disagree, 4 = Somewhat agree, and 5 = Strongly agree. Labels, student statements, and positive or negative indicators described in Table 4 below.

Table 4
Labels and Items for Figure 3 Label Statement

Indicates Positive Direction
Module_Loaded Each page of the module loaded easily for me. Agree

Easy_Find
It was easy to find what I was looking for in the online learning module. Agree

Text_Understandable
The text used in the module was easy to read and understand. Agree

Easy_Navigation
It was easy to navigate around the online learning module. Agree

Images_Attractive
The images used in the module were attractive. Agree

Well_Designed
The module was well-designed. Agree

Layout_Clear
The layout of the module was clear. Agree Device_of_Choice I could easily use the module on my device of choice (iPhone, laptop, etc.). Agree

Unnecessary_Time
The module took extra time that was unnecessary for learning the material. Disagree

Childish_Inappropriate
The module seemed childish and not appropriate for professionals. Disagree

Too_Complicated
The module was too complicated. Disagree

Unclear_Point
In the end, I did not really understand the point of the module. Disagree As described in Table 4, four of the twelve items were reverse coded, in that disagreement indicated a positive direction. As shown in Figure 3, between 79% and 93% responded with either somewhat or strongly agree to the statements where agreement indicated a positive direction, and between 57% and 83% responded with either somewhat or strongly disagree where disagree indicated a positive direction. No overwhelming technical issues were identified from these quantitative results. For module use for future classes, 32% recommended it be required, 64% recommended it be offered for extra credit, and 3% recommended it not be used.

Qualitative Survey Results
In the qualitative part of the survey (Appendix A), the students were asked to answer four open-ended questions addressing ways to improve the module to better meet learning objectives, any specific complaints about the module, any parts of the module that stood out as excellent, and what improvements they would suggest we make. First, this section will present evidence of the DLPs and CR that were provided in the open-ended responses from students. Next, the final coding framework with themes will be presented.

Evidence of DLPs.
As described earlier, the DLPs intended to be reflected in the module were active and engaged learning, learning context and ownership, and real-world simulation. We looked for evidence of these in the responses to open-ended questions.
Many comments expressed overall appreciation for the module in a general sense, saying it helped them learn the material. For example: Participant 20: I was struggling with the case study and decided to try this learning. It helped me very much. Thank you for including it.
While it was heartening to see many positive reactions, the lack of specificity in the statements made it difficult to understand which DLPs might have been induced by the model, if any, and how these might have led to CR or improved learning. One recurring theme expressed in the open-ended questions is that students liked the ability to have the text both in written form and in audio, and expressed that they liked the interactive nature of these functions: We interpreted these responses reporting use of the interactive elements of the module and the desire for more interactive elements as evidence of engaged learning. It also provided evidence of learning context and ownership, in that the students were choosing which interactive items to use, and how they preferred to consume the material. Other comments suggested that the authenticity of the real-world scenario was helpful:

Evidence of CR.
We also looked in the responses to open-ended questions for evidence of CR. While it was harder to assess if CR occurred, a few quotes suggested that it did: Taken together with the feedback about the authenticity of the module, it appears that at least some students who used the module identified with the characters in the module and were able to apply CR with respect to addressing the management challenges faced by the characters.
Final coding framework. Table 5 presents a summary of the final coding framework derived from analyzing these answers. As can be seen in Table 5, the comments seemed to relate to one of three main categories: comments on actual electronic format of the module and its components, comments about the content of the module, and feedback about other areas. For the format and content, the comments fell into categories of changes requested, as well as positive feedback.
As shown in Table 5, the first main theme had to do with format. The module contained a lot of content, so to reduce the cognitive load of the user, much of the content was hidden behind collapsible accordion controls that were manually opened when the user wanted to interact with the content. This turned out to be a problem, as it made the module confusing and difficult to navigate. A different approach to this design problem needs to be used when the module is upgraded. The other main comments about format change indicated that the module should be professionalized, with professional voice actors and real photographs of individuals, not the volunteers and basic electronic tools we used to make the prototype. This would be necessary to support the real-world authenticity associated with DLPs.
The second main theme had to do with content (see Table 5). As described earlier, students reported evidence of CR and DLPs, indicating that the content helped them envision applying the theory to real management problems. They reported being able to relate to the characters and finding it helpful that the characters roleplayed not only how they thought about the situation, but what they said in the meeting. This appeared to help deepen student understanding of how to work with other managers to apply a management theory to solve a problem. Further, students reported using the various electronic resources provided, which is evidence of DLPs. Many expressed positive feedback about the inclusion of audio and video, and even requested more, though some reported the module already had too much content and was too long. The main challenge with the content was having it apply more clearly to the topic. It would have been better to get a TQM expert to advise the module, rather than having the authors design it without this subject matter expert (SME). This aspect could be incorporated as part of upgrading and professionalizing the module.

Discussion
Our quantitative and qualitative results provide evidence that higher education students who used our GBL online roleplaying module as a supplement to their management case study essay assignment experienced higher levels of learning from using the module. The students reported in engaging with various features and aspects of the module and were satisfied with the module. Students using the module provided evidence of the DLPs of active and engaged learning, learner context and ownership, and solving real-world problems. Students also reported that the module facilitated CR around the management theory described in the case study.
One implication of these findings is that with some basic tools and careful design, online learning modules can be created that increase DLPs and induce CR when trying to teach the acquisition of management soft skills-although to professionalize a module like this would probably require a dedicated development effort similar to the one used to create the roleplaying module for rural communities in Zimbabwe (Perrotton et al., 2017). Recently, there has been a call to improve academic curricula in management to better ensure workplace readiness in students in terms of soft skill acquisition (Ritter et al., 2018). Considering workplaces as external stakeholders, Ritter and colleagues (2018) contend that academic management curricula should be redesigned such that they focus on developing teamwork-related skill sets in students so that they can demonstrate these soft skills in workplaces after they graduate. Additionally, GBL roleplaying modules have been proposed specifically for teaching soft skills in management (Acharya et al., 2018;Dias, 2017;Maratou et al., 2016;Sousa & Rocha, 2017). This suggests that use of modules such as the one tested in this study will grow in demand for use in higher education in management.
Another implication of these findings is that real-time roleplaying is not necessary to induce DLPs and CR in students. Traditionally, in F2F classrooms, case studies and management roleplaying can be done in real time. For example, Acharya et al. (2018) demonstrated in an experimental study that participating in a customized roleplaying assignment in a traditional higher education classroom was associated with not only applied learning, but greater enjoyment of the learning process. Clearly, the roleplaying approach in the traditional classroom can easily impart DLPs.
However, imparting DLPs using roleplaying by way of an online module to encourage CR is more challenging, because a design decision must be made as to whether students will be roleplaying with each other or the professor, as is done in an F2F setting, or with static characters, as was done in the prototype module tested. The disadvantage of the approach used for the module is that it lacks the social learning of the traditional classroom roleplaying scenario, which is associated with DL (Wickersham & McGee, 2008). Interestingly, the results of our study show that even though social learning was not incorporated, and students were not interacting with each other or the professor in real time, they still engaged in CR and DLPs, and received educational value from the exercises in the module. This suggests that even a roleplaying module with static characters, if designed with the proper format and content, can induce DLPs and CR in online higher education students.
Studies of online modules in higher education that increase the use of DLPs and CR are seriously lacking. As mentioned earlier, a review we conducted on studies of DLPs in higher education found that even now, traditional classrooms are studied at a higher rate than online ones, so the approaches being developed and tested cannot be deployed online (Pereira & Wahi, 2019). This may be true because it is likely more challenging to both develop as well as research an online roleplaying simulation module for higher education compared to developing and studying a roleplaying activity for deployment in a traditional classroom setting. Nevertheless, regardless of whether the module is online or F2F, the design of the module is important, because if it does not induce CR, the roleplaying aspect will have little value. As an example, Fominykh et al. (2018) created an online three-dimensional simulation environment for teaching professional counseling skills in an online higher education course. Compared to our module, theirs was much more technologically sophisticated, and appeared to have professors playing out two scenarios while electronically interacting with the students (Fominykh et al., 2018). The professors appeared to play scripted roles of a patient in an ethically challenging encounter with the student (Fominykh et al., 2018).
Because the authors collected both quantitative and qualitative data aimed at measuring "experiential learning," it was not clear from their results that students underwent CR as to what they would do when encountering a similar scenario in real-life with a patient, because CR was not the specific behavior studied. While the comments published in the article indicated that the roleplaying experience helped students prepare for a similar real experience with a patient, as we saw in our study, much of the feedback was centered around the technological aspects of the module (Fominykh et al., 2018).
It seems that in their attempt to make the module more realistic, the authors inadvertently introduced technical issues; this appears to be a tradeoff faced by online educational designers of roleplaying modules (Fominykh et al., 2018). Because these modules are so difficult to develop, it is imperative that they be improved such that they successfully induce DLPs and CR so that their educational value outweighs the effort needed to develop them.
Our study has both strengths and limitations. In terms of strengths, our module was designed based on DLPs for the goal of inducing CR, underwent rigorous field testing, and it was found that the module did encourage DLPs and the use of CR. Therefore, we were able to demonstrate not only how to make a successful online educational module like this, but how to successfully study it to receive actionable feedback. However, the study itself is limited in many ways. First, we only tested this module in two higher education classes at one college in the Northeastern US. We also found evidence of bias, in that higher performing students were more likely to volunteer to test the module. The module may perform very differently in a different student cohort and may have been successful partly because it was specifically designed for this population. Next, it is not clear that TQM was the optimal topic for a module like this, and from the feedback, it seems the use of an SME in developing module content would have improved the module. Also, before being used widely in higher education, the module format and content would need to be professionalized and upgraded.

Conclusion and Recommendations
In conclusion, we developed an online roleplaying GBL module to teach management theory to higher education students enrolled in an MBA program, and in the approximately 50% of students who chose to use it, evidence of DL and engaging in CR was reported. Feedback was given to professionalize and upgrade the module, which should involve improving the quality of the electronic resources and including an SME on the team to improve the content. More studies of online roleplaying modules for teaching soft skills that aim to increase DL and CR for applied learning in higher education, especially in management, should be conducted. This is because, with the rapid conversion of F2F classrooms to online programs currently taking place in higher education, especially in response to COVID-19, online resources that are shown to produce DL and CR in college students are currently in high demand, and few evidence-based options are available commercially.

Figure C2
Example Scenario Introduction from the Online Learning Module