Comparisons of Synchronous and Asynchronous Discussions in an Online Roleplaying Simulation to Teach Middle School Written Argumentation Skills

In this study, different degrees of synchronous and asynchronous online social interactions are investigated in the context of an online educational roleplaying simulation game that is played across multiple classrooms simultaneously to teach argumentation skills and social studies. Results from 45 K–12 middle school social studies teachers and 867 students over 3 study conditions were compared based on the degree of real-time discussion that was embedded in each condition’s version of game (i.e., two scheduled live conferences, one scheduled live conference, and asynchronous-only interactions or zero live conferences). All conditions exhibited significant small to moderate-level pre-post effect sizes, including the condition featuring asynchronous-only discussions. Additionally, the “mid-range” 1 live conference condition exhibited the greatest pre-post effect size in comparison to the other two conditions. This study demonstrates evidence for the benefits of implementing asynchronous-only discussions in digital interventions in comparison to live discussions when synchronous interaction may not be feasible. For designers, implementing both asynchronous and synchronous interactions based on available resources and feasibility can be used to maximize social presence among participants in educational roleplaying games and other virtual learning environments.

Riel, J., Lawless, K. A., & Oren, J. B., (2020).Comparisons of synchronous and asynchronous discussions in an online roleplaying simulation to teach middle school written argumentation skills.Online Learning, 26(4), 146-167. DOI: 10.24059/olj.v26i4.3468For over 20 years, a central policy initiative for K-12 education has been the effort to promote student skills and interest within the STEM disciplines (Committee on STEM Education, 2018;NRC, 2014).Researchers and policymakers have repeatedly issued warnings of a great shortage of workers to meet STEM career openings and that working within the modern knowledge economy requires development in strong scientific and technological literacy skills that should begin as early as the elementary and middle grades (English, 2017;NRC, 2011NRC, , 2022;;van den Hurk, Meelissen, & van Langen, 2018).To meet this need, governments, researchers, and policymakers worldwide have continually advocated for more STEM education offerings to engage students with socio-scientific content (Newcombe et al., 2009;Scogin et al., 2017).Specifically, these groups have called for teaching students not just the content of STEM disciplines, but also to develop essential cognitive skills for using content, such as critical thinking, problem-solving, and argumentation (Van Laar et al., 2017).Such skills are frequently cited as necessary for success in the STEM and knowledge-economy workforce where digital information is now ubiquitous, of varying quality, and from multiple perspectives (Noroozi, Dehghanzadeh, & Talee, 2020).
Among this call for critical STEM skills training within schools is the mastery of argumentation and the skills for evaluating and generating arguments to succeed in navigating the deluge of information that is encountered in everyday life (NRC, 2014).To this end, argumentation is often cited as an essential life skill for success during this age of information ubiquity (Bathgate et al., 2015;Kuhn, Hemberger, & Khait, 2016a;Özdem Yilmaz, Cakiroglu, Ertepinar, & Erduran, 2017).Additionally, it has been argued that the teaching of argumentation skills provides opportunities for robust learning experiences in any discipline and for any career, as argumentation establishes relevant active learning contexts for teaching subject content instead of teaching through rote memorization of facts and conceptual definitions, particularly in social studies (Cavagnetto, 2010;Iordanou, Kuhn, Matos, Shi, & Hemberger, 2019).
Research on the differences between asynchronous and synchronous social interactions is particularly important for providing insights toward the design of learning environments.This is especially the case in which the learning objectives are skills that are best developed in social situations like argumentation training, as it takes at least two people to hold an argument.Although asynchronous activities have always existed in K-12 through homework assignments, or, more recently, through out-of-class communications with teachers via media applications, the effects of asynchronous-only interactions in educational interventions that are deployed in K-12 schools are only recently becoming more regularly studied (Loncar, Barrett, & Liu, 2014;Lowenthal & Dunlap, 2020).
To contribute toward this literature, this study examined GlobalEd, an online educational roleplaying simulation game designed for middle school social studies classrooms.Originally designed to have both synchronous and asynchronous components for play among students across multiple classrooms, a recent edition of the game featured and investigated the effects of exclusively asynchronous-only discussions without any synchronous component.For this study, we evaluated whether an asynchronous-only condition was beneficial to students in comparison to versions of the game with synchronous discussions.Specifically, we experimentally investigated how two different live-discussion conditions compared to an asynchronous-only condition in terms of observed effects on students' argumentation skills.As argumentation is best learned in a social space that allows for regular dialogue between participants, the efficacy of an asynchronous-only design could dramatically increase the flexibility and design potential for social learning interventions.

Background Argumentation as a Cross-disciplinary, Socially Learned Skill and Mechanism for Learning Disciplinary Content
Of the many skills that are necessary for scientists to be successful, mastery of argumentation and scientific reasoning are often cited as priorities for STEM instruction (Kuhn, Hemberger, & Khait, 2016b;McNeill, Lizotte, Krajcik, & Marx, 2006;Sandoval, Enyedy, Redman, & Xiao, 2019).Argumentation, as it is frequently used in the STEM disciplines, is more than just having disagreements with people (Andriessen, Baker, & Suthers, 2003).As the research on scientific argumentation and STEM career skills has grown over the last three decades, argumentation skills and the ability to critically analyze arguments have increasingly been cited as required critical skills within large-scale educational reforms and standards for socio-scientific literacy and competency within STEM disciplines, such as the Next Generation Science Standards (NGSS Lead States, 2013), the Common Core State Standards (CCSS, 2010), and the National Curriculum Standards for Social Studies (NCSS, 2010).
Indeed, the practical aspect of scientific communication of findings and persuasion through argumentation achieves a core function of the scientific process.However, additional benefits can also emerge when students are engaged with argumentation.Participants not only persuade others of their explanations, but they also engage in a collaborative and social process of understanding the content being argued (Coffin, Hewings, and North, 2012).Importantly, engaging with argumentation encourages students to confront, analyze, and refine their own understandings as well, such as that which has been demonstrated in the growing body of research that adopts the approach of Arguing to Learn (Andriessen, Baker, & Suthers, 2003;Bathgate et al., 2015).Within this approach, although students are simultaneously developing their argumentation skills, they have also been observed to develop critical thinking skills, writing skills, and the ability to learn content knowledge across domains as a direct result of engaging with argumentation processes (Kuhn, Hemberger, & Khait, 2016a;Suephatthima & Faikhamta, 2018).Additionally, because information is more readily available for retrieval at a moment's notice in today's digital landscape, it has even been suggested that the ability to interpret and analyze facts and concepts is perhaps more important than simply knowing these facts (Van Laar et al., 2017), a role for which argumentation training is well poised to support.
When learning skills like argumentation that are inherently grounded in social interaction and require the consideration of multiple perspectives, repeated practice within authentic social contexts is often seen as a necessary condition for learning such skills (Crowell & Kuhn, 2014;Iordanou et al., 2019).Otherwise, as argumentation is fundamentally a process that occurs between two or more people, any attempts at learning these skills without discussion or collaboration deprives learners of experiencing the authentic, situated contexts in which the skills are used (Noroozi et al., 2012).For instance, simply learning facts about argumentation or its structure does not sufficiently prepare students for engaging with actual argumentative tasks, as it lacks the opportunity to experience the transactive back-and-forth dialogue that underlies the process (Mercier, Boudry, Paglieri, & Trouche, 2016).Therefore, argumentation instruction is necessarily situated in social interaction: the practice of making and analyzing arguments always occurs between at least two people (Mercier et al., 2016;Scardamalia & Beriter, 2006).As a result, a consensus among argumentation scholars is that these skills are necessarily taught in socially rich environments in which participants regularly engage in dialogue with each other and conduct argument analysis, construction, and feedback in a back-and-forth, transactive way (Henderson et al., 2018).
To this end, social processes such as argumentation require learning environments that enable social interaction to fully learn how to perform the skill.Especially in the post-pandemic educational environment, it has become increasingly important for researchers and instructional designers to create learning environments that can leverage the unique opportunities provided by digital technologies to enable authentic discussions and other social interactions, albeit at a distance.When people cannot be physically present together, synchronous and asynchronous online social discussions can be employed to provide spaces for socially intensive learning activities (Mercier et al., 2016;Noroozi et al., 2012).
Furthermore, online interactive approaches might afford unique conditions, opportunities, and motivations for learners that are not otherwise present in face-to-face learning contexts.In recent reviews, highly social online learning environments for teaching social skills such as argumentation have shown promising results; however, there has been virtually no research performed on the modality differences between face-to-face and various online, computermediated social interactive modalities for teaching argumentation (Asterhan & Schwarz, 2016;Lowenthal & Dunlap, 2020).The unique technological affordances for online socialization, including synchronous and asynchronous online discussions, should thus be further researched to maximize the potential for online learning in both K-12 and higher education (Henderson et al., 2018;Nussbaum, 2021).

Considering Simultaneity of Social Interaction and Social Presence for Online Learning Designs
The timing by which someone interacts in an online space may matter just as much as whether it is socially interactive in the first place.Knowing not just whether someone is expected to interact in a learning space, but also when someone is expected to interact are both primary components of the degree of "social presence" within an online Community of Learning (Garrison, 2016).The construct of social presence within a Community of Learning framework argues for the required presence of rich social interactions among learners in online learning environments.Opportunities for social interaction can activate the interpersonal and transactive processes that are essential for learning and meaning-making processes, such as discussing and determining the meaning of phenomena and concepts, debating concepts, and encountering other points of view to refine one's own understanding (Kozan & Richardson, 2014).Toward this focus on social presence, it has been regularly observed that the expectation of the degree and timing of which participants will interact will often influence variations in the type of behaviors that are exhibited in learning environments (Chen, Park, & Hand, 2016;Coffin, Hewings, & North, 2012;Koehler et al., 2020).
Varied expectations by the learner of the timing and simultaneity of responsiveness from peers in the social setting may determine the types of responses, depth of thinking, and included content associated with a given learner's participation (Cui, Lockee, & Meng, 2012;Foo & Quek, 2019;Larrain, Freire, Lopez, & Grau, 2019;Peterson, Beymer, & Putnam, 2018).Additionally, technology-based supports and scaffolding may be more readily implemented in asynchronous online activities than those requiring more real-time adaptations and assistance (Jeong & Joung, 2007;Jeong & Fraiser, 2008;Lin, Hong, & Lawrenz, 2012).Furthermore, although the inclusion of real-time interactions might create a more immersive and engaging environment that requires the participant to be cognitively attentive, such real-time expectations could demand more of the learner's attention, as well as be taxing on teachers who face various classroom and scheduling constraints when implementing live, synchronous interventions (Cui, Lockee, & Meng, 2012;Nieuwoudt, 2020).
It has become increasingly necessary given the post-pandemic educational landscape to investigate the effects and mechanisms connected to different levels of social presence within online learning environments that rely on social interactions.Although live social interactions in an online intervention have regularly been assumed to yield better results, such interactions may not always be feasible for a teacher to implement.This is especially true in situations where students may be having discussions or otherwise collaborating with people outside of a physical classroom.Various classroom constraints are typically present and teachers often need flexible options, or at least options for students to engage with environments outside of their scheduled classroom time or in a virtual manner.

Online Educational Simulation Games (ESGs) and Roleplaying: Enabling Flexible Implementation of both Synchronous and Asynchronous Discussions
The use of educational simulation games (ESGs) and interactive roleplaying is one approach that is well-suited to provide rich contexts for social interactions and exposure to social studies concepts in an authentic way (Devlin-Scherer & Sardone, 2010;Liu, Cheng, & Huang, 2011).The use of simulations as educational interventions is certainly not new, but advances in digital technologies over the last two decades have enabled the virtualization of both physical and social processes in ways never possible before.ESGs and roleplaying games that specifically model social processes (Gredler, 2013) can allow players to interact with social forces and assume the role of actors within the system through authentic roleplaying.In such games, players are assigned roles with specific goals within a simulated social event or system that models realworld social phenomena (Sauve et al., 2007).When a social simulation is additionally integrated with game mechanics, players, as agents in the game, gain clear goals on how to win the game, a set of rules for interactions and allowed player "moves" in the game, and feedback mechanisms (e.g., points, penalties) to guide their play and improve motivation (Brom, Stárková, Bromová, & Děchtěrenko, 2019;Vlachopoulos & Makri, 2017).Thus, authentic roleplaying in this manner allows for deep and authentic investigation of the forces and concepts under study within the game and to foster opportunities for social interaction to grapple with skills that are socially learned, like argumentation (Squazzoni et al., 2014).
Although modern ESGs and roleplaying games that model social processes can be played both in-person and online, online games are particularly timely for social studies education in today's post-pandemic world due to their ability to provide uninterrupted continuation of gameplay both inside and outside of the classroom.As seen from the widespread school closures as a result of the global COVID-19 pandemic, effective online interventions that facilitate ongoing interactions among students and teachers can be valuable in the situation of school closures or student absences from school.As they are educational interventions that can enable motivating synchronous and asynchronous modes of social interactivity, ESGs are well-poised to permit continuous dialogue and collaboration among students in their own class based on the teacher's pedagogical needs.

The Present Study: Observing Effects of Variations in Simultaneity in the GlobalEd Game
Studies have been performed recently between the varying degrees of simultaneity in online social interactions in K-12 learning environments, generally showing that both synchronous and asynchronous interactions, such as online written discussions, among participants have shown benefits based on the intended learning goals for which they were implemented (Gašević et al., 2015;Lowenthal, Dunlap, & Snelson, 2017;Yamagata-Lynch, 2014).Fewer studies, however, have been performed comparing the varying types, levels, and benefits of asynchronous-only and live, real-time discussions specifically in the context of online roleplaying and social simulations and how they can foster student achievement.
This study reports an experiment on multiple designs of GlobalEd, an online educational roleplaying simulation game for middle school social studies classrooms.GlobalEd simulates a social process of a complex international crisis in which students play the roles of different countries that come together to research and develop proposals to solve a given real-world problem scenario (Lawless et al., 2018;Riel & Lawless, 2022).Through gameplay, social interactions like discussion are a fundamental principle to the design of GlobalEd as a pedagogical approach for developing students' argumentation skills (Mercier, Boudry, Paglieri, & Trouche, 2016;Scardamalia & Beriter, 2006).
Specifically, because previous iterations of the GlobalEd game over its ten-year history had always included a synchronous discussion opportunity to online players, we were particularly interested if the game could be played in an asynchronous-only way and still generate an observable effect on the argumentation skills learning outcome.We wanted to investigate if increasing levels of simultaneity or synchronous play had a positively trending effect in comparison to asynchronous play.This would help test an assumption of whether including the most or highest-level live discussion is the best option in online and socially intensive learning interventions, such as social simulations or roleplaying games.
The following two research questions guided this study to respond to the need for additional research on comparing the differences in the effects on learning outcomes between synchronous and asynchronous discussions in online simulations and games that prioritize social interaction for learning: RQ1: Does an asynchronous-only version of the GlobalEd intervention demonstrate either comparable or higher effects in written argumentation skills (i.e., the primary learning objective of GlobalEd) than two other versions of GlobalEd that emphasize synchronous discussions among players?RQ2: Do increased levels of synchronous discussions in GlobalEd demonstrate progressively higher effects in written argumentation skills (i.e., the primary learning objective of GlobalEd).

Context for the Study-Description of the Intervention The GlobalEd Online Roleplaying Simulation
The intervention in this study is an online roleplaying simulation called GlobalEd.GlobalEd is designed for play across multiple social studies classrooms simultaneously to simulate complex international social interactions and systems in an authentic way (Lawless et al., 2018;Riel & Lawless, 2022).This allows for players to discover and apply real-world knowledge related to socio-scientific issues that do not often have a "correct answer" solution.Such ill-defined challenges mirror the authentic problems that scientists, technologists, diplomatic professionals, and policymakers face with solving authentic global issues.
In the game, students play the roles of scientific advisors to an assigned country.Each country that is roleplayed by students in the game is invited to an international summit (represented by synchronous or asynchronous discussions) to solve an assigned problem scenario.Up to 20 countries (i.e., different classrooms) play in a single GlobalEd game.

Interactive Discussions within GlobalEd
Play of GlobalEd progresses over three phases during a multi-week period: an initial research phase, an interactive discussion phase, and a summary debriefing phase.The primary goal of play is for each team to develop a single final proposal that has been co-sponsored by at least two other country teams (i.e., other classrooms).When the final proposals are submitted, they are voted upon by all teams, with the winner of the game being the one who has received the most votes.The essential feature of GlobalEd is the dialogue that is generated by students during both asynchronous messaging and live synchronous conferences across teams.In the first type of dialogue, players solve the assigned problem scenario via live, real-time conferences between classroom teams in collaboration on solutions to the problem scenario.The live conferences take place within a synchronous, instant-messaging-like online communications system where all players meet at a scheduled time.Before each live conference, students are provided with an agenda of the topics that will be discussed, which allows the students to prepare their ideas, solutions, and evidence to submit to the other teams for consideration.All student dialogue is moderated by a trained coordinator for both appropriate content, for prompting students to maintain their assigned roles in the game, and for coaching students in the use of argumentation skills.An example screenshot from a live conference is provided in Figure 1.

Figure 1 Screenshot from conference
In the second form of dialogue, students also interact with each other via asynchronous messaging (i.e., email-like messages) throughout the entire duration of the game.In asynchronous messages, players negotiate their positions and perform collaborative research over the full duration of the interactive phase.The asynchronous messaging is performed in an email-like interface with which students can log on at any time, including outside-of-classroom time or at home.An example asynchronous message and reply between two country teams from the actual game environment is illustrated in Figure 2.

Figure 2 Screenshot of asynchronous messaging between teams in actual GlobalEd play
Through the asynchronous messages, players continue the conversation and to debate issues with teams as they work toward developing well-argued proposals that will gain cosponsorships and alliances with other teams.Both the asynchronous and synchronous messaging discussions in the simulated international summit are facilitated in an online communications platform that moderates all communications between players, hosts scheduled events, and promotes interaction among players.Within both types of discussions, players regularly are encouraged to challenge each other to strengthen their arguments, to provide more evidence about their claims, or to provide additional context for the solutions that they are proposing.
GlobalEd has been in continual development and iteration over the last 10 years and has repeatedly demonstrated high levels of efficacy in development of student argumentation skills, content knowledge, and interest and self-efficacy in social studies and science topics and careers (Lawless et al., 2018(Lawless et al., , 2019;;Yukhymenko, 2011).However, live synchronous discussions have been the highlight for each iteration of the game for the past ten years.For this study, we attempted a game version that only used asynchronous communications for player discussion, with no live synchronous discussions.Additionally, we also wanted to identify if more live discussion opportunities had a stronger effect than the asynchronous-only alternative.

Participants
In the present study, 45 middle school social studies teachers in the United States participated, along with the students (n = 867) in each of their classrooms.Teachers each played a version of the GlobalEd game with their students based on the condition to which they were assigned.The simulation's program, content, and structure among conditions were identical except for the number of scheduled real-time, live conferences in which students would participate.Table 1 provides a breakdown of participants (students and teachers) by condition.
Teachers from different schools in both suburban and urban classrooms were randomly divided into one of three study conditions, which represent the level of live, real-time synchronous discussions (i.e., live conferences) that their assigned simulation would have: two scheduled live conferences (n = 17 teachers, 341 students), one scheduled live conference (n = 13 teachers, 263 students), and no scheduled live conferences or asynchronous-only discussions (n = 15 teachers, 260 students).Table 1 provides a breakdown on participant totals by condition.

Data and Instruments
Students were presented with identical pre-and post-intervention essay assignments to demonstrate their skill with written argumentation and to exercise their knowledge of the social studies concepts they encountered.In this assignment, students were presented with a prompt related to the simulation that they were tasked with writing about.The text used in the essay assignment for both the pre-and post-instruments is featured in Figure 5.

Figure 5 Pre-and post-essay assignment
The assigned problem scenario for all students in each of the three conditions was a global water scarcity dilemma to solve collaboratively with other teams, so it was expected that students would improve in the post assessment in both content knowledge of social studies as well as their written argumentation skills in response to the assessment prompt.We intentionally used instruments that captured students' writing as they made and defended a claim, as the instrument specifically prompted students to demonstrate their skill in complex thinking and argumentation.Thus, direct evidence of students' written argumentation skills and content knowledge were captured with a high degree of resolution for identifying the connections between the content knowledge and use of argumentation (Albanese, 2000;Savin-Baden, 2004).
The research team developed a rubric before implementation to analyze the pre-and postessay writing instruments.This rubric measured the level of argumentation skills on multiple parameters, including the presence and quality of students' use of claim, evidence, reasoning, and addressing the opposition, as well as to capture evidence of the use of social studies concepts that students encountered during the game.The rubric scored essays on seven items related to argumentation skills, with the post-coding values for each item being combined into a single summative scale value for each the pre-and post-essay.
Each essay was scored by three graduate-level students who were trained on the rubric and had 100% interrater agreement on a test set of essays after conference.After completing the test set, each coder graded each essay, pre and post.Because the instruments were identical, the pre and post versions of the essays were blinded to the coders as to reveal whether it was a pre or post during scoring.Although each of the three coders coded each essay, for data imputation purposes each essay was randomly assigned by computer to two of the coders.Each item was analyzed for alignment by computer between the coders.Any disagreements within 1 point between the two coders on the spreadsheet were resolved by adding the third coder's score and

ESSAY WRITING ASSIGNMENT Prompt
The world is in danger of running out of fresh water.Do you think this is true?Do you agree or disagree with this statement?Why?

Assignment
Write a persuasive essay stating your point of view on the prompt above.Give evidence to support your answer and provide your reasoning why this evidence supports your claim.Use your knowledge about water, science, world geography and cultures to help you write your response.You will have a total of 30 minutes to complete your essay.
taking the mean, averaging to the nearest half-point.No additional coding disagreements emerged after a third coder was introduced.Coding reliability between raters was > 0.80.Table 2 presents the scoring parameters in the rubric that were used for coding the identical pre-and post-essays.

Data Analysis
We conducted a hierarchical linear modeling (HLM) analysis (mixed) with the pre-and post-essay writing scores to compare the three conditions of the study and account for pre-test skills exhibited by students, as well as any classroom-or teacher-level effects that might be observed.HLM is a type of mixed-level multiple regression analysis that accounts for multiple "nested" levels of data and potential effects on the dependent variable that could occur at the different levels.HLM uses maximum-likelihood estimation to estimate the coefficients for each fixed effect that is entered into the model as the model predicts the output dependent variable.
HLM is increasingly used in educational research due to its robustness to detect classroom-or teacher-level effects among student achievement and other outcome variables (Raudenbush & Bryk, 2002).HLM is well-suited for education research as its models account for the moderating effects of teachers or even schools that are within different hierarchical levels (i.e., students within classrooms within schools).Furthermore, like ordinary multiple regression, HLM can account for other independent mediating or moderating factors within the analysis as fixed effects or random effects.
We employed the HLM 7 software suite (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) to conduct the analysis.Due to the naturally stratified nature of educational research data originating from multiple authentic classroom sites, student participants (at level 1-L1) were nested in the HLM model within teacher classrooms (at level 2-L2).In this multilevel analysis, a nested structure allows for the researchers to account for any possible teacher effects via inclusion of the pretest of students' writing performance at L2 centered around the grand mean to account for students' skill level at the outset of the intervention and their growth over time (Raudenbush & Bryk, 2002).A third nested level (L3) that represents the schools in which classrooms are nested was not necessary in this analysis, as there were no school-level effects to observe with multiple classrooms within single schools participating in the study.Different schools participated in the analysis.
The three experimental conditions were each coded as binary variables (0/1) that represented whether a student participated a given condition.In the model, the conditions of "2 live conferences" and "1 live conference" were entered as fixed effects in the conditional model.The binary coding scheme for each condition's variable assigned a value of 1 if a student was a part of the condition, or 0 if not.Thus, if a student was in the 1 live conference condition, the variable would be value = 1, otherwise it would be 0. The condition of "0 live conferencesasynchronous only" represented the baseline comparison for the model and was therefore not entered as a fixed effects term.The 0-conference condition is instead represented in the model's intercepts (i.e., when the "1 live conference" and "2 live conference" conditions are both value = 0).These comparison conditions were entered at L2 to represent each classroom's experimental condition to which they were randomly assigned.
Additionally, students' pre-scores on the essay instrument were entered as an L1 fixed effect that was centered around the group mean at L1 to account for students' prior knowledge and skills with the instrument and to identify the degree of pre and post student gains.Groupmean centering at this level is appropriate due to the potential classroom-level effects that might be observed within each classroom group.Furthermore, teacher-or classroom-level effects were also accounted for in the model, which was represented by students' pre-test scores centered around the grand mean at L2 to consider pre-scores between groups.
The results from the HLM analyses were then used to determine the effect size of each condition.The HLM equation for this study is provided in Equation 1.

Equation 1 Expanded 2-Level Equation for Hierarchical Linear Model Analysis
Post-achievement (Y) = g00 + g01*1Conf + g02*2Conf + g03*TC_achievement + g10 *SC_achievement + u0 + u1 + r In the model, Y represents the dependent variable for student achievement, as measured by student written argumentation scores on the post-essay instrument.The fixed effects terms for the experimental conditions are 2Conf (2 live conferences) and 1Conf (1 live conference), which were binary terms that indicated participation in the particular condition or not.The 0 live conference condition is represented in the model as the baseline measure through the intercept g00 when both 2Conf and 1Conf are value = 0. TC_achievement represents the level-2 teacher-centered grand-mean value for the pre-essay instrument to account for teacher-level classroom effects, SC_achievement represents the student-centered group-mean value for the pre-essay instrument, and u0 , u1 , and r collectively are random effects terms in the model.

Results
Table 3 displays the descriptive statistics on essay writing scores (as a summative scale score of the seven items on the essay rubric) for all conditions.Table 4 displays the results of the HLM analysis.The fixed effects of 1-conference and 2conference are in comparison to the 0-conference condition, which is represented as the baseline in the model.Comparatively, the 1-conference condition yielded higher positive results in comparison to the 0-conference condition, as indicated by a positive coefficient estimate.Because of its negative coefficient, the 2-conference condition fixed effect demonstrated that the 0-conference asynchronous condition outperformed the 2-live conference condition.It is important to take care with interpreting the 1-to-0 conference comparison (i.e., the 1conference term), as it was observed at p = .062and thus the observed differences may be due to chance.Although the comparison between 0 conference (asynchronous) and 1 conference closely approached significance at the p < .05threshold commonly accepted in social science research, there could also be no difference between the two, or instead interpreted as roughly equal groups.
Additionally, through the inclusion of the pre-writing assessment at both L1 (student) and L2 (teacher), the model also accounts for students' skills prior to starting the intervention.A significant L2 teacher-level pre-writing assessment term indicates that there were classroom-level effects observed and that students performed differently between collective classrooms.The HLM model accounts for these potential effects in calculating the overall estimates of the coefficients and their relationships to the dependent variable of written argumentation achievement.
Table 5 further interprets differences between the comparison conditions by providing pre-post effect sizes for each condition (reported as Cohen's d) to compare which condition had the highest pre-post effects across the study.For each condition, pre-post effect size was calculated as the difference between the means between the pre-and the post-tests divided by the pooled standard deviation of the condition.The comparison of pre-post effect sizes, otherwise known as a standardized difference of means, is appropriate in situations where identical instrumentation is used in educational pre-post assessment and effect sizes are thus interpretable in a standardized, comparable way (Morris, 2008).Each of the three conditions were confirmed to have been effective as intended, as each condition demonstrated significant positive mean differences favoring the post-test within confirmatory paired-samples t-tests (p < .001for all).This indicated that within each condition, the students performed better in the post-than the preassessment, Subsequently, this can be interpreted as having demonstrated learning and growth (or, alternatively, that the intervention achieved its learning objective goals).Note.Pre-post differences in means for each condition were confirmed by paired-samples t-tests, all of which were observed to be p < .001.Effect sizes reported as Cohen's d.
In Table 5, the 1-Conference condition was observed to yield superior pre-post student achievement effects in comparison to the other two conditions.The 2-Conferences and No-Conference also demonstrated effects in the HLM model and were confirmed by paired-samples t-tests, but to a lesser degree than the 1-Conference condition.These results indicate evidence for the efficacy of the intervention regardless of condition.In a conventional interpretation effect size, each condition can be seen as having a small to moderate effect (0.3-0.5) on student achievement.Indeed, the 1-conference condition yielded the highest effect, but the 2-conference and asynchronous-only 0-conference conditions both also yielded effects that trend toward moderate levels.
Because the difference between 0 and 1 live conference was not observed to be significant at the p < .05threshold generally accepted by the education field, these two effects are relatively the same.Although the difference was not significant in the HLM model, this study does suggest that some degree of combined live discussion and asynchronous-only discussion might provide a boost to student learning outcomes in comparison to asynchronous-only discussion, especially when the learning outcomes are highly social in nature (such as from learning argumentation skills).
Also of note is the significant negative difference between the 0-conference condition and the 2-conference condition in the HLM model, providing evidence that higher levels of live discussions may not always be the best option in virtual learning environments in comparison to providing asynchronous-only discussions.This observation is corroborated by observing a lower effect size between the 2-conference and 0-conference conditions, with 0-conference demonstrating a higher effect size.

Conclusion
Each condition in the study yielded a moderate effect size, providing evidence for flexibility in how designers develop socially intensive online spaces and for teachers in the degree to which they choose to engage with online social activity for their students synchronously.For this study, it was useful to identify evidence for designers that when course time is limited, an asynchronous-only condition can still be feasible and yielded a moderate effect in the achievement of learning outcomes.In many cases in the post-pandemic landscape, virtual asynchronous social interactions may be a teacher's best or only option.In this study, the asynchronous-only condition of the GlobalEd intervention was demonstrated to be effective.
More study and theorization on this concept are certainly needed to understand how and why the higher degree of live discussion was observed to have a lesser effect than the mid-range live-discussion condition and the asynchronous-only condition.In terms of social presence, live interactions are thought of to be a "richer" learning experience but may not always be necessary to indicate the presence of other individuals and groups (Chen, Park, & Hand, 2016;Garrison, 2016;Koehler et al., 2020).In today's digital ecosystem, a high degree of live discussions may serve to be distracting for some individuals or demand a high level of cognitive load, which may actually counter the benefits of the learning activity.Live interactions, particularly over time, might be mentally taxing to some learners but invigorating to others (Cui, Lockee, & Meng, 2012;Nieuwoudt, 2020).
Additionally, in virtual discussion, social presence also is dictated by the level of expectation of a person's behavior in the learning experience, as well as how the learning environment facilitates both asynchronous and synchronous discussion (Chen, Park, & Hand, 2016;Coffin, Hewings, & North, 2012).As such, the expectations of learners' social presence when interacting in a virtual space may be different than the expectations of the instructional designers and game developers who design activities and interactions for play (Cui, Lockee, & Meng, 2012;Larrain et al., 2019).
If real-time interaction and synchronous social presence are deemed the most desirable in online and hybrid learning environments, further study should be pursued in virtual learning contexts to investigate if and why students might perform better with only some but not the highest number of real-time interactions possible.
However, with the evidence from this study, it is heartening for instructional designers and teachers alike that any level of social interaction chosen still elicited the desired learning outcomes.Additional studies on the level of simultaneity of effective virtual interventions should be conducted to investigate whether asynchronous-only, mixed, or high-synchronous discussions all work effectively at achieving learning objectives, as to give educators increased choice in the implementation of virtual learning products with varying levels of required social presence.This is particularly important in the post-pandemic landscape where teachers may need to move rapidly from a synchronous learning context to an asynchronous-only context.Research on the efficacy of innovations tested with varying levels of simultaneity will help decision makers with selecting robust curricular materials.
This study is limited in scope related to asynchronous and synchronous learning conditions as it investigated just one single roleplaying game, one context in which discussions occurred by students, and one set of learning objectives.Additionally, the intervention is a simulation roleplaying game and not another type of online learning activity, preventing too broad of claims about simultaneity of discussion.Despite these classic limitations that are common in educational research, what has been demonstrated is that there was value to the asynchronous-only version of play as it yielded a beneficial effect.Additionally, the most live discussions were not found to be the condition to have the highest impact.Primary research like this study that richly describes the intervention design and evaluates the effectiveness of single intervention designs are necessary for teachers, policymakers, and instructional designers to make sound decisions on development and implementation of interventions.
In our reflection as instructional designers and researchers of the GlobalEd project after over ten years of implementation of the GlobalEd game in hundreds of classrooms, one of teachers' biggest hurdles was the scheduling of live discussions during constrained curricular time.Within the classroom, teachers have only limited time to get students to interact together, especially if working in small groups.Additionally, GlobalEd players are afforded the opportunity to interact across classrooms through extended play.Thus, the GlobalEd roleplaying game enables two layers of discussions, both of which are enabled through asynchronous interactions that can be performed outside of class through homework, small group work, or even remote learning at home.The results of a substantial effect size for the asynchronous-only condition confirmed for us the value in providing teachers flexibility in the play and implementation of GlobalEd.When designed in a principled way, asynchronous discussions can still promote social presence among participants, including those in the K-12 age range.However, this study also highlights the importance of evaluating whether designs work as intended and if learning objectives are met, otherwise designers risk the intervention yielding no effect and possibly a disappointing social experience for participants.
In the post-pandemic educational landscape where shifts to virtual learning can happen in an instant, online learning activities such as games and simulations that model social processes can continue to foster inquiry and development of key social studies skills without any interruption.Online games and simulations can be played in face-to-face classrooms, when possible, but also can allow for the virtual game platform to facilitate and organize high-impact play discussion regardless of whether the game is played in the classroom or online, or whether it is played synchronously or asynchronously.

Table 1
Number of Participants by Condition

Table 2
Essay Grading Scoring Parameters for the Identical Pre-and Post-Writing Assignments

Table 3
Descriptive Statistics

Table 4
HLM Analysis Results: Model Statistics

Table 5
Pre-Post Effect Size Results for Synchronous and Asynchronous Interaction Conditions