Using Structured Pair Activities in a Distributed Online Breakout Room

With the increasing availability of synchronous video-based breakout rooms within online courses, a growing need exists to understand how to best leverage this technology for enhanced online education. To help address this challenge, this paper reports on a case study that explored student activity within online video-based breakout rooms via a Structured Paired Activity (SPA) methodology. SPA, which is adapted from the concept of Paired Programming, defines a general way to structure roles and activities for the participants within the breakout room. Initial qualitative results suggest that the use of SPA in online breakout rooms increases student engagement and process effectiveness. These results are potentially applicable to a broad range of web-based synchronous online courses.

observed that students often have difficulties coordinating their interactions and achieving the benefits of peer collaborative learning when left to their own devices (Hesse, Garsoffky, & Hron, 1997; cited by Weinberger, 2011). Hence, to achieve the benefits of peer collaborative learning, instructors must create an effective classroom structure for teamwork. This challenge, of how to design synchronous video-based breakout room student interaction, is especially acute since, in this type of breakout room environment, the instructor may not able to actively monitor all the breakout rooms at the same time. In the face-to-face classroom, an instructor can more easily observe, at least at a high level, all the team interactions at once.
To address the challenge of how to effectively use such rooms, this paper explores one approach to structuring the activities in online breakout rooms. Specifically, it reports on a case study observing two semesters of an introductory data science course that used a structured methodology within its virtual breakout rooms. This approach, described as a Structured Paired Activity (SPA) methodology, is loosely based on Pair Programming (PP), in which two programmers work together at one keyboard. SPA can be considered a form of a collaborative script designed to provide learners with a specific socio-cognitive structure that maps their roles and interactions (Weinberger, 2011), and thereby overcomes some of the difficulties observed in unscripted peer collaboration.
This case study was done within a data science course. Data science integrates concepts across a range of fields, including computer science, information systems, software engineering, and statistics. It combines basic computer coding with iterative problem-based discussions to understand the goals of the effort, the knowledge needed to reach the goals, and the best approach to solving the problem at hand. For these reasons, a data science course is an appropriate domain to evaluate the use of this more structured process. While this case study was done within a data science course, an additional goal of this research was to understand the potential applicability of SPA in breakout rooms to other domains.
The case study compared team behavior when using SPA to behavior when students were left to their own devices on how to work in virtual breakout rooms. It also explored how graduate data science students perceived the utility of using SPA in breakout rooms. Specifically, this research focused on the following questions: RQ1: How does student team behavior change when using SPA, compared with unscripted collaboration?
RQ2: Do students perceive SPA as being a useful structuring mechanism within virtual breakout rooms? This paper begins by describing previous research related to breakout rooms. Then a description of SPA as well as an explanation of the methodology used in the case study is discussed. This is followed by a review of the findings from the case study. Finally, a concluding discussion includes possible next steps and limitations.

Review of Related Literature
This section first reviews the general topic of distributed synchronous group learning, which has been in existence for almost twenty years. Next, research with respect to the use of video-based breakout rooms is discussed. This is followed by a review of pair programming and more importantly, distributed pair programming.

Emerging and Scripted Role Assignments in Distributed Synchronous Collaborative Learning
There has been significant research on the use of distributed synchronous group learning, much of it occurring when basic synchronous computer mediated communication technology was first realized, approximately fifteen years ago. While there were many technical challenges, such as network bandwidth limitations, these research efforts typically focused on how an instructor should interact with a class during a synchronous online session. Of course, much of that computer mediated communication was hindered by the lack of audio and video capabilities (Wang, 2004). Despite these technology challenges, there was still a research focus on distance-based group collaboration. Generally, case studies (e.g., Chen, Ko, Kinshuk & Lin, 2005), found that online synchronous live instruction could be valuable to students. However, when examining synchronous collaboration in a chat environment, Pfister & Mühlpfordt (2002) noted that "lack of coordination and coherence among contributions is a typical problem" and found that establishing scripts within the chat environment helped provide some structure and improved student learning.
As the technology improved, it was noted that students, while skilled at watching videos, still lacked the knowledge of how to collaborate in a formal synchronous learning environment (Cole, 2009). Perhaps even more important, Warden, Stanworth, Ren & Warden (2013) culminated nine years of research evaluating synchronous learning environments and found that issues were typically not due to technology, but rather, from human behavior, and observed that "while students are familiar with virtual worlds and video meetings, they are inexperienced as virtual learners." Since scripts were shown to improve synchronous collaboration in the chat environment, scripts may also be a useful approach to help students overcome their inexperience as online collaborative learners using other synchronous technologies. Weinberger (2011) suggests that scripts can help learners engage in activities that are related to knowledge construction, reduce process losses in complex collaborative learning arrangements by taking over coordination tasks not inherently related to learning, and can make learners aware of the different responsibilities within the group and thereby facilitate beneficial motivational states and self-regulation.
One scripting approach that has shown promise is the use of scripted role assignments. Within this context, roles are defined as stated functions and/or responsibilities that guide students' behavior and group interaction, and scripted role assignments specify and externalize the roles expected from learners during collaboration (Strijbos and Weinberger, 2010). In a study of undergraduate students in an asynchronous environment, Olesova et al. (2016) found that scripted roles were an effective strategy to improve both learning processes and outcomes. They randomly assigned students in online discussions into one of three roles (starter, skeptic, wrapper) or no role at all. Role assignments were rotated. They found that students demonstrated a higher level of cognitive presence when assigned a role than they did with no role assignment. In an earlier study of an asynchronous learning environment, Aviv et al. (2003) found that knowledge construction and critical thinking reached their highest level when the learning network was more highly structured. Other researchers (e.g., Schellens et al., 2005;DeWever et al., 2010) have found that different roles have different impacts on knowledge construction, with the summarizer role in online discussions having the most positive effect. Research on scripted role assignment has also suggested the importance of rotating assigned roles (O'Donnell and Dansereau, 1992.) There has been much discussion concerning the strengths and weaknesses of using scripts to structure collaborative interaction in the computer supported collaborative learning (CSCL) community. Kollar et al. (2006) and Weinberger (2011) have pointed out that the preexisting, internal collaboration scripts may be in conflict with whatever scripted role assignments an instructor might design. These internal, or emerging, scripts are evident and may be observed when a collaborative activity is unscripted and allowed to proceed as the participants desire. Weinberger (2011) cites three potential risks inherent in overscripting collaborative interactions. First, overly constraining scripts can dampen student motivation (Rummel, Spada, & Hauser, 2009). Second, externally provided scripts may also interfere with existing, well-functioning internal collaboration scripts (Kollar, Fischer, & Slotta, 2007). Finally, externally provided scripts, may by their very nature, interfere with self-regulated, playful, and exploratory thinking (Dillenbourg, 2002.) Thus, it is important to observe and understand both scripted and emergent role assignments when exploring role effects in distributed synchronous online breakout rooms.

Use of Breakout Rooms in Online Learning
Breakout rooms are increasingly used within online learning environments. For example, Martin and Parker (2014) found that 25% of the surveyed online educators used breakout rooms. In general, the use of breakout rooms encourages "learner-learner interaction," which as noted by Moore (1993), is a valuable resource for learning. Chandler (2016) found that breakout rooms are useful for facilitating collaborative learning and interaction. Chandler noted that breakout rooms provide distance-learning students with the opportunity for peer-to-peer contact, which can be invaluable in building relationships and confidence. Some have argued that the effectiveness of active learning techniques such as breakout rooms lies as much in the enhancement of engagement as in the ability to generate in-depth exploration of the topic (Redish, Saul, & Steinberg, 1997). Some educators have even begun to research the incorporation of "escape-room" narrative and gamification to provide experiential structure to the use of breakout rooms. They claim two benefits of adopting the escape room strategy: a clear problem-based structure for students, combined with a higher level of engagement.
However, there has been minimal research exploring the pedagogical aspects of breakout groups in face-to-face or online classrooms. Lougheed et al. (2012) reported that research about the use of breakout groups in postsecondary education is sparse. They also reported that most of the published literature pertaining to the use of breakout groups describes the feedback generated during the breakout sessions rather than specific pedagogical elements of the breakout groups themselves. This dearth of published information highlights the need for research related to factors that affect their use in this context. There has also been little research into online interaction during synchronous breakout sessions (Brown, Schroeder, & Eaton, 2016). Two papers briefly discuss online breakout rooms, but without any explicit focus on the viability of breakout rooms or the process to be used in the breakout room. In one paper, Martin and Parker (2014) noted that using breakout rooms could enhance interaction and build a sense of community. However, there was no examination on use or the effectiveness of breakout rooms. Ellingson and Notbohm (2012) also discussed the use of breakout rooms, but focused on the technical details, such as how to setup a breakout room. They described breakout rooms as an "appealing feature," but did not discuss any guidelines on how to use the breakout rooms, nor did they report on any observations of use of breakout rooms.
There are also some indications that breakout rooms do not always magically create engagement and higher levels of learning. Blackstone and Oldmixon (2016) found that students in a breakout from a lecture class were not more satisfied and did not succeed at higher levels compared to their peers in a lecture-only class. Lougheed et al (2012) found that higher-GPA students had a significantly less favorable response to the use of breakout groups than did their lower-GPA colleagues. In these studies, researchers speculated that possible reasons for these findings were that students may not have been clear about what they were supposed to gain from the breakout group sessions, or that some (e.g., high-GPA students) may have found that the structure of the sessions did not meet their needs.
These concerns are consistent with Kuhn's (2015) critique, who argues that results of collaborative learning are often precarious, and it therefore should not be considered a "silver bullet." Kuhn argues that without careful design attention to the nature of the task or problem, and specification of the learning goals expected, the outcome of any collaborative learning intervention is likely to be unpredictable. Thus, what little research exists on the subject of synchronous distributed breakout rooms suggests that much more attention needs to be paid to the pedagogical structures and scripts used to prepare students to use them.

Distributed Pair Programming
Pair Programming (PP) is an agile software development technique that is part of Extreme Programming (XP). When using PP, two developers work together, side-by-side, at one keyboard. One person, "the driver," types at the keyboard. The other person, "the observer," reviews each line as it is typed, checking for errors and thinking about the overall design (McDowell et al., 2002). Distributed Pair Programming (DPP) is pair programming with the two programmers working at a distance via online tools (Hanks, 2005). Pair programming is thought to provide several benefits, including fewer errors in the code, enhanced ability to share best practices, faster team learning, and social support that improves morale.
Research in DPP within an educational context has typically reported on the use of DPP when the students have been able to build a relationship within a face-to-face context. Early research with respect to DPP, such as Stotts et al. (2003), used students within a face-to-face class to compare the results of DPP and PP. Even though the technology used was not as advanced as what is possible today, in those early experiments, DPP was shown to have a positive impact on outcomes, similar to PP. In more recent research, Tsompanoudi et al. (2016) implemented a system that supports the application of DPP within an interactive development environment (IDE), and found that the use of collaboration scripts, defined to implement DPP, yields improved results, such as improved student learning. Like many of the earlier studies, their experiment was for a face-to-face class that used DPP, not for a distributed team using DPP. In fact, in a review of DPP research, Estácio (2015) notes that while there have been 34 articles discussing DPP, these papers have primarily covered tools to support DPP, or reported on experiments where a face-to-face class uses DPP (e.g., Stotts, 2003;Tsompanoudi et al., 2016), and that "few studies explore DPP as a pedagogical tool and how DPP could be integrated with the trend of online courses." Overall researchers have not often explored DPP when the students were not taking a colocated, face-to-face class. This distinction is important, since colocation enables students to establish a connection in a face-to-face context and then use online tools to do DPP. This gap in the research has also been noted by Edwards et al. (2010), who called for more comprehensive and intensive investigation into the power of pair programming when used within purely online courses.

Motivation for the Study
Gaps in the literature reviewed above provide three dimensions of motivation for this study. First, scripted collaboration role assignments have primarily been studied in asynchronous environments. This prior research suggests that there are potential benefits and risks that should be also be explored in synchronous environments, especially in the comparison of scripted versus naturally emerging role behavior. Thus, this study observes role behavior in both emergent and scripted situations. Second, while technology advances have made the use of distributed online breakout rooms more common, research on the pedagogical structures supporting their use has been sparse. Finally, while the use of the Distributed Pair Programming concept provides a potential model for breakout room role assignment, there has been little research exploring its use in purely online courses. Thus, this study explores the impact of using a Structured Pair Activity (SPA) methodology for scripted role assignment on students' collaborative behavior in distributed online breakout rooms.

Methods
Pair programming concepts were used to develop the SPA scripts, which structured student collaboration in breakout rooms during an online data science course.
The impact of using SPA within breakout rooms was explored via a case study. Merriam (1988) indicated that a case study should have a bounded system that can be identified as the focus of the investigation. This study examines the process of using SPA within synchronous online breakout room sessions, where students have access to video conferencing, chat, and the sharing of files.

Case Study Context and Setting
SPA was evaluated within two one-semester sections of an online graduate-level introduction to data science course. In addition to the class's asynchronous activities, the course also met in a synchronous online session weekly at a specific day and time. For part of each synchronous session, students worked in two-person teams using breakout rooms. Over the two semesters, 26 graduate information system students participated in the study. Students were randomly assigned into teams of two people for work in the breakout rooms. Twelve students (six teams) were in the first semester's class and 14 students (seven teams) were in the second semester's class. The same breakout teams were used across the entire semester. The students had a wide variety of educational and career backgrounds. Twenty-five percent of the participants were female. The students were geographically distributed across multiple time zones, with students participating from North America, Europe, and the Middle East. Eighty percent of the students had a STEM-focused undergraduate degree. Finally, 92% of the students had full-time jobs. The instructor, a coauthor of this research, was the same for each of the two semesters and had previously taught the data science course many times.
Each week, over an eight-week period, there was a different breakout-room assignment. Five of the assignments were programming assignments. In them, students were required to use the R programming language, a popular data science tool that is used in both industry and academia. For these assignments, the student teams were expected to do R programming, using typical data science techniques such as machine learning algorithms and geographic information analysis. For two of the assignments, the work focused on a more qualitative task that required students to document the result of the team's discussion. The final assignment was the creation of a client presentation visualizing the results of the data analyses and documenting the outcomes that would likely be actionable by their client. The sequence of these breakout assignments is shown in Table 2. The synchronous sessions were 90 minutes long and typically the students were in breakout rooms for 40 to 50 minutes. The technology used for the synchronous sessions was similar to that described by Martin & Parker (2014) and included video conferencing, chat, screensharing, and the sharing of documents. Each virtual breakout room was equipped with similar tools.
Over the two-semester period, 104 breakout sessions were monitored. In each semester, the first four breakout sessions used a baseline condition where the instructor provided the assignment to be done in the breakout room but provided little guidance with respect to how the students should collaborate. These sessions provided an opportunity to observe naturallyoccurring, or emergent, role behavior. Three of these four breakout sessions focused on coding tasks. For the following four breakout sessions, two of which were primarily coding tasks, the students used SPA. Thus, these sessions provided an opportunity to observe the impact of scripted role assignments.

Structured Pair Activity
For the first four sessions, the process used within the breakout room was left to the students. There was no specific process defined for them to use. Based on industry best practices (McKinnie, 2018), as well as the lack of identified research addressing how to use breakout rooms, this baseline condition, with unscripted role assignments, appears to be a common practice for many instructors that use breakout rooms. SPA was then used to provide a structure of scripted role assignments for the following four weeks.
Before the first use of SPA, the SPA process was explained to students via discussions and a documented presentation. Specifically, SPA role assignments were described to the students in terms of the following key concepts: • Within each breakout room, there was one driver (the person that had control of the shared screen and was typing within a shared document). The second student was the active observer that, via the shared screen, saw what was being written by the driver (R programming code or other documents such as a PowerPoint presentation). These roles were explained to the students.
• Drivers were instructed with the following scripted role assignment: When you're the driver: Agree with your partner on one tiny goal at a time, something you can complete within a few minutes. State the problem in words. Talk to your partner! Ensure that you both know what you are working on right now. Complete the current tiny task (e.g., coding goal, presentation text, etc.) as quickly as you can. Ignore larger issues (but note them out loud).
Trust the observer to be your safety net.
• Observers were instructed with the following scripted role assignment:

When you're the observer:
Read what the driver is writing as he or she writes it; evaluate it for accuracy. Your job is to review and think how it fits into the larger picture. Pay total attention, aiming to let nothing get by you. Think about possible issues and ways to simplify. Bring up issues directly related to the tiny task Wait until the current tiny goal is done to bring up larger issues and ideas for design improvement. Don't dictate-driver should be actively thinking about how to achieve the current tiny task, not just typing. Exploit the fact that you don't need to focus on the details.
• All students were encouraged to be actively engaged with each other, to share their thoughts and ideas, and to ask questions.
• Students were instructed to frequently rotate roles between driver and observer, with a goal of rotating every fifteen minutes.

Evaluating the Impact of SPA
To evaluate the impact of SPA, the research adapted Hackman's team effectiveness model (1987). This model, shown in Figure 1, states that to evaluate the effectiveness of a team process, one should observe task process and output, the team's continued desire to work together and the satisfaction of individual team members. To evaluate the model shown in Figure 1, multiple data sources were used, which is consistent with Eisenhardt (1989). First, since an instructor was able to easily move between the breakout rooms unobtrusively, systematic instructor observations provided insight into how the teams were working together, the group dynamics within each team, and the ways the scripted role assignments affected team behavior (answering RQ1). Students were informed that the instructor would periodically observe their interactions. During each breakout session, the instructor systematically moved through each of the different breakout rooms, observed the student teams in each breakout room, and documented those observations. Each room was observed for 3-5 minutes at a time and each room was visited 2-3 times per class session.

Satisfaction of individual team members
Willingness of team to work together on future tasks

Process Effectiveness
The observations of student behavior patterns of were focused through a set of specific questions: who is leading the conversation? Are students equally participating in the dialog? Does the team appear to be productive and effectively working toward completion of the task? Are there indications of expert-novice conditions (or experience gaps)? In sessions where SPA was used, the instructor added an additional question: are students rotating roles? These systematic observations provided a qualitative view of task process and output for each team.
Student satisfaction with SPA (RQ2) was explored through a three-item student satisfaction scale, which had a reliability, or internal consistency, of 0.94 (based on Cronbach's alpha). The scale consisted of the following three items: I want to use SPA for future small group assignments.
SPA was useful for our work.
I am satisfied when using SPA.

The survey also included an open-ended question:
What were the strengths and weaknesses of the SPA breakout room process?
The survey was given to students at the end of each semester as part of a voluntary course evaluation process. Response rate was 69%. Consistent with IRB review guidelines, student survey participation was voluntary and students were informed that survey results could be used, in an anonymous fashion, as part of an ongoing pedagogical research project.
Finally, an indicator of students' willingness to work together on future tasks (RQ2) was obtained when students had the opportunity to reform teams for a subsequent project. Table 1 maps the data sources to the key measures defined in our model to evaluate the effectiveness of SPA. Table 1 Measuring Team Effectiveness

Key Measures How Measured
Task process and output Instructor Observations

Satisfaction of individual team members Student Survey
Willingness of team to work together on future tasks Selection of (new) project team members

Task Output and Team Process
An assessment of the effects of SPA on team process was made based on a systematic, week-by-week observation of the student teams, as described above. The weekly observations before the introduction of SPA are summarized in Table 2. Initial analysis determined that there were similar results in both semesters in the sense that there were codeveloper teams and oneperson-dominant teams at the outset, and the one-person dominant teams modified their behavior similarly after the introduction of SPA. As a result, results from the two semesters have been combined into a single presentation. Discussion of a realworld situation and how it could use data science • Pairs were mostly polite with each other.
• All teams began discussions of "How should we proceed?" • 7 teams seemed to have a more talkative person, who appeared to lead or dominate the discussion.

R Coding
• The teams exhibited three distinct patterns of role behavior: • 7 Teams: One-Person-Dominant. One person, seemingly the most experienced, was the dominant person and did all the coding (this week was a coding assignment). In most of these teams, the other person was quiet and relatively uninvolved. • 4 Teams: Codevelopers. Both team-members contributed equally, cutting and pasting code to each other via the chat function. Neither dominated the interaction. They appeared to have clear emergent role expectations, or internal collaboration scripts, that were compatible. • 2 Teams: Looking-For-Guidance. These two teams were continuously asking the instructor what to do next. Neither person was dominant, but they did not appear to have a functional emerging collaboration script.

R Coding
• The two teams that had previously asked for help migrated to the One-Person Dominant strategy. This left: • 9 One-Person-Dominant teams: In most of these teams, the non-dominant person continued to be relatively uninvolved, trying to understand what the more experienced partner was doing. • 4 Codeveloper Teams: These teams continued to work effectively. Their internal collaboration scripts were active and functional.

R Coding
• The same pattern continued as in week 3. Roles had become normalized into the two basic emergent role scripts: One-Person-Dominant (9) and Codeveloper (4). • The less-experienced person in One-Person-Dominant teams remained relatively uninvolved, and there was a growing gap in their level of knowledge, since the "doers" were learning more while doing. Thus, this emergent script was not producing the desired learning outcomes for these individuals.

Observations on the Unscripted Sessions.
Across both semesters, in the first week, it was observed that the student's use of the breakout room was often a bit awkward. For example, students did not know each other well and did not want to "step on the other person's toes." Since this was the first week of the course, this could be explained due to the fact that the students did not know each other well, and hence, had to develop a social connection, especially since they were only connected via computer mediated communication. Unfortunately, during the following three unscripted sessions, only four teams were perceived to work effectively (the co-developer teams). The dynamics between the students during these unscripted weeks appears to have been driven by a number of factors, such as how outgoing the people were and how much knowledge each person had with respect to the assignment. Hence, often times, the more outgoing and/or knowledgeable person dominated the two-person discussion.
The weekly observations after the introduction of SPA are summarized in Table 3. From these weekly observations, four key themes emerged that suggests that task process improved, which are discussed in the rest of this section. Discussion of a realworld situation and how it could use data science • There was some initial confusion on the roles and how to "rotate" who was "driving". • The instructor clarified questions and encouraged teams to swap who was driving and who was observing. • Switching roles was technologically challenging due to the limitations of the platform.

6
R Coding • Most teams started to get the hang of SPA. They figured out workarounds to more easily switch who was driving (e.g., using Google Drive or emailing files). • Teams started to become more productive and got into a rhythm of doing work. • In seven of the nine original One-Person-Dominant teams (often due to an experience imbalance,) the less experienced person clearly was more engaged and doing more. The amount of discussion was greater this week compared to last week. • Two of the original One-Person-Dominant teams were still unable to swap driver/observer roles, and in these teams, the observer remained fairly uninvolved. One of these teams made no effort to switch roles. • Some observers expanded their role to do outside research (e.g. they looked for solutions to problems in websites like Google or Stack Overflow). The two original Codeveloper teams in the first semester were the leaders in this role expansion. They modified their previous co-equal collaboration scripts to include observer research while the driver was doing the writing/coding (this became an "active researcher/observer" role).

R Coding
• At this point, there were only two One-Person-Dominant teams remaining. Eleven were classified as Codeveloper teams. • The active researcher / observer role spread further in week 7, with more observers becoming proactive in searching for answers using external resources. • As teams become more comfortable, the teams seemed to be better at decomposing work into smaller tasks (short bursts of work). This was perhaps due to the need to switch roles and their improving experience in being able to switch roles. • Students still did not switch roles as frequently as the SPA instructions called for (every 15 minutes). Actual switching time was approximately 25 minutes. • One team was still unable to switch roles. 8 Creation of a presentation with visualization of findings • Similar to week 7, teams were fairly predictable in how they were interacting. • Student-to-student engagement and dialogue continued to increase.
• By the end of this week, only one team was still struggling to swap driver/observer. The observer on that team remained uninvolved.

Observations on the Sessions with SPA Scripted Role Assignments.
Improved team coordination and focus when using SPA. The dynamics within the breakout rooms changed when students were introduced to SPA. The nondominant person in the One-Person-Dominant teams started to be more productive. For example, it was noted that these teams, when using SPA, "would quickly determine who was the driver, and what was their shortterm goal." SPA seemed to provide two key advantages. First, it provided a framework where being a leader was divided between two roles: doing the writing (the driver) and doing the brainstorming (the observer). This was helpful for the originally One-Person-Dominant teams (the definition and switching of the roles helped to balance the dominance). Thus, SPA provided a framework for the observer to be more active and for both students to have well-defined roles. In general, it was observed that there was more two-way dialog (due to the active nature of the observer) under SPA as compared to the baseline condition.
Expanded observer responsibilities. During the first semester, the instructor observed that in both of the initial codeveloper teams, the person in the SPA observer role often started to work on tasks beyond what was suggested for the observer. Specifically, the students who were observers would sometimes start to actively look for solutions (via websites such as Stack Overflow or a specific data science website). These students then shared their insight with their driver so that the driver could leverage that insight. The questions addressed by the observer when doing the searching ranged from specific coding details (such as the parameters of a specific R function) to much more conceptual open-ended questions (such as how one might handle missing data). Since it was believed that this type of active research improved the team effectiveness, the description of the observer was expanded to include this type of activity in the second semester. This addition did not change the basic pattern of unscripted versus scripted role behavior in the second semester. In the first semester, there were two original codeveloper teams and four original one-person-dominant teams by week three. In the second semester there were two original codeveloper teams and five original one-person-dominant teams by week three. In both semesters, all but one of the One-Person-Dominant teams evolved to become Codeveloper teams.
Role-switching difficulty. Even though the frequency of role rotation increased under SPA, the teams did not rotate between driver and observer at the frequency suggested. Specifically, the SPA instructions suggested that students rotate every fifteen minutes. However, most teams rotated at a rate of approximately once every twenty-five minutes (i.e., one rotation within the breakout session). This decrease in role-switching was at least partly due to the technology being used, in that switching roles was not seamless. For example, files needed to be explicitly "uploaded and then downloaded" from one student to the other student.
Increased student engagement. An unexpected observation was that, later in the course, there was a perceived increase in student engagement (i.e., questions to the instructor, dialog between students) compared to the first half of the course and to other course sections that were offered in previous semesters (course sections that used unscripted breakout rooms but did not use SPA). This might have been due to the observed bonding that occurred within the SPA-breakout sessions, where the social sharing of information was much greater than what occurred when using a more traditional breakout room process. In other words, using a more well-defined breakout room process might have improved team bonding due to the structure of alternating who was is "in charge" (i.e., the person typing at the keyboard).

Team Member Satisfaction
To explore student satisfaction, the three-item student satisfaction scale, described above was used. The voluntary survey was administered at the end of each term and the response rate was 69%. The average student response for this scale was 4.4, suggesting that students were relatively satisfied with SPA.
The open-ended qualitative feedback, within the same survey, was analyzed to more deeply explore the drivers of student satisfaction when using SPA. Three key themes emerged that seemed to drive their satisfaction. These themes are described below: Improved learning. Students thought that their learning improved when using SPA. This improved learning was driven by better insight shared between the partners. For example, one student stated "I got to learn more by working with my partner in this way." Improved coordination & collaboration. Since a key goal of SPA is to improve coordination between the two students, it was not surprising that several students noted that they thought that SPA improved coordination, which often led to a feeling of improved collaboration. For example, "it helped me coordinate with my partner" and "it allowed us to collaborate much easier" were statements that exemplified how the students perceived their improved collaboration when using SPA. However, one student did note a disadvantage to using SPA, in that "some people are hard to keep on track, or are very rigid in needing control." Note that this last feeling could have been instilled during the first four sessions. In any event, this personality trait might suggest that additional initial discussion with respect to working in a team is required prior to the use of breakout rooms.
Improved productivity. Students also focused on their perceived improved productivity. For example, one student noted that "we were most productive during class time when we used SPA versus on our own when we did not." This productivity was also aided by the fact that students thought it was easy to work with their partner, perhaps due to the structured dialog with using SPA.
For example, one student simply noted that SPA "Made it easy to work with someone else remote" and another stated "It was an easy way to work with my partner."

Willingness to Work Together on Future Projects
In terms of the students' willingness to work together on future projects, after the four SPA breakout sessions, the students had to form a project team to work on an end-of-the semester project. Students were given the opportunity to stay in their current "breakout team" or select different team members (with or without the help of the instructor). Ninety-two percent of the students wanted to continue working with their breakout team member, and the others did not strongly object to staying with their current breakout team. While this could have been driven by students being comfortable with the status quo and not wanting to risk working with a "bad" partner, it nevertheless does show that the students were at least not frustrated with the current partner. Hence, there was a clear favorable response with respect to the students' desire to continue to work together on future projects.

Discussion
This paper defined a process, Structured Paired Activity (SPA), for use within breakout rooms of an online course. A case study was performed to explore the effects of using SPA within a breakout room. Systematic observations suggested that SPA was a useful way to provide structure within breakout rooms and positively modified student behavior (thus addressing the first research question). In addition, students also thought that SPA was a useful way to provide structure (addressing the second research question). Furthermore, based on the fact that (1) task output was thought to improve, (2) team members were very satisfied while using SPA, and (3) the students wanted to continue working with their teammate, our model of process effectiveness suggests that SPA was an effective intervention. One additional finding was that the use of breakout rooms seems to have enabled learning via a social and constructive process. This connectedness was evident via increased student-to-student interaction during class as well as increased student-to-student communication outside of class.
Prior to the introduction of SPA, there was clearly a mismatch between the internal collaboration scripts possessed by nine of the thirteen teams and the requirements of the virtual breakout rooms. Since there was no instruction on how to use the breakout rooms, it is not surprising that these teams experienced a momentary lack of support (underscripting) as described by Dillenbourg (2002). Because of the scarcity of research on breakout rooms (virtual or face-toface) described by Lougheed et al. (2012), it is difficult to know how often students are provided with little or no structure to guide collaboration in real breakout-room environments. But personal experience suggests that such underscripting may not be uncommon. The introduction of SPA alleviated this underscripting in all but two of the teams. The results support the idea that at least some of the problems previously observed in breakout rooms (e.g., lack of success and satisfaction (Blackstone and Oldmixon, 2016;) dissatisfaction in higher-GPA students (Lougheed et al., 2012)) can be attributed to lack of structure and underscripting.
It is also interesting to consider the four codeveloper teams that began the course with seemingly effective internal collaboration scripts. Some research has suggested that external scripts may interfere with previously effective internal collaboration scripts (Weinberger, 2011;Kollar, Fischer, & Slotta, 2007). In this case, however, these student teams not only adopted the SPA script, but also positively modified it by expanding and enriching the SPA observer role. The concerns expressed by Dillenbourg (2002), that externally provided scripts, may by their very nature interfere with self-regulated, playful, and exploratory thinking, were not evident in this exploration of SPA. Thus, the SPA script seems robust and flexible enough to avoid both overscripting and underscripting.
Ideally, the provision of an external collaboration script is intended to achieve several different outcomes. First, the goal is to regulate learning activities and provide complementary process knowledge that leads to more effective team performance (Weinberger, 2011). SPA appears to have achieved this goal. Collaboration scripts also hope to increase both individual and shared domain knowledge. While this appears to be the case in this study, the qualitative design provides no direct evidence. Future studies of SPA should develop explicit measures of individual and team learning to study the learning effects of SPA. Designs such as those used by Kuhn (2015) would be beneficial. This is especially germane since this study revealed expert-novice experience gaps in a number of the teams. Such gaps may be common in many types of courses. Finally, instructor-provided collaboration scripts are intended to help students learn how to collaborate more effectively in the future; that is, the ultimate goal is that students will gradually transition from external to internal collaboration scripts. This study indicated that students found SPA to be useful. Future research should investigate how much of the SPA collaboration script is internalized.
While there were over one hundred breakout sessions observed, there were only two classes in this case study, and each had a low number of students in the course. Hence, one limitation is the small sample size, in terms of the number of students and number of courses in the study. Another limitation of this study's design was, as mentioned above, the lack of a direct measure of learning. In addition, SPA was compared to a straightforward no-script alternative. While this alternative may reflect reality in a number of classrooms, a possible next step could be to explore the value of using the SPA process versus breakout rooms with different structuring methodologies.
While data science was an interesting class to evaluate SPA (since the assignments ranged from open-ended discussions to more structured programming tasks), it would be interesting if other types of courses evaluated SPA. For example, more discussion-focused courses could be explored to better understand the strengths and weaknesses of SPA in other contexts. Kuhn's (2015) probing discussion of the types of skills best suited to collaborative learning (e.g., argumentation skills, inquiry skills) can be useful in guiding this future expanded research. In our study, inquiry skills were clearly needed, as students labored to develop new ways of approaching problems in data analytics. Research in other domains would help us to understand if some tasks are better suited for using this methodology (or, in general, if some tasks are better suited for breakout rooms).
In summary, this case study suggests that when students use the scripted SPA role assignments in a video-enabled web-based breakout room, student process, productivity, motivation and connectedness to other students improve. While additional research on how to best structure student interaction in breakout rooms is required, this research indicates that the practice of just sending students into a breakout room without much structure is not ideal.