A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities

Despite the pressure from potential employers and higher education administrators to develop students’ global and intercultural competence, traditional study abroad programs simply are not feasible for many postsecondary students (Berdan & Johannes, 2014; Fischer, 2015). The purpose of this study was to evaluate the effectiveness of an online delivery model for study abroad activities. Building upon the findings of an initial exploratory program using Adobe Connect web conferencing tools, this evaluative case study was the second in a series of design based research studies intended to identify effective practices and develop recommendations to further refine the model through an iterative evaluation process. Using the Online Learning Consortium’s Quality Framework, each of the Five Pillars that support successful online learning (access, student satisfaction, faculty satisfaction, learning effectiveness, & scale) was evaluated through a combination of anonymous surveys, pre/post assessments, observations, and student & instructor interviews (Moore, 2005). Regarding access, 26 students who were enrolled in an intercultural communication course were able to participate in a study abroad experience in Italy; 10 students participated in the traditional study abroad trip in Italy while the other 16 participated virtually. The online students were able to join the live meetings, thus expanding their access to international experiences that normally would be closed to them. In terms of student and faculty satisfaction, both groups of students and the instructor reported specific areas of satisfaction, offered critical feedback, and felt that the concept was a viable one. While the students who traveled to Italy had a far more immersive experience, both groups demonstrated gains in learning. Using Morais and Ogden’s (2010) global citizenship pre/post assessment, both groups showed improvement on the selfawareness and intercultural communication scales, and when comparing the two groups the online students improved more on the social responsibility scale while the students who traveled improved more on the global knowledge scale. Both groups submitted assignments of similar quality, engaged in communications between the abroad and online groups, and interacted with the instructor and experts in the field. In terms of differences in student engagement, students had differing opinions on the interaction with the technology and the online group asked more questions during live meetings. The implications of this pilot study should inform the planning of the next case evaluation and are important for other educators who wish to implement a similar approach to internationalizing the curriculum through online instruction.


Introduction
In response to increasing demands by employers for globally prepared graduates, many institutions of higher education set strategic goals aimed at enhancing internationalization.Both employers and educators promote the importance of global and intercultural competence to success in our modern-day workforce, which has resulted in a push to increase enrollment in study abroad programs (Berdan & Johannes, 2014;Fischer, 2015).There are many obstacles to study abroad such as cost, lack of suitable opportunities, rigid degree pathways, and competing obligations from internships, work, or family.According to the Institute of International Education (2017) the national average of U.S. undergraduate students who traveled abroad during their degree program in 2016 was 10%.The student population that does study abroad typically does not reflect the diversity of most U.S. higher education institutions overall (Institute of International Education, 2017).
In tandem with efforts to internationalize the curriculum, online education has steadily increased as a strategic priority for higher education in the United States.According to the Babson Survey Research Group's Grade Level: Tracking Online Education in the United States, over 70% of Chief Academic Officers that responded agreed that online education is critical for their institution's long-term strategy (Allen & Seaman, 2016).Within a culture of growing technology acceptance, it would be logical to explore technology-based online programs that would directly benefit those students that simply cannot study abroad by offering access to high-impact international experiences without the barriers discussed above.
The purpose of this study was to evaluate the effectiveness of using web conferencing tools to allow students to actively participate in live study abroad activities from home or on campus.Building upon the findings of Howard and Gunter's (2017) initial exploratory program using Adobe Connect, this evaluative case study is intended to further refine a technology-based model that expands student access to high-impact international experiences by connecting students in the U.S. with their peers, instructors, and experts abroad.

Literature Review
Traditional study abroad programs, including short-term programs, have an established history of demonstrated benefits for students (Chieffo & Griffiths, 2004;McKeown, 2009;Spencer & Tuma, 2002;Tarrant, Rubin, & Stoner, 2014;Vandeveer & Menefee, 2006).For example, Chieffo and Griffiths (2004) found that short-term study abroad programs had a positive impact on students' intercultural awareness and functional knowledge.Tarrant, Rubin, & Stoner (2014) also found through experimental research that study abroad paired with academic focus yielded increases in global citizenship in a 4-week course.McKeown (2009) even coined the phrase "first time effect" to describe the profound transformational impact even one study abroad experience (regardless of length) may have on a student's intellectual development.
Internationalization efforts at home have also shown that students may develop intercultural competencies without the burden of traveling to a foreign country (Baldassar & McKenzie, 2016;Leask, 2004;Lilley, Barker, & Harris, 2015;Nilsson, 2003;Soria & Troisi, 2013).Jones (2013) argues that all students, not just those with international mobility, should enjoy an internationalized curriculum in order to develop transferable skills that will allow them to compete in the current job market.Lilley, Barker, and Harris (2015) suggest that "...learning to In addition to trends in increased internationalization efforts in higher education, online learning is also on the rise.Allen and Seaman (2016) reported a multi-year trend in increased online enrollments in U.S. higher education while overall enrollment decreased.They also reported that more than one in four students (28%) take at least one online course and that over 60% of chief academic officers agreed that online education is critical for their institution's long-term strategy (Allen & Seaman, 2016).
In this culture of technology acceptance and upward trend in online learning, educators have turned to technology to expand internationalization of curriculum in innovative ways.Roberts and Monroe-Baillargeon (2012) describe how technology such as web-based videoconferencing, learning management systems, and social media sites allow for new multicultural online learning environments.Specifically, Roberts and Monroe-Baillargeon (2012) state, "The benefits of multiculturalism and cross-cultural exchange, once only available to individuals with the resources to travel abroad, are increasingly available through information technology and creative pedagogy.Those professors who effectively integrate technology in their teaching will now have the ability to reach across borders to create virtual multicultural learning communities" (p.41).Scovotti and Spiller (2011) utilized synchronous and asynchronous technology to provide MBA students in the U.S. and Germany the opportunity to collaborate on a real-world business challenge at a distance, and they found that introducing video conferencing enhanced productivity and student satisfaction.Kenny and Lenz (2009) reported on a satellite-based project that allowed students in the classroom on the main campus to interact firsthand with their peers in the desert through a live broadcast from the field.One of the more interesting findings was that students reported increased attention levels when they were permitted to ask questions during the live broadcast (Kenny and Lenz, 2009).In a follow-up study that employed the same technology to broadcast from India, Kenny and Gunter (2015) found that the additional use of the text chat feature during the live broadcast appeared to increase student engagement and their sense of social presence.Howard and Gunter (2017) then initiated a series of formative case evaluations to develop a more cost-effective delivery mechanism for providing similar live broadcasts from the field with interactive chat.Advances in mobile technology now allow educators to take instruction outside of the classroom to facilitate learning in creative delivery methods with sound instructional strategies (Sung, Chang, & Liu, 2016).As a result, Howard and Gunter (2017) found that a valuable international learning experience could be provided to students online in the U.S. by connecting them with their peers on a traditional study abroad trip through web conferencing tools like Adobe Connect.
The purpose of this study is to continue the cyclical, design-based research started by Howard and Gunter (2017) to refine and further develop the web conferencing based delivery of internationally-focused instruction through a series of case evaluations.Three guiding questions drove this formative case evaluation: (1) Did the program accomplish its goals?(2) What was the impact of how it was implemented?(3) What changes to the design are necessary to firmly establish it as a valuable alternative to traditional study abroad?These guiding questions generated a set of evaluation questions that were then organized using the Five Pillars of the Online Learning Consortium (OLC) Quality Framework (Moore, 2005)  1.What is the potential impact of this program for providing students access to international learning opportunities?Student Satisfaction: 2. What is the difference in student satisfaction with the overall experience between the online and abroad groups?3. What were online participants' reactions to the web conferencing system? 4. What was the impact of technology on the students' ability to participate in remote instructional activities?
Faculty Satisfaction: 5. What were the instructor's perceptions about his teaching experience with this technology-mediated approach?
Learning Effectiveness: 6.What is the difference in student engagement with activities and experts in the field between students who travel abroad (abroad group) and those who participate via Web conferencing (online group)?7. How were the learning experiences of the online and abroad groups similar and how were they different?
Scale: 8. How does the cost of this case compare to previous technology-mediated attempts?9. How can this approach be improved?

Methods
The Online Learning Consortium's Quality Framework was used and each of the Five Pillars that support successful online learning was evaluated through a combination of quantitative and qualitative methods including anonymous surveys, observations, student and instructor interviews and Morais and Ogden's (2010) global citizenship pre/post assessment (Moore, 2005).This formative case evaluation is the second in a series of iterative studies where the recommendations of the previous study are implemented and then evaluated.Based on Howard and Gunter's (2017) study with a global health management course in Brazil, the methodology was replicated in this study with the addition of a pre/post global citizenship assessment to address potential learning outcomes in a more formal class structure.
The instructional model that was evaluated mirrored Howard and Gunter's (2017), which employed the Adobe Connect web conferencing tool to provide online participants in the U.S. access to the live study abroad activities in Italy, including tours of historical sites, cultural venues, and communities led by local experts.The online facilitator used an iPad with Adobe Connect to link the abroad group with the online group which enabled discussions between the groups using the chat, audio, and video functions.
Once logged into the web conference, the online participants were able to view the video feed from the perspective of the online facilitator as though they were part of the group of students in the field.They could hear the live discussion, but their individual microphones remained muted until they were ready to speak to the group or type in the chat.All live sessions were recorded and videos were captured and uploaded to the online course hosted in the university's learning management system to allow for convenient access between live sessions.
Ultimately, the instructor was responsible for delivering instruction for both the abroad and online groups.The online facilitator was responsible for all technical aspects of running the web conferencing tool and keeping the online students engaged in the tours and discussions during the live meetings.In some cases, the instructor was also the guide and led the discussions.At other times, the tours were led by local experts such as the live meeting at the Forum in Rome.For tours like this, the instructor started the meeting by situating the students' thinking and reviewing the learning objectives before turning the lesson over to the local expert.Throughout the tour of Rome, the online group was given the opportunity to interact by typing their questions for the guide through the Adobe Connect group chat.The online facilitator then asked these questions on their behalf as the students listened to the tour guide's responses and gathered information.

Study Population/Sampling
The population for this study was 26 American students from a large U.S. university.The students were enrolled in two combined sections of the same intercultural communication course with a study abroad component to Italy.Ten students participated in the traditional study abroad trip while the other 16 participated online.Both sections received the same course content and were required to complete the same assignments.For the first three weeks of the 6-week intensive summer term, both sections met together in the same classroom and completed group activities with 2-3 students from each section per group.During weeks four and five, one section traveled to Italy (abroad group) while the other section joined the live meetings via technology (online group).During this time, one-hour live meetings were scheduled for each day.The final week of the course was online for both sections and was dedicated to completing the group project and final assignments.

Data Collection
To explore the evaluation questions and effectively evaluate a study abroad program, Rubin and Matthews (2013) recommended gathering data from multiple data sources.Replicating Howard and Gunter's (2017) methodology, data sources included an anonymous student survey, instructor interview, session recordings, financial budgets, and online facilitator observations.For this study, additional researchers served as observers throughout the live meetings and reviewed the recordings when necessary.They also completed observation forms in order to record their experiences and capture what did and did not function well.This helped the researchers to properly identify technical challenges and perceptions of student engagement throughout the live meetings.Two additional sources were added: student interviews and Morais and Ogden's (2010) pre/post global citizenship assessment.Collectively, these sources were used to triangulate and evaluate the effectiveness of this web-based intervention and extract effective practices and recommendations for improvement.
Instrumentation.The anonymous online survey consisted of 45 Likert-scale questions with responses ranging from 1 (Strongly Disagree) to 5 (Strongly Agree) and was delivered to both groups of students at the end of the course.The online students were asked additional questions about the web conferencing technology and their online experience, and both groups were asked the same set of questions about their perceived learning and satisfaction.
A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 120 The questions were adapted from multiple sources including the National Survey of Student Engagement (NSSE) developed by the Indiana University School of Education ( 2014) and Picciano's (2002) survey instrument, which measured student satisfaction in an online environment.Also, researchers used the IBM Computer Usability Satisfaction PSSUQ Questionnaire for feedback on the technology (Lewis, 1993).In addition, Part I of Richardson and Swan's (2003) and Picciano's (2002) surveys were used to measure social presence in an online learning setting.The survey also contained 5 open-ended questions developed by the researchers to gather additional clarification.
In addition to the anonymous online survey, the instructor was interviewed after the trip to record his perceptions, students were interviewed, financial documents were compiled to evaluate cost, researchers recorded observations after each live event to note student participation and acknowledge areas for improvement, and researchers reviewed the recorded sessions in order to triangulate the data in an effort to provide a comprehensive formative evaluation (Maxwell, 1996).Finally, the Global Citizenship instrument developed by Morais and Ogden (2010) was also used as a pre-assessment with 52 questions (including demographics) and a post assessment with 45 questions to compare the two groups on ten factors organized under three dimensions: social responsibility, global competence, and global civic engagement.

Data analysis.
In addition to reviewing qualitative data from open-ended questions on the feedback survey, a t-test was run on the same two subscales established by Howard and Gunter (2017).The learning effectiveness (LEARNEFF) subscale (a = .92)was based on twelve items, and the satisfaction with experience (SATEXP) subscale (a = .86)was based on six survey items.The Global Citizenship instrument (Morais & Ogden, 2010) was also used to compare the changes in perceptions of the two groups.After running a t-test for significance, Cohen's effect size was also calculated for practical significance.
For all qualitative data sources (instructor interview, student interviews, and observation forms) one of four researchers transcribed the interviews and independently open-coded to reveal emerging themes, noting anything that related specifically to the evaluation questions.Then a second researcher independently verified the transcriptions and performed a second independent round noting themes specific to each evaluation question.Finally, the team reviewed and agreed upon the themes.

Results Access
Evaluation Question 1.What is the potential impact of this program for providing students access to international learning opportunities?The access to international learning opportunities for students seems to have been impacted in two ways: 1) use of technology, and 2) inspiration to travel abroad.Live meetings via technology provided access to an international experience for 16 online students who otherwise would not have been able to participate.It is important to note that the technology did not limit online enrollment to only 16 students --that was the number of students who chose to register for the online section of this class.Additional online enrollment was possible in terms of the technology.
The average online attendance for the seven live meetings was approximately 75% with 11.5 students participating out of the 16 enrolled students.An online student shared the A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 121 significance of this experience, "All in all this class was a very demanding experience, but a class that has left an impact on me, and to me, that's all you can really ask for in a course." According to the feedback provided, the use of technology to allow access to international learning opportunities was successful.All students reporting via the feedback survey were satisfied at some level with the Adobe Connect software used for the live meetings.One hundred percent agreed that even though they were not physically together in a traditional classroom, they still felt like they were part of a group in the live meeting, and two-thirds of students agreed that the live meeting provided a personal experience similar to the classroom.Finally, the online students reported that their level of learning that took place in this meeting was of the highest quality.
On the feedback survey, both online and abroad students agreed that this program helped them to: • Think critically and analytically: 83% online students and 88% abroad students.
• Understand people of other backgrounds: 83% online students agreed and 100% of abroad students.• Be an informed and active citizen: 100% of online students and 75% of abroad students.
These findings suggest that the use of technology in this case evaluation did increase student access to impactful international learning opportunities.

Student Satisfaction
Evaluation Question 2. What is the difference in student satisfaction with the overall experience between the online and abroad groups?The qualitative data revealed two main categories for feedback on student satisfaction: 1) the concept of connecting students who travel abroad with those participating online, and 2) the setting of the live meetings.The overall experience for students who traveled abroad and those who participated online were similar in their interaction with others during the program (instructor, experts in the field, online facilitator, other students).There were some differences between the groups for student perception of learning quality and learning expectations.
The most recurring area of dissatisfaction for the abroad students was due to the setting (both location and time) of the final live meeting.It took place on the last evening of the trip during the group dinner.Originally the instructors planned for this to be a family-style dinner discussion.However, the restaurant was crowded and loud which made it impossible to do a traditional live meeting.Instead, the iPad was passed around the table so that group members abroad could directly communicate with their group members online.The abroad students did acknowledge that the live meeting was interactive with their online classmates; however, they also expressed frustration at not being able to focus on the food, wine, and people physically at the table in Italy.One of the students abroad commented, "The idea of this broadcast is great, however the location and timing was not.Students in Italy were hungry, wanted to eat and reflect with other students who travelled abroad." The data provided by the students via the feedback survey showed different levels of satisfaction with some major aspects of the program: • My level of learning that took place in this meeting was of the highest quality.100% of online students agreed with this statement while 62% of abroad students agreed.Other data provided by the students via the feedback survey show similar levels of satisfaction with other major aspects of the program: • 100% of online students reported an excellent experience with the instructor and 100% of abroad students reported with a Very Good or Excellent experience.• 100% of online students reported a Very Good or Excellent experience with Experts in the Field and 75% of abroad students reported an Excellent experience.• 100% of both online and abroad students reported a Very Good or Excellent experience with the online facilitator.• 100% of both online and abroad students reported a Good to Very Good experience with other students.
In addition, the satisfaction with experience (SATEXP) subscale (α = .86),which was based on six items in the feedback survey showed that there was no significant difference in satisfaction with the overall experience between the abroad group (M = 23.75,SD = 4.86) and the online group (M = 26.33,SD = 2.25) t(14) = .25,ns.Despite some differences in student satisfaction between the groups for learning quality and learning expectations, the overall experience of interacting with others during the program were similar for students who traveled abroad and those who participated online.
Evaluation Question 3. What were online participants' reactions to the web conferencing system?According to the researcher observations, the web conferencing system was not as interactive as they had hoped and they noted the online students had more discussion in the chat with the online facilitator speaking softly as compared with the students physically present in the room who were mostly silent out of respect for the tour guide.
In the feedback survey, online students reported feeling comfortable using the Adobe Connect technology.They chose to communicate through the chat feature because it was easy to use and manageable with the online facilitator either repeating questions to the tour guide or immediately responding directly to the students.Researchers observed that at one point a student tested her microphone, but there was a lot of background noise which made it difficult to hear well.
The data provided by the students via the feedback survey (Question 8) showed different reactions to the web conferencing system.
• 100% of the online students agreed or strongly agreed that they felt comfortable conversing through this medium.
• The students felt comfortable introducing themselves in the online environment and agreed that the instructor created a feeling of an online community.• The students overall felt that they were satisfied with the usability of the system because it was simple and effective to complete the tasks.
They also commented that the system had all the functions and capabilities that met their expectations.However, there was one student out of a total of six respondents that said that learning to use the system was not easy.
Some data showed that the live sessions met their learning expectations.
• The students were neutral that the online instructional activity stimulated their desire to learn.• The live meetings allowed them to express their feelings and learn.
• The students did not find the online meeting threatening.
A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 123 • Quality of interactions with the instructor, experts in the field, the online facilitator, and the other students were indicated overall as good or excellent.
One student responded that being able to see remarkable historical landmarks in real time surrounded by people they had previously met and were actively communicating with was really interesting and enjoyable.
In one of the interviews, a student said that it was interactive and the experience was great being able to connect live, ask questions, and see with their own eyes.Another noted that the scenery from Italy was probably their favorite part of viewing the online meetings.Students liked the interaction between not only people of different groups, but different cultures.They thought it was a cool concept to engage a group from around the world and hear different perspectives and points of views.
Student feedback indicated areas of improvement for the technical aspects of the web conferencing system.This feedback will be reviewed in Evaluation Question 9 and the Discussion sections of this article.
Evaluation Question 4. What was the impact of technology on the students' ability to participate in remote instructional activities?Technology does appear to have impacted the students' ability to participate.The online facilitator observed that the online students were very participative and started conversations with one another in the chat which they preferred to use over their microphones.They asked great questions and wanted to hear responses not only from the instructor, but also from their classmates too.Therefore, technology enabled the online group to connect remotely with other students, the instructor and online facilitator, and experts in the field.
Based on feedback survey Questions 11 and 12, online students reported that there were some limitations at times regarding the quality of the sound.There were times when the video streaming was not working smoothly which hindered the online group's participation and noise of various types interfered with the technology.One student suggested, "The last meeting shouldn't take place in an environment that is noisy.It was hard to interact this way.I would suggest next time hold it in a quiet setting."Another student commented, "There were a couple of moments where the Wi-Fi wasn't as great and so it wouldn't show a good connection or picture.Overall though the instructors were really great at getting the connection back and explaining what we missed."Online students also experienced a longer delay compared to the abroad group when responding to the instructor's questions: they had to first consider their response, type it, and then wait for the online facilitator to relay the message.
According to feedback survey Question 6, only one online student replied that they felt uncomfortable conversing through this medium.Overall, all the online students reported that they felt comfortable introducing themselves in the online environment and the instructor performed well with creating a feeling of an online community.One online student disagreed that an online meeting allowed for social interaction or provided them a reliable means of communication, however, overall, the online students replied in the survey that their experience in a live interactive session was enjoyable and it helped them stimulate their desire to learn.
While several online students expressed a desire to have joined the others abroad, one online student noted a benefit of technology, "I paid considerably less and was able to continue to take other classes…" They also felt emotions transferred through the live screen as if they were in A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 124 Italy.According to the student surveys, some online students said that the technology affected their learning because they could see and experience the difference between cultures halfway around the world.
One challenge of using technology to connect students in live meetings across cultures and time zones is the need to plan and coordinate schedules.One online student noted the importance of "knowing the schedule of them being abroad ahead of time, I had to really fix my schedule around it."Also, in the feedback survey Question 18, one student abroad mentioned that at times it was hard to get organized regarding assignments with the online group.

Faculty Satisfaction
Evaluation Question 5. What were the instructor's perceptions about his teaching experience with this technology-mediated approach?The instructor reported the overall experience with the program as being positive.However, there were some areas of dissatisfaction.After reflecting upon the physical and online technicalities of the program, the instructor explained areas of concern and provided additional suggestions for improvement of future programs.
Pertaining to curriculum development, the instructor offered these suggestions for future improvement.He was dissatisfied with some of the outcomes of the group activities and would suggest that discussion prompts for students be prepared prior to the beginning of the study abroad experience.He felt that this would increase dialogue and facilitate connection between the abroad and online groups.The instructor suggested that this type of curriculum be designed ahead of time so that it is reinforced during group work to improve the learning experience of both the participants abroad and online.When in Italy with the students, the instructor would have prepared scripts for the live meetings and precisely mapped out the route of the abroad group prior to travel in order to create a more organized and less stressful teaching experience.
The instructor found it challenging to engage both the abroad and online groups simultaneously during live meetings.He explained, "Yeah, it was a real challenge for me because I found myself first catering a little bit more to the online group, like talking to the camera.And then some of the feedback from some of the in-country students noted that I was paying more attention to the camera.And then I tried to flip it, and I felt that I was not giving -I felt disconnected from the online students.For me, it was a real challenge trying to divide my attention between the two groups."Finally, the instructor expressed complete satisfaction with his online facilitator throughout the program and process of this research study.He was particularly pleased with her organization, technical skills, and overall contributions to the success of the online abroad program and its participants.He was also satisfied with the university's study abroad office.The instructor expressed that the study abroad office handled the logistics of the travel portion of the course well and took his ideas and objectives for the course and made it a reality.

Learning Effectiveness
Evaluation Question 6.What is the difference in student engagement with activities and experts in the field between the abroad group and the online group?Five categories of student engagement emerged from the qualitative feedback.The feedback for 1) quantity of questions, and 2) interaction with the technology, suggests that there were some differences in student engagement between the groups.These differences are explained in this section.Note that the feedback for the other three categories 3) quality of assignments, 4) communication between the A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 125 abroad and online groups, and 5) interaction with the instructor and experts in the field, indicates that there was not a noticeable difference.
Quantity of questions = differences in student engagement.According to the instructor interview, online students asked more questions than those abroad during the live meetings.In the student interviews, one of the abroad students said that they appreciated when the online students asked questions as it stimulated conversation and brought in fresh perspectives.Abroad students also liked when those online asked them questions instead of directing everything to the guide, instructor, or online facilitator.
Interaction with the technology = differences in student engagement.The technology appears to have created some perception of difference in engagement.A researcher noted in their observations that the abroad students who were randomly put 'on the spot' with the iPad in their direction seemed intimidated and would not talk as much as when there was no pressure with the camera.In the feedback survey, a student mentioned that they felt it hindered both the abroad and online groups because the technology detracted from the natural communication setting.It is important to note that some students commented that the technology was neither a help nor a hindrance and "was just kind of there."However, one hundred percent of the online students who participated in the feedback survey, agreed with the statement, "I enjoyed the online instructional activities."The online students reporting via the feedback survey said that the Adobe Connect web conferencing system was a beneficial tool in three main ways: • Video stream from Italy was Very Important or Critical • Audio stream from Italy was Critical • Text chat was Very Important or Critical The feedback survey responses from both the online and abroad students indicate that the majority (100% online and 87% abroad) of both groups felt comfortable participating in the group discussion and felt comfortable interacting with other participants in the live meeting.Evaluation Question 7. How were the learning experiences of the online and abroad groups similar and how were they different?While the students who traveled to Italy had a far more immersive experience, both groups demonstrated learning gains.The global citizenship pre/post assessment measures seven subscales: social responsibility, self-awareness, intercultural communication, global knowledge, involvement in civic organizations, and political voice (Morais & Ogden, 2010).Table 1 displays the subscales where each group demonstrated significant improvement including practical significance based on Cohen's effect size.Both groups showed improvement in self-awareness and intercultural communication, but the students who traveled to Italy also showed improvement in global civic activism and global knowledge.When comparing the two groups, the online students appear to have improved more on the social responsibility scale.There was a statistically significant difference between the two student groups, students abroad (M = -1.22,SD = 2.10) and students online (M = .267,SD = 1.27), t(1, 22) = 4.69, p ≤ .05.Further, Cohen's effect size value (d = .86)suggested a large practical significance.Conversely, the students abroad appear to have improved more on the global knowledge scale.There was a statistically significant difference between the two student groups, students abroad (M = 2.11, SD = 2.26) and students online (M = .40,SD = 1.81), t(1, 22) = 4.19, p ≤ .05.Further, Cohen's effect size value (d = .84)suggested a large practical significance.There was no statistically significant difference between the two groups on the remaining five scales (self-awareness, intercultural communication, global knowledge, involvement in civic organizations, or political voice).
On the feedback survey, the learning effectiveness (LEARNEFF) subscale (α = .92)also showed no significant difference between the students who participated face-to-face in Italy (M = 52.88,SD = 7.26) and those who participated online (M = 52.67,SD = 5.09), t(14) = .95,ns.This subscale consisted of 12 items that asked about the quality of their interactions with the instructor, experts in the field, the online facilitator, and other students in addition to the impact on their knowledge, skills, and personal development in the following areas: During the instructor interview, he elaborated on how the learning experiences between the two groups were similar and different.For similarity, he felt that the, "experiences were the same in that the curriculum was the same.They both had to explore the projects and the theories from the same basic standpoint…" Also, the instructor observed that the "in-field experts, the way they interacted with the face-to-face students and the online students were very similar in my opinion.I didn't notice a great difference." The instructor did think there were some differences in learning experiences.He explained: So how I think they were different: the students who were in country got to use all five of their senses where of course the ones at home just got two --sight and sound.And then from the student feedback based on culture shock, it gave some insight on how the experience could have been different as well.The students who were abroad talked about how changed they were...And the online students, they either had to stretch to answer that question and refer back to experiences they had from different travels on their own or they just said, I didn't travel abroad so I didn't experience culture shock.
Despite the similarities and differences of the learning experiences of the online and abroad groups, both provided evidence of learning gains.

Scale
Evaluation Question 8. How does the cost of this case compare to previous technologymediated attempts?Every effort was made to minimize cost in order to develop a scalable, sustainable approach.The initial proposal for the pilot study made use of satellite technology, which also required a crew to operate.Because that budget was cost prohibitive, a new Wi-Fi based approach with Adobe Connect web conferencing software and one online facilitator was actually implemented.Each course will carry a variable cost based on factors such as the duration or location, but it closely approximates the cost for an individual student enrolled in the travel option.For these studies, costs were reduced by utilizing an existing Adobe Connect license and equipment such as iPad, laptop, and webcam.
Evaluation Question 9. How can this approach be improved?Critical Feedback.In addition to positive feedback, the online survey and interviews provided clear areas for improvement.Some were more pedagogical in nature while others were technical.The technical areas of concern included audio and Internet connection quality.In the feedback survey, one student suggested, "have the tour guide wear a microphone of sorts just because it was really hard to hear at times."Another responded with "The last meeting shouldn't take place in an environment that is noisy.It was hard to interact this way.I would suggest next time hold it in a quiet setting."Three observers also noted concerns about background noise.For example, one reported "since the broadcast took place in a restaurant, noise disruptions were present for online students" and another recorded "this broadcast would have worked perfectly if it was done in a quiet setting without many distractions."The online facilitator also noted in an observation form "I had to stay within a few feet of the tour guide to make sure online students could hear well.Possibly consider a better microphone in the future." During two of the live sessions the Wi-Fi disconnected briefly.While students were prepared for this possibility in advance, it is never desirable.In an interview one student explained, "Every now and then the live sessions would pause or cut out and I know that is just the way technology works but maybe with technological advances it would be possible to have a better connection in the future."Whenever possible, the instructor and online facilitator visited the sites in advance to test the connection and prepare contingency plans.
One of the pedagogical concerns was related to efficient use of time and engaging students in constructive dialogue during the live sessions.The online facilitator observed that during the tours often there were stretches of time where the group in the field was walking from one stop to the next, and during that time she felt the need to improvise and fill that time with group discussion to keep the online students engaged and encourage student to student interaction.Student survey comments such as "have more interactive questions" and "It would have been nice if there was a little more participation from the students in Italy" justify the need for preparing discussion prompts in advance.One of the researchers suggested, "As an icebreaker, encourage dialogue between online students with those abroad regarding initial thoughts about the trip.This would also probably make students abroad feel more comfortable talking in front of a camera."The instructor also reported challenges trying to balance the two different audiences and give them equal attention, and prepared discussion prompts may be directed to the entire group for a more inclusive experience.
The other pedagogical concern focused more on group dynamics.In this particular course, two students in the field were paired with three online students to complete a cultural comparative group project.In their interviews, several students who traveled abroad reported that they experienced challenges working with their online partners at a distance and would have preferred to work just with their fellow travelers with whom they had bonded.For example, one student stated, "I feel like it would've been easier for all of us in Italy to have done a project together and then everyone in Florida to have done a project together so we could talk face-to-face about our project."While this would negate one of the primary learning objectives of the course (using Providing Access to Study Abroad Activities 129 technology to collaborate at a distance), it does indicate that the group formation process was rushed prior to physically separating.In their interviews, several students requested more group time in class prior to the trip.

Recommendations for improvement.
The feedback provided by both sets of students, the instructor, the online facilitator, and observers resulted in the following recommendations for improvement: • Improve audio quality.Often background noise in the field made it difficult to hear the instructor or tour guide clearly.While some noisy environments like restaurants may add to the immersive cultural experience, it should not be allowed to negatively impact instruction.This could be improved by using a directed microphone or a lavalier microphone on the instructor or meeting in a private room rather than in the main dining room of a restaurant.
• Test Wi-Fi strength in advance.It may not always be possible to test in advance and sometimes the signal suffers during the live event even when the test went fine, but every effort should be made to minimize this risk.Testing signal strength in the various locations also gives the instructor and online facilitator an opportunity to work out backup plans, but they should also be prepared to improvise when unforeseen connection issues arise.
• Prepare question prompts.Often there are brief downtimes in the field when the group is moving from one location to another.Together with the instructor, the online facilitator could prepare discussion prompts in advance in order to make effective use of that time and promote student engagement.
• Establish team roles, communication protocols, and dynamics.Provide groups more time to connect and bond during the weeks prior to splitting up and assist with the group formation process.This may include requiring pre-travel group activities or assignments in class, providing roles for individual group members, and coaching or structured guidance as they work in their teams.

Explanation of the case evaluation
The 6-week course "Communication, Culture and Technology: An Italian Case Study" combined a traditional study abroad experience with an innovative twist: live meetings for students who have chosen to join the group abroad virtually.The overall goal was to make study abroad more accessible to students in order to promote the acquisition of global competencies.Building upon the work of Howard and Gunter (2017), the introduction of technology brought the abroad experience into the homes of students who would otherwise miss out on a valuable international learning opportunity.Two student groups were enrolled in the course: an abroad and online group.Both groups met together on campus for the first three weeks prior to traveling abroad.When designing the curriculum, both student groups needed to have the same course objectives.To accomplish this, the assignment activities of the course required that classmates work in groups composed of both abroad and online students to produce a module on some aspect of culture and communication.
A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 130

Findings from the case evaluation
Students liked that they were able to hear different perspectives and points of views.They thought it was a cool concept to engage with a group around the world to have them experience Italy from their professor's perspective and ask questions to the class.
Students reported the challenge of feeling emotionally connected to their group members who were not physically with them.The abroad students naturally bonded with one another through shared emotional experiences from their Italian travels.While the abroad students did collaborate with their online group members on the projects, the nature of the relationship seems to have been more information-focused than relationally-focused which does not naturally lead to bonding through shared emotional experiences.
A possible hindrance between the groups was that the online students liked to use chat and chose not to utilize a microphone during the broadcasts.Therefore, the online facilitator had to act as a liaison and connect the two groups by repeating the online student questions aloud to the abroad group.They were not directly talking to one another.
The online students asked more questions than those abroad during the live meetings.An explanation of this could be that the abroad students were with the instructor 24 hours a day and had access to ask questions at any time and not be limited to just the live meetings.Online students would either need to email the instructor with a question or ask during the live meetings.
Finally, the instructor found it challenging to engage both the abroad and online students simultaneously during live meetings.Stronger relationships seem to have been formed with the abroad students than the online students.The instructor suggests that the relationship-building aspect that comes from the travel abroad experience may not be able to be replicated via technology.

Recommendations for future case evaluations
While this case evaluation included just one online section, it could be possible to increase the number of online sections to allow other related courses to join the live meetings for specific tours that are related to their coursework.When replicating similar programs in the future, this case evaluation generated several recommendations: 1) improve audio quality 2) test Wi-Fi strength in advance 3) prepare question prompts, and 4) establish team roles, communication protocols, and dynamics.
A Bluetooth microphone might improve the audio quality during the live meetings.Wi-Fi should be tested prior to broadcasts to determine its strength both indoors and outdoors.The instructor can plan alternative lessons in the event that the Wi-Fi does not permit broadcasting.The lesson should include question prompts to engage both abroad and online students during the live meetings.Finally, official roles for each group member could help the teams establish better dynamics.More relationship-building exchanges could help create a stronger bond between abroad and online group members as well as with the instructor.Also, allowing the student groups to interact directly via the iPad can allow the abroad students an opportunity to experience the role of the online facilitator as an additional benefit.
A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 131

Conclusion
The purpose of this study was to evaluate the effectiveness of an online delivery model for study abroad activities.Twenty-six undergraduate students studying intercultural communication were able to participate in a study abroad experience in Italy: 10 students participated in the traditional study abroad trip while the other 16 participated online.This case evaluation continues the cyclical, design based research started by Howard and Gunter (2017) to refine and further develop the web conferencing approach to internationalizing the curriculum through online instruction.
The study was driven by three guiding questions: (1) Did the program accomplish its goals?(2) What was the impact of how it was implemented?(3) What changes to the design are necessary to firmly establish it as a valuable alternative to traditional study abroad?These questions were evaluated through a combination of anonymous surveys, pre/post assessments, observations, and student and instructor interviews.
Overall, the program accomplished its goals and provided a valuable international learning opportunity.While the students who traveled to Italy had a more immersive experience, both groups demonstrated learning gains.Both groups showed improvement in self-awareness and intercultural communication.These findings expand the established history of study abroad programs by furthering demonstrating the benefits for students (Chieffo & Griffiths, 2004;McKeown, 2009;Spencer & Tuma, 2002;Tarrant, Rubin, & Stoner, 2014;Vandeveer & Menefee, 2006).
The way the study was implemented did have an impact.The students who traveled to Italy showed improvement in global civic activism and global knowledge.The online students were able to join the live tours and discussions, thus expanding their access to international experiences that normally would be closed to them.One online student thought that the live meetings were, "a good enough way to experience another culture while not having to spend the extra money."Another student thought that the university "should offer more classes like this for people who don't have the opportunity to go abroad.It's a good opportunity to get the experience without being there."Other students referred to the live meetings as a way to, "explore Italy side by side with those who went abroad." The program inspired students from both the abroad and online groups to travel and explore cultures.One student said, "This class has made my desire to travel stronger and I feel I am better prepared than before."And another commented, "I would still do it again.I would still want to do the long dining restaurants..." Finally, the research revealed some changes to the design that are necessary to firmly establish the online option as a valuable alternative to traditional study abroad.Future programs implementing a similar model that connects online and abroad students should 1) improve audio quality 2) test Wi-Fi strength in advance 3) prepare question prompts, and 4) establish team roles, communication protocols, and dynamics.
Based on these findings, the instructor and online facilitator for this case evaluation encourage future program leaders or facilitators with a passion for study abroad to consider implementing an online model to increase student accessibility for international learning experiences.

•
Speaking clearly and effectively • Thinking critically and analytically • Acquiring job-or work-related knowledge and skills • Working effectively with others • Developing or clarifying a personal code of values and ethics • Understanding people of other backgrounds (economic, racial/ethnic, political, religious, nationality, etc.) • Solving complex real-world problems • Being an informed and active citizen A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 127

•
Overall this session met my learning expectations.100% of online students agreed with this statement and 62% of abroad students agreed.
A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 122

Table 1 .
Significant Improvements on Global Citizenship Pre/Post Assessment

Table 2 .
Table 2 compares the original proposed costs, the pilot study cost, and the actual cost of this study.Cost Comparison A Formative Case Evaluation for the Design of an Online Delivery Model Providing Access to Study Abroad Activities 128