Systematic Reviews of Research on Online Learning: An Introductory Look and Review

In this introduction to the special issue on systematic reviews of research on online learning, we introduce the need for systematic reviews on online learning. Utilizing a three-tier lens focusing on systems, pedagogical, and people levels, we have selected nine articles for this issue. At the systems level, there are two articles that focus on research trends during COVID-19, and features of high-quality online learning. At the pedagogical level, five articles were included that address online learner collaboration, help-seeking strategies, intersubjectivity, invisible participation, and online assessment. Finally, at the people level, there are two articles. The first focuses on online learning for minoritized and first-generation students. The second examines moderators in asynchronous online discussions. This introductory article provides a short summary of the nine articles and concludes with implications for practitioners and researchers on using and conducting systematic reviews on various topics in online learning.

approaches for understanding a new popular delivery method as HyFlex (Beatty, 2019) or fully online learning.

Need for Systematic Reviews of Research
Systematic reviews rely on a methodology used to "examine secondary data by retrieving, synthesizing, and assessing existing knowledge on a subject in a logical, transparent, and analytical manner" (Martin, Dennen et al., 2020, p.1613).Systematic reviews address critical questions and synthesize sources that otherwise might be considered inconclusive and smallscale.Early research in an area typically focuses on what Borko (2004) refers to as "existence proofs," or one-off studies of individual implementations.It takes time for a more systematic, mature body of research to emerge and fill research gaps.As research accumulates and matures, systematic reviews not only help to identify research themes and answer critical questions but also provide an opportunity to address topics of mixed findings (Ioannidis et al., 1999).
Systematic reviews have several benefits, including a reduction in bias due to the use of a transparent and rigorous process, a greater study breadth due to thorough searches, and the quality of primary research examined.However, conducting systematic reviews also present challenges; high quality reviews, for instance, are time intensive.Other methodological challenges exist, including research questions that are often defined too broadly or narrowly, a lack of access to certain research or publication databases, and subjectivity during the screening and coding process.Nevertheless, benefits outweigh challenges in most cases and offer findings that guide research and practice.

Focus of Systematic Reviews of Research in this Special Issue
This special issue features seven systematic reviews and two scoping reviews.To foster a better understanding of the state of online learning research, we have structured the issue by focus area: (1) systems level; (2) pedagogical level; and (3) people level (see Figure 1).The first and third sections each contain two articles, while the middle section contains five.At the systems level, the issue includes reviews focusing on research trends during COVID-19 and examining the features of high-quality online learning.At the pedagogical level, reviews on engagement and assessment are featured, including collaboration, help-seeking, invisible participation, intersubjectivity, and online learner assessment.The people level contains a review of the research on the role of moderators in an asynchronous online discussion and a review of the research on online learning for minoritized and first-generation students.

People
Table 1 provides the author names and titles of the articles in this special issue.

Systems Focus
Each study of online teaching and learning can prove helpful in understanding how to design high-quality and engaging online activities, courses, and programs.Information about the nuances of each pedagogical strategy and refinement to that instructional approach helps instructors and instructional designers to design and deliver new online courses.Sometimes, however, an understanding of the overall system in which online learning operates is warranted.Two articles in this section offer a systems-level focus.The first, by Doo et al. (2023), explores general research trends during the pandemic.The second, by Wright et al. (2023), investigates the components and factors that enhance online course quality and foster learner success.

Research Trends in Online Learning During the Pandemic
The Doo et al. (2023) article explores the research topics published from the start of the pandemic in early 2020 to April 2022.The article begins with a historiographical discussion of online and distance education research, especially useful to graduate students and novices to understand the evolution of online teaching and learning.Doo and colleagues then detail a couple of existing reviews of the research on online learning during the pandemic, a practice that has often been labeled "emergency remote teaching" (Hodges et al., 2020).There is much to glean from this review, as their findings provide a coherent picture of trends in the research in online learning during the past few years.Doo et al. (2023) decided to utilize a framework from Martin, Sun et al. (2020) which was first designed and used to summarize the research on online learning from 2009 to 2018.In effect, the Martin, Sun et al. (2020) study combine with the present Doo et al. (2023) research to offer a more complete picture of the topics researched during the past decade as well as the shift in online learning researcher attention during the pandemic.Interestingly, the Doo et al. (2023) study found an uptick in the research on course design and development, course technology, teachers' experiences and perceptions, and instructor characteristics during the pandemic.Unsurprisingly, learner engagement has remained a highly targeted area of research over the past couple of decades.This timely review also identified two new areas of research: parent involvement in online learning situations and adaptation to online learning.Neither category was surprising, given that millions of parents and children were at home during the pandemic and had to adapt to a virtual learning environment.
Those reading the Doo et al. (2023) article will gain insights into the topics that are increasing in salience.They will also better understand the journal dissemination outlets for research on online learning.Clearly, the 191 studies analyzed for this systematic review indicate that online learning research has received increased global attention.Educators, researchers, parents, and politicians have all been impacted by online teaching and learning and, therefore, are interested in it.More interestingly, perhaps, is the shift from a heavy emphasis on learner engagement and characteristics to now include research on online course development, the technology tools and features utilized in such courses, and instructor training for online settings.

Features of High-Quality Online Learning
The second article found in the systems level section, by Wright et al. (2023), explores the components of high-quality online courses.And, as with the Doo et al. (2023) article, an interesting historical overview is provided, offering a better grasp of the common frameworks which have been employed in the past to understand online course quality, such as Community of Inquiry (CoI), as well as more recent frameworks such as Universal Design for Learning (UDL) and Quality Matters (QM).The proliferation of online and blended forms of learning such as Hybrid-Flexible course design (i.e., HyFlex; see Beatty, 2019) across all sectors of education heightens concerns about the quality of those courses.
This article demonstrates that the components of quality are wide ranging and include technology systems, platforms, and tools employed as well as the course designs and organizational structures, pedagogical strategies and refinements for an engaging online learning environment, and the methods of assessment employed.For those seeking an accessible overview of course quality components and considerations, Wright et al. (2023) provide an excellent summary and insights about online course communication practices, discussion guidelines, appropriate feedback mechanisms, valuable organizational components, and a few assessment considerations for high-quality online courses.
What seems apparent is that enhanced course quality should provide some degree of flexibility in course design and delivery, more than one mode of communication between instructors and students, and multiple means of assessment.At the same time, Wright et al. (2023) caution that there must also be some sense of balance in terms of instructor presence in the course to prevent instructor burnout.Wright and colleagues acknowledge that balance could come from relying on additional sources of course support and feedback such as teaching assistants, tutors, and artificial agents.The article suggests future research related to the professional development and training of instructors who teach via online delivery.

Pedagogical Focus
In attempting to clarify common research themes in distance education, Zawacki-Richter et al. ( 2009) categorize management, organization, and technology at the meso level, and teaching and learning in distance education at the micro level.At the micro level, focusing on teaching and learning, Zawacki-Richter et al. ( 2009) include instructional design, interaction and communication in learning communities, and learner characteristics as some of the primary research areas examined in distance education.Focusing specifically on the online learning environment, this special issue offers review articles on engagement and assessment.The five studies with a pedagogical focus include recommendations for the design and delivery of online courses critical to online teaching and learning.
Student engagement is crucial in online learning as it is more likely that learners will drop out of the learning process if they are not engaged.Martin and Borup (2020) define online learner engagement as "the productive cognitive, affective, and behavioral energy that a learner exerts interacting with others and learning materials and/or through learning activities and experiences in online learning environments" (p.164).While educational psychology has emphasized the importance of affective, behavioral, and cognitive engagement, this research emphasizes the importance of reflecting on communication, collaboration, presence, interaction, and community in the online environment.
Like engagement, assessment is critical to the learning process, and a few systematic reviews have focused on online assessment (Gikand et al., 2011;Wei et al., 2021).Gikandi et al. (2011), for example, examined 18 studies to study effective online formative assessments, and Wei et al. (2021) synthesized 65 studies focusing on different assessment types in MOOCs.However, the need for a systematic review to broadly examine online assessments is addressed in this issue by Heil and Ifenthaler (2023) who synthesized publications for assessment modes, formats, and types.

Learner Collaboration
Over the past few decades, online collaboration has gained prominence in both educational and workplace settings.Several waves of technology tools have emerged for online collaboration and teamwork since the early 1990s (Bonk et al., 1994;Bonk & Wiley, 2020).As a result, there is a pressing need to determine the effectiveness of such tools in online environments.In response, Oyarzun and Martin (2023) conducted a systematic review of research on online learner collaboration which examined collaborative technologies, design, facilitation, and outcomes.Particularly, they refer to online learner collaboration as "student interaction that supports socially constructed meaning and the creation of knowledge." In their timely review, Oyarzun and Martin (2023) and Martin synthesized findings from 63 studies; importantly, in this review of the research on collaborative technologies, they found that learning management systems (LMS), discussion boards, writing tools, and synchronous tools were the technologies primarily selected for online learner collaboration, whereas wikis, blogs, social networks, and annotation tools were employed in just a few studies.The most used collaborative methods were group projects and discussions, with fewer studies mentioning peer review, social/informal, and collaborative experience surveys.In addition, they also examined group size and instructor roles to enhance online learner collaboration.Based on Oyarzun and Martin's review, increased learning, communication, and collaboration skills, and relationshipbuilding were the top opportunities, whereas time, technical issues, and anxiety/fear/stress were challenges that appeared most frequently in online learner collaboration research.

Help-Seeking Strategies
Like collaboration, negotiating and contributing to the online environment is important, and students frequently need assistance in these areas.Just how and when do online learners effectively seek help in their online courses and activities?To investigate these questions, Yang and Stefaniak (2023) explore help-seeking strategies in online learning environments.According to the authors, help-seeking occurs when learners identify a gap in their understanding and seek help to bridge the gap.
In their review of 36 articles, Yang and Stefaniak (2023) outline four types of helpseeking: formal help-seeking, informal help-seeking, instrumental activities, and altering goals.The authors identified a need for additional research studies examining learners' psychological decision-making process when they lower performance aspirations or alter their online learning goals.Most of the studies in their review focused on formal and informal help-seeking strategies, which is not surprising given the proliferation in ways to learn informally and self-direct one's own learning during the early decades of the 21 st century (Bonk, 2009(Bonk, , 2016;;Bonk et al., 2016).Significantly, these authors call for generalizable studies rather than small case studies.

Intersubjectivity
Related to the prior two articles in this issue on online learner collaboration and helpseeking behaviors is an article that looks at the research on intersubjectivity in online learning.In their systematic review, Dennen et al. (2023) explore research on intersubjectivity, a psychological construct that is a foundation to meaningful learner engagement.Intersubjectivity, which is necessary for mutual understanding to develop, can be evident through archived interactions in both synchronous and asynchronous environments.Unfortunately, not all online interactions achieve intersubjectivity.Instead of attempting to foster it, grading systems in online courses frequently focus on indicators like post count, word length, and accuracy, or quality of content (Dennen, 2008), none of which provide evidence of either engagement or intersubjectivity.
To address this issue, Dennen et al. (2023) examined 48 studies related to intersubjectivity.Their review showed a very slow but steady stream of publications in this area; however, a deeper examination of cross-citations shows that this research has two strands.One strand is focused on asynchronous discussion, and the other on synchronous learning, primarily in language learning contexts.These strands are not connected, and even within these two strands the research is not heavily interconnected or even representative of a systematically developing research area.Nevertheless, Dennen et al. (2023) reaffirm the importance of intersubjectivity as an underlying construct that influences discussion-based learning and encourage future researchers to pursue this area, noting that greater research focus on intersubjectivity could lead to improvements in practice.

Passive Participation
How learners participate in, or contribute to, an online course can entail vastly different behaviors from what they exhibited in face-to-face courses.Participation can occur at any moment during the course, not just during a limited allotted block of time once or twice a week.Highly reflective and introverted learners, as well as those who are concerned with their language skills, might be deemed to be passive in online courses.In exploring this topic, Choi and Hur (2023) conducted a scoping review of passive participation, an online learning phenomenon in which students are present in the course space but not actively posting messages and interacting with their peers.Instructors might mistakenly consider these students to be absent from the course or believe they are not learning, but the reality can be much more complex.
Examining 42 studies and considering a behavior that goes by different terms (e.g., "lurking" or "listening"), Choi and Hur (2023) found that researchers attempt to understand when and why students are passive participants and how it affects their learning outcomes.Other researchers seek to reduce this behavior, viewing it as a negative form of interaction.Through their review, the authors demonstrate that passive participation remains an underdeveloped research area, with more work needed to understand how learning outcomes are affected and how different pedagogical strategies might shape this behavior.

Assessment
As online forms of teaching and learning accelerate across K-12, higher education, and workplace settings, vital questions remain about assessment practices.Too often, insufficient or surface level answers are provided by researchers and those asking the important assessment questions walk away disappointed.Hence, it is vital to turn to the research literature for such answers, insights, and guidelines about effective online assessment.Heil and Ifenthaler (2023) provide some answers in the next article of this issue.Their review describes online assessments as processes through which information and communication technology is used to gather information about learners and the learning process to make inferences about learner dispositions.In their systematic review synthesizing 114 publications, the authors analyzed assessment modes (i.e., peer, teacher, automated, and self-assessment), assessment formats (i.e., format or summative), and assessment types (i.e., quiz, essay, etc.).The authors also examined the objectives and success factors of online assessments in higher education.Their implications include how online assessments support student learning, but also extend possibilities by, for example, providing feedback and assessing collaboratively.They conclude that setting expectations is critical to the assessment design process and encourage the selection of various modes, formats, and types of assessments in online learning.

People Focus
The final articles in this issue focus on individuals and their role in the learning context.For all the emphasis placed on designing educational systems at the macro level, curricula at the meso level, and courses at the micro level, course members are the drivers of course interactions (the nano level) no matter how highly designed the course is.Dennen (2022) notes that course design elements such as content and technology are important as learning enablers but that ultimately, learners and instructors have agency and determine what occurs in a course.
Most learners have experienced online learning for many years and perhaps even decades.As a result, online learners bring personal experiences, prior knowledge, and individual needs to the learning context.As expected, they can seamlessly navigate a course as designed.However, they can also purposefully push back on the course design and struggle when there is a mismatch between the course design and their individual characteristics and choices.They may function as a group of individuals, but also may find a new collective identity as they learn together.In other words, as much as one might like to think about online course design and outcomes as a top-down endeavor, the people involved in day-to-day course activities exert pressure on course design in a bottom-up manner.
To assume that behind the keyboard all online learners are alike would be naïve.Not only do they bring different backgrounds, including areas such as socioeconomic status (Yalcin, 2022) and nationality (Choi et al., 2020), but they also navigate their online identities in varied ways (Dennen 2021;Dennen & Burner, 2017).Similarly, it would be erroneous to assume that online instructors simply execute a predetermined design, adding nothing unique to a class.The instructor role in online classes is multifaceted and requires being responsive to student characteristics and needs (Berge, 2000;Bonk et al., 2001;Dennen & Jones, 2022).For this reason, the study of students and instructors as autonomous agents within the online learning context is important, considering not only how each performs in class (i.e., outcomes) but also what they bring to, and need from, the learning experience.

First-Generation and Underrepresented Minoritized Students
One article in this issue examines research on a specific student group: first-generation and underrepresented minoritized students.In their review, Gardner and Leary (2023) focus on the challenges that these students face and the supports that they need to be successful in an online learning setting.Drawing on Borup et al.'s (2020) student engagement model, they consider the experience of these students in affective, behavioral, and cognitive dimensions.Their broad search yielded 42 articles, from which they identified 15 themes across three major areas related to the student experience: (1) Learner Characteristics, (2) Personal Environment, and (3) Course Environment.Using these themes, the authors discuss challenges and offer recommendations for how online courses could better meet the needs of first-generation and underrepresented minoritized students.Ahlf and McNeil (2023) provide an overview of the research on the moderator's role in asynchronous online discussions (AODs).They appropriately note the wide diversity in research questions as well as in research designs, areas targeted in that research, and overall outcomes.

Role of Moderators in Asynchronous Online Discussions
The historical overview of moderation in the field of online discussion in the initial pages of this article may prove as valuable to people reading this article as the actual research results from this systematic review of moderation in AODs.As Ahlf and McNeil (2023) highlight, the frameworks and models cited from leading AOD scholars such as Gilly Salmon, Andrew Feenberg, Zane Berge, and Panos Vlachopoulus have been effectively utilized for decades.Ironically, when the AOD field arose four decades ago, it quickly attracted researchers and theorists who designed frameworks that have withstood the tests of time.Importantly, this article offers an insightful taxonomy of moderator roles.
There is much to reflect on in the Ahlf and McNeil (2023) article.For instance, the article forces one to consider the history of the field and the shifting nature of the countries of the researchers conducting research on AOD moderation.It is apparent that AOD research seems to have intensified from 2007 to 2012, with 26 of the 52 included studies published during that time.It is also vital for early career scholars to note that the earliest research took place in the 1980s; in effect, this is a quite established field with a comparatively long history.
What should also be of value to young scholars and those new to this field are the types of research conducted about moderation in AODs where single case studies are predominant, followed by experimental designs and qualitative research.The many tables included are also highly informative in laying out the themes in AOD research and descriptions of those themes.What they most obviously indicate is a wide range of potential AOD roles and expectations within those roles (e.g., weaver, knowledge construction supporter, question asker, metacommentor, feedback giver, leader, guide, manager, social facilitator, etc.).Clearly, the topic of moderation in AODs has far ranging implications in terms of both the design and the success of an online course.As with much of the research discussed in this issue, the field of moderating AODs may have deep roots but, as the authors point out, is nonetheless still evolving with much discord to address and resolve.

Conclusions and Recommendations
The systematic reviews in this special issue of Online Learning, while comprehensive within their defined scopes, do not in aggregate provide a comprehensive overview of all research in online learning.Still, taken together, these articles have a collective value.What the articles share is an interest in pedagogy, which they examine from different vantage points and angles.As researcher lenses shift from the macro to the micro level and pan across varied research themes, these articles yield insights into the nature of online learning and its pedagogical trends, including detailing specific learning and assessment strategies and lending thoughts into the agency of learners and instructors in the online classroom.Importantly, the articles also provide meaningful recommendations for online learning practitioners.
Viewed as a whole, this special issue provides anchor points of knowledge in the broader landscape of online learning.Most of the authors have situated their systematic reviews in the context of earlier systematic reviews of online learning.When examining well-established research areas, they nest their findings into existing gaps or extend earlier reviews.Other reviews represent forays into newer areas where research has yet to fully mature; in effect, they offer an initial vision of what is known and how future studies might connect and extend some of the earlier research.In the spaces between these systematic reviews and the ones that came before them are gaps that remain to be filled.Some of these holes represent knowledge about directly related parallel topics (e.g., research on different learner groups or different pedagogical strategies), while others represent complementary topics (e.g., online learning technology, policy, and administration).
Beyond the content-focused insights offered by these systematic reviews, the articles also serve as models for future online learning reviews.They demonstrate varied ways of viewing and synthesizing a body of related research, including the use of existing frameworks, development of new thematic coding systems, and examinations of time, trends, and even cocitation.They provide methodological guidance and leave ledges onto which future researchers can develop future studies with meaningful foundations as well as update these reviews as years pass and additional research is conducted and published.Future researchers are encouraged to also focus on meso-level topics such as management, organization, and technology as this special issue did not include any studies on them.
Our hope is that readers enjoy the nine articles found in this special issue and utilize their insights in their own future research, teaching, or research translation efforts.Whatever your intended use or situation, we wish that you find this issue informative and beneficial.Given that the application and impact of online learning during the coming decade will likely continue the rapid pace set in the previous ones, there will be assorted uses and applications, many of them unintended or unplanned, of this issue of online learning research as well as the many such journal issues to follow.

Table 1
List of Articles in Special Issue