The Role of Evaluation Methods in Health-Related E-learning: A Rapid Review

Training and development programs are increasingly delivered online with numerous studies reporting no differences in learning outcomes between online and traditional learning. However, there are no established standardized methods to evaluate the effectiveness of online learning. This review aims to map the state of research around health-related education to determine what e-learning evaluation methods are being used, the strengths or deficiencies of these methods, and which are appropriate for measuring the effectiveness of online education. Databases searched were PubMed, ProQuest, Education Resources Information Centre, Cumulative Index to Nursing and Allied Health Literature, Scopus, PsychInfo, and Medline. Studies were included if they were published between 2011 and 2021, reported health-related online education and included an evaluation component. Thirty studies were obtained from numerous countries with varied methodologies and designs. Participants ranged from undergraduate students to medical professionals. Evaluation methods included student participation, students’ reaction to the training program, self-efficacy, knowledge assessment, long-term performance, and the Kirkpatrick Evaluation Framework. The review identified that course evaluations, such as measuring student satisfaction scores alone, are insufficient when used to quantify learning effectiveness for online education. This was particularly important as studies are reporting these single metrics as positive effects of training interventions without justification. Suggestions within the reviewed papers were to adopt and implement an appropriate validated method within the course curriculum to evaluate learning outcomes.

Stemp, J.D., Ghosh, D., Khan, U. R., & Boyd, J. H. (2022).The role of evaluation methods in health-related e-learning: A rapid review.Online Learning,26(4), 369-398. DOI: 10.24059/olj.v26i4.3115Workplace educational training and development plays a critical role in staff development and organizational efficiency, helping organizations achieve goals and objectives.The way training is designed, delivered, and implemented contributes to the success or failure of these outcomes (Salas et al., 2012).The last decade has seen the workplace training function driven by a legal requirement to ensure businesses comply with regulations, such as health and safety requirements (Khan, 2011).
In many organizations, training and development opportunities have been encouraged to improve staff skills and improve operational efficiencies (Hughes et al., 2016).This has resulted in an increase in professional development opportunities to extend skills and knowledge in the workforce and allow organizations to take advantage of technological advances.
As part of quality improvement and patient safety in health, Australia introduced continuing professional development requirements in 2015 to educate staff about current advances in health and care practices and the use of innovative technologies in healthcare (Australian Health Practitioner Regulation Agency, 2019).
The recent 2019 coronavirus pandemic (COVID-19) has had a major impact on teaching and learning, with organizations and higher educational facilities worldwide shifting to online platforms instead of the traditional face-to-face learning environment (Dhawan, 2020;Pokhrel & Chhetri, 2021).In health for example, e-learning in specialized medical training, such as in surgical settings, can include virtual patient cases, digital modelling, online tutorials, and standardized videos and images (Jayakumar, 2015).
Despite large investments in workplace education and professional development activities, there is little evidence about the effectiveness of online education compared to traditional face-to-face learning (Vaona et al., 2018).There is a variety of individual metrics for measuring training effectiveness and evaluation frameworks like the Kirkpatrick evaluation model (Kirkpatrick, 1994;Kirkpatrick & Kirkpatrick, 2006) for the measurement and evaluation of learning.However, there are no agreed standardized methods to measure effectiveness and no assessment of outcomes between online and traditional learning (Vaona et al., 2018).

Background
In response to the COVID-19 pandemic, governments around the world introduced a series of phased restrictions and lockdowns to manage the spread of the disease.This included limiting face to face interactions and encouraging online work, training, and education.In health, the COVID-19 pandemic resulted in a significant increase in e-learning across many aspects of the professional development education and training program.This highlighted a need for better measurement, assessment, and evaluation of online learning.
With a significant uptake in online education and training, health organizations need to ensure that professional development training allows health care professionals to maintain and improve standards of practice through the development of knowledge, skills, and behavior.This process requires robust methods for the measurement, assessment, and evaluation of online education.In this rapidly changing environment health organizations are keen to know about the changes in e-learning practices and outcomes across all aspects of health.This rapid review identifies some of the new and emerging methods and practices for evaluating e-learning.This includes building on previous reviews that were limited in focus and identifies changes to practice, to assess what is already known about e-learning practice and gaps in evaluation methods.

Previous Systematic Reviews
In the past ten years there have been seven health-related systematic reviews undertaken to investigate the effectiveness of online training to improve participants' knowledge, skills and competencies (Barteit et al., 2020;Campbell et al., 2019;Lima et al., 2019;Moehead et al., 2020;Rouleau et al., 2019;Salter et al., 2014;Zafar et al., 2014).Looking at e-learning in nursing, pharmacy, radiology, dementia, and orthodontics.Many of the reviews identified a need for better measurement, assessment, and evaluation of online learning.
The challenges reported within these reviews highlighted a need to understand whether elearning models can improve professional practice, professional knowledge, and the long-term effects compared to face-to-face learning.Many of the studies identified in the reviews were small-scale and short-term, often with limited granularity of reported details, overrepresentation of the effects of e-learning intervention, and underrepresentation of patient and practice outcomes.
Previous systematic reviews have focused on very specific areas within health without looking across the health landscape to identify and report different practices.This review covers ten years (including two pandemic years) during which advances in internet bandwidth, technology, and software have supported a shift to online training.

Objectives of the Review
This review aimed to identify new methods of measurement and assessment, as well as gaps and limitations to initiate discussion of valid evaluation within the health field.The objective is to map the state of research to determine what evaluation methods are currently used in health-related online education.In addition, the review aimed to summarize the strengths and limitations of these evaluation methods and recommend which of these methods could be used to measure the effectiveness of online health education.

Methods
We conducted a rapid review to identify online education evaluation methods specific to health-related training.A rapid review is an abbreviated systematic review that gathers and synthesizes study findings in a short amount of time.A rapid review can be used to address a wide range of issues and to help provide recommendations that can be used to inform policy and systems decisions (Tricco et al., 2017).Methods and results were reported using Rapid Reviews to Strengthen Health Policy and Systems (Tricco et al., 2017) and the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Page et al., 2021).See Supplementary Table 1.For this review, we define online learning and e-learning as an educational intervention that is delivered electronically through computer networks with no physical classroom attendance.The review does not include face-to-face or blended education models.

Protocol Development
The protocol was developed based on the population, intervention, comparison, and outcome (PICO) framework (Huang, Lin, & Demner-Fushman, 2006).Table 1 provides an overview of the protocol used to inform the search strategy.

Population
Health care professionals or health-related students.

Intervention
Health-related courses delivered online with no face-to-face component.

Comparison
Type of evaluation method used.

Outcome
Performance, effectiveness, and limitations of the evaluation component.

Database Search
Seven databases (PubMed, ProQuest, Education Resources Information Centre [ERIC], Cumulative Index to Nursing and Allied Health Literature [CINAHL], Scopus, PsychInfo, and Medline) were searched for studies published between 2011 and early 2021.Using appropriate search strings and Medical Subject Headings (MeSH), the keywords used were related to the PICO framework and included e-learning, performance, efficiency, evaluation, assessment, and Kirkpatrick (see Table 2).

Inclusion Criteria
Eligible studies were defined as scholarly, peer-reviewed articles published between 2011 and 2021 available in full text.A timeframe of 10 years was agreed upon and selected to limit the results of studies published using distance education methods described above.This included e-learning interventions relevant to health involving higher education students or healthcare personnel (i.e., continuing professional development) with an evaluation component.To understand the effectiveness of the evaluation components, eligible studies were required to report on the strengths and weaknesses of the methods used.

Exclusion Criteria
Grey literature articles, book chapters, conferences, opinions, proposals, or comment pieces were excluded from the review.In addition, technology acceptance, software evaluations concerning the e-learning platform, and medical interventions (such as clinical trials) were also removed since learning effectiveness was the focus of the review.Any blended or hybrid learning models, which included face to face or correspondence-based learning not completely delivered online, were excluded.

Screening and Study Selection
After restricting the database search to full-text, peer-reviewed articles, a total of 462 studies were retrieved and imported into Covidence Systematic Review Software (Veritas Health Innovation, 2019).Of these, 105 were duplicates leaving 357 for screening.Two reviewers screened the studies for relevance based on titles and abstracts, and then later by full text.The screening strategy was broad, looking to exclude articles that were not health-related, had no mention of an online education component or met the exclusion criteria.Uncertain articles were retained for review in the full-text screening stage.Of the 357 studies, 108 were retained for fulltext screening, and 81 were finally excluded.The final 27 articles were considered appropriate and retained.Reference lists were exported from the Scopus database and citations from the retained articles were exported using the Publish or Perish software (Harzing, 2007).These articles were then imported into Excel and screened by title and abstract by the two reviewers, where three articles were retained.There were 30 articles included in the final selection.A PRISMA flow diagram shows the articles selected for inclusion and exclusion (Figure 1).

Data Extraction
Data were extracted into Microsoft Excel using a template designed by the reviewers (see Appendix A) that included the country in which the study was undertaken, study design, education setting, course, population, evaluation methods, limitation of evaluation component, and study design considerations.The data were extracted by the two reviewers, who independently identified emerging themes and then agreed upon the outcome.

Study Characteristics
A total of 30 articles published between 2011 and 2021 were included in the final review (see Appendices A and B).The number of participants within studies ranged from 16 (Adwan, 2016) to 3,752 (Hegerius et al., 2020).The studies were from 16 different countries, with the most common from the United States (9 studies), followed by Spain (3 studies) and Canada (3 studies).One multinational study, based in Sweden, used data from 137 countries (Hegerius et al., 2020).Study specifications are summarized in Appendix B.

Educational Level
The education level of participants varied, with the majority of studies from continuing professional development ( 15), followed by undergraduate education (9), then a combination of undergraduate, postgraduate, and continuing education (3), postgraduate education (2), and one combination of undergraduate students and teaching staff (Garrett et al., 2013).

Educational Institution
Course delivery was online, with more than half facilitated by universities (16 or 53%), followed by hospitals (6) and then a combination of universities and health centers (6), one research center (Hegerius et al., 2020), one combination research center and university (Tannenbaum & van Hoof, 2018) and one council (Willman et al., 2016).

Evaluation Methods
Methods for evaluating e-learning effectiveness were the focus of this review and are summarized in Table 3.This section describes the tools and methods that were used in the literature to assess learning effectiveness.The methods include student participation, student satisfaction, performance measures, and training models, among others.

Student Participation
A small proportion of studies measured participation using a variety of methods including class attendance (Lee et al., 2020), interaction with class discussion forums or completing class exercises (Adwan, 2016;Carrizosa et al., 2018;dos Reis et al., 2019;Salinas et al., 2017), obtaining learning platform analytic data (Reese et al., 2021;Wlodarczyk et al., 2017), and finally student evaluation about their participation experience (Figuccio, 2020;Liaw et al., 2016;Peterson et al., 2016).While participation data reflects student reactions, it does not evaluate learning effectiveness (Lima et al., 2019) and neglects to inform teachers of how students used online platforms (Backhouse et al., 2017).Carrizosa et al. (2018) further reported that while students were participating below staff expectations, the data could not provide reasons behind the participation rates.

Students' Reaction to Training Programs
Student reaction to the course is a subjective measure (such as students' self-reported satisfaction with the course) that is typically completed mid-way (formative) or towards the end of the course (summative).Less than half of the studies measured student reaction, or acceptance of pedagogy, using either of these methods (13; e.g., Adwan, 2016).Hegerius et al. ( 2020) measured students' acceptance of the information system or platform used to deliver the course.Evaluation tools that were delivered as formative (mid-way) studies were found to have higher response rates (e.g., 85%; Peterson et al., 2016), compared to summative evaluations that were completed towards the end of the course, which had lower response rates (e.g., 62.4%; Backhouse et al., 2017).Questionnaires were typically voluntary, and issues included low response rates, such as 13.2% (Hegerius et al., 2020), and some report high course dropout rates (dos Reis et al., 2019).Incentives included regular reminders using email (Hegerius et al., 2020) or by earning points that contributed to their final grades (Adwan, 2016).Studies attributed low response rates when participation in questionnaires was voluntary (e.g., Whitt et al., 2016).Peterson et al. (2016) identified the advantage of early evaluations, as poorly rated courses were able to respond quickly and make changes when questionnaires were conducted mid-way through the course.
Another limitation to these methods is the inability to explain the reasons behind course withdrawals (Reese, 2021) or to capture student satisfaction with the course over time (Tannenbaum & van Hoof, 2018).Evaluations that were conducted mid-way through the course were helpful in updating course delivery when courses were rated poorly.It was reported that qualitative, open-ended surveys provided varying degrees of information, from too little to too much information, and was the least useful aspect of the course evaluation (Le Marne et al., 2020).It was identified that when questionnaires are voluntary, there is a potential problem of selection bias between those who respond and those who do not (Hegerius et al., 2020).Poor response rates can also impact the reliability of the information from questionnaires (Garrett et al., 2013).Adwan (2016) used Google Docs to conduct the evaluations and reported issues with the useability of the information system by staff and security concerns with students.

Performance Measures: Assessment of Knowledge
Several studies used grades from assessment tasks and final exams on two or more occasions to assess student knowledge (7; e.g., Annan et al., 2020).However, these articles did not discuss the value of the metrics used in their assessments or exams.For example, participants had to pass an examination to complete the course by achieving 60 percent or more (Carrizosa et al., 2018), while another used a 100 percent pass rate (Willman et al., 2018).Types of assessments varied from multiple-choice questions (e.g., Schulz-Quach et al., 2018;Whitt et al., 2016) to oral assessments (e.g., Elzainy et al., 2020).Electronic portfolios were another method designed to assess clinical practice (Garrett et al., 2013).While studies did not report the limitations or deficiencies of their grading systems, using portfolios to measure clinical competency raised student concerns around privacy and confidentiality (Garrett et al., 2013).
Studies also varied in the format and delivery of questionnaires to measure changes in knowledge.Tannenbaum and van Hoof (2018) used a self-report questionnaire to test students' knowledge after the course.However, the authors identified that the questionnaire had not been previously validated ( 2018).Studies included those with externally validated questionnaires to measure students' performance (e.g., Kemper, 2017;Willman et al., 2018) and studies which had validated their own questionnaires (e.g., Heuberger et al., 2019;Schulz-Quach et al., 2018).Some questionnaires were specific to their subject content, such as stroke assessment (Gorchs-Molist et al., 2020), drug dispensing (dos Reis et al., 2019), and seizure management (Le Marne et al., 2016) and others measured self-directed learning readiness (Gagnon et al., 2015;Reviriego et al., 2014).Finally, the study by Kemper (2017) focused on questionnaires specific to measuring mindfulness (refer to Table 3) but did not measure the long-term impact on the participants.
Pre-test/post-test designs were also used to measure students' knowledge (e.g., Salinas et al., 2017) and performance (e.g., Backhouse et al., 2017) before and after training.However, some studies reported high dropout rates in the post-test phase (Annan et al., 2020;dos Reis et al., 2019;Gagnon et al., 2015;Reese, 2021;Reviriego et al., 2014), while others reported difficulties in accurately measuring the long-term impact of knowledge acquisition (Backhouse et al., 2017).

Performance Measures: Long-term or Follow-up
Various studies used follow-up questionnaires, ranging from one month to four years after course completion.Follow-up questionnaires implemented at one month had a 78 percent response rate (Wlodarczyk et al., 2017), while others implemented at eight months achieved 67 percent (Salinas et al., 2017).Gorchs-Molist et al. (2020) reported multiple follow-up periods, including after 1-2 years (71% response rate) and 3-4 years (91% response rate).Several studies identified the need to follow-up participants but could not undertake this process (Le Marne et al., 2016;Liaw et al., 2016;Simonsen et al., 2014;Uden-Holman et al., 2014).

Kirkpatrick's Training Evaluation Model
Various articles identified the need for learning evaluation methods, such as Kirkpatrick's Training Evaluation Model (Kirkpatrick & Kirkpatrick, 2006).This model has four levels of training outcomes: Level 1 (reaction) measures student responses about the quality of training; Level 2 (learning) quantifies learning using assessments and exams; Level 3 (behavior) measures the extent to which learning can be applied to the workplace; and Level 4 (results) measures how training has impacted organizational goals (Bates, 2004).Six studies reported on Kirkpatrick's evaluation model.Single measures ranged from self-reported student satisfaction at Level 1 (Hegerius et al., 2020) to course completion at Level 3 (Reese, 2021).In addition, hospital information system data were used to determine changes in compliance rates for Level 3 and clinical outcomes for Level 4 (Gorchs-Molist et al., 2020;Liaw et al., 2016).Level 3 was also measured using a 6-month post-test evaluation (Uden-Holman et al., 2014).However, one study reported improvements at Levels 1 and 2 but not at Levels 3 or 4 (dos Reis et al., 2019).Other studies identified in the review that did not use Kirkpatrick's model have inadvertently used elements from Kirkpatrick's model (e.g., reaction, learning, behavior, and outcomes).

Self-efficacy
Bandura's Conceptual Model of self-efficacy (Bandura, 1997), part of Level 2 of the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006), were delivered using pre-and post-test methods and included a Likert-type scale design with a validated nine-item, ten-point scale (Aper et al., 2012); a non-validated three-item, ten-point scale (Tannenbaum & van Hoof, 2018); and a one-item, five-point scale (Reese, 2021).Schulz-Quach et al. (2018) identified the need to measure self-efficacy to improve methodological quality.However, the voluntary nature of these self-report questionnaires has had response rates as low as 60 percent (Whitt et al., 2016).

Other Methods
Other themes that emerged from the review were focus groups, reflections, and costeffectiveness.Numerous studies within the review articles used focus groups.For example, focus groups using open-ended questions can examine learning experiences within the course (Garrett et al., 2013), while software such as FocusGroupIt can address themes using a SWOT (i.e., strengths, weaknesses, opportunities, threats) analysis (Elzainy et al., 2020).In contrast, Heuberger et al. (2019) conducted focus groups before their study and used the results to pilot and validate their course satisfaction survey.Furthermore, focus group transcripts and written reflections can be combined using thematic analysis to provide student feedback (Posser et al., 2021).However, the use of focus groups and reflections was impacted by the time requirements to train staff, and written reflections provided little additional information.Finally, formative feedback delivered to students in real-time has demonstrated success at commending high performers and encouraging low performers to improve their grades (Adwan, 2016).
Few studies reported the cost-effectiveness of the evaluation methods and their outcomes.Several studies reported the need for additional cost-effectiveness research (e.g., Kemper, 2017).Other studies commented on the cost savings of delivering training online instead of face-to-face (e.g., Martinez et al., 2019).While the cost-effectiveness of the evaluation methods was not always measured, some authors evaluated cost-effectiveness from self-perception scores, increase in knowledge and self-efficacy (e.g., Carrizosa et al., 2018).

Discussion
This rapid review identified research articles that used evaluation methods to measure the impact health-related online education has on student performance.When organizations are faced with emerging technology-driven changes and digital disruptors, as with the COVID-19 outbreak, there is a need for learning and development to support improvement in workplace performance.However, training and learning design and delivery methods need to be assessed to ensure education is efficient and relevant.This is particularly important in assessing the change from traditional face-to-face delivery to online models for teaching and learning.This review attempts to summarize research in this area and provide actionable and relevant evidence to help organizations plan learning interventions and measure the impact of student performance over time.

What were the Evaluation Methods and their Limitations?
From this review, it is apparent there is no single method that comprehensively measures the impact of learning interventions.Based on the level of data obtained, tools ranged from selfreport evaluations measuring participation rates and student's satisfaction to metric data such as course grades (and in health, very specific performance metrics related to clinical information system data e.g., changes to hospital length of stay) (Liaw et al., 2016).The distinct types of data collection were self-report questionnaires that used Likert-style scoring with or without openended questions; assessment tasks and exams that were scored on a grading system; and focus groups, reflections and portfolios that provided qualitative information and uncovered themes.There were also various applications of these tools, including before and after the course (e.g., using a pre-test/post-test approach), mid-way and at the end of the course (in a formative and summative approach), only at the end of the course (e.g., with final grades or course evaluations), and follow-ups after the course (e.g., six-month follow-up; Wlodarczyk et al., 2017;Carrizosa et al., 2018;Salinas et al., 2017;Gorchs-Molist et al., 2020).While available tools were used with a combination of students and teachers, the Kirkpatrick's evaluation method was the only complete framework described and used in some of the studies identified by the review (Hegerius et al., 2020;Reese, 2021;dos Reis et al., 2019;Gorchs-Molist., 2020;Liaw et al., 2016;den-Holman et al., 2014).
Several systematic reviews identified deficiencies with some of these evaluation methods, such as whether the use of non-validated measurement tools affected the validity of the outcomes or whether the training design affected student performance (Campbell et al., 2019;Moehead et al., 2020).Recommendations highlighted the need for validated frameworks to better synthesize learning effectiveness and a need for more robust study designs to enhance research methodologies (Barteit et al., 2020;Salter et al., 2014;Rouleau et al., 2019).

Which Tools are Appropriate for Measuring the Effectiveness of Online Education?
Given the limited evaluation methods and limitations with study designs (outlined in Table 3), caution is needed when assessing the utility of tools used.Nevertheless, the results suggest that using a framework, such as Kirkpatrick's, enables hierarchical measurement of learning effectiveness based on research-based findings.However, this involves using a collection of several types of evaluation tools, such as self-report questionnaires and comparing final grades, all with their own strengths and weaknesses.
Even though there is little evidence to suggest that e-learning has different outcomes to traditional face-to-face learning (Vaona et al., 2018), studies using Kirkpatrick's framework tended to limit the depth of learning effectiveness to Levels 1 and 2 and do not investigate how the course impacts performance in the workplace or over time.
While there are several explanations for limited evaluation, including the time and costs associated with measuring student performance, the following summary is a breakdown of tools and how they could be improved.

Participation Rates
Participation rates are the most straightforward metric used to measure student engagement.However, they provide little evidence about learning outcomes and fail to explain the reasons behind student dropouts.The use of self-report data could supplement information about dropouts and how these can be reduced in the future.

Self-Report Evaluation Tools
Self-reported course evaluation tools were found to be appropriate in obtaining students' experiences during the course and are best conducted early to allow time for the educators to adapt their teaching towards student preferences.
Furthermore, the use of teacher evaluation tools during the teaching enables more transparency within the teaching/class environment.Additionally, the use of validated course evaluation tools was found to be more robust than teacher-designed tools, providing more standardized results and allowing comparisons between classes (Barteit et al., 2020;Salter et al., 2014;Rouleau et al., 2019).Focus groups have also been used to validate evaluation questionnaires (Heuberger et al., 2019).

Knowledge Assessment
Assessments that are completed before, during and after the course (e.g., pre-test/post-test models and self-efficacy questionnaires) provided evidence of measuring learning effectiveness.Suggestions to enhance this method include the use of mandatory, or incentive-driven, delivery of these tools (e.g., grade incentives).This can help reduce the likelihood of nonresponse errors and self-selection bias.Validated subject-specific questionnaires are also recommended.
While mid-course assessments and final exams provide data about individual attainment, they fail to measure the long-term impact of the course (Backhouse et al., 2017;Kemper, 2017).One approach to measure this is the use of student follow-up questionnaires (Garrett et al., 2013), although they face the same responsiveness challenges of surveys.

Focus Groups and Written Reflections
Some studies used focus groups to obtain qualitative information about staff and student experiences from the course (Garrett et al., 2013).However, these were time-consuming and costly.Alternatives include written reflections which are less time-consuming to administer and provide a similar level of information (Prosser et al., 2021).Suggestions to improve written reflections were around incentives for completion (e.g., grade incentives), using validated methodologies (e.g., SWOT), providing real-time feedback (i.e., immediate versus delayed feedback), and capturing long-term data (e.g., post-course follow-up).

Recommendations
The majority of studies investigating the impact of online education programs did not consider a cost analysis or return on investment.This is important because the cost of one evaluation tool compared to another is an essential factor in the decision-making process around cost and benefit.Furthermore, an analysis of clinical significance was not performed in the majority of articles, as most outcomes were based on predefined goals such as achieving a pass mark (i.e., a 60+ percent score) or positive self-evaluation scores.Recommendations for future research are for more longitudinal studies that capture the effects of training after a six-month window and methods that can compare pre-and post-test outcomes.

Limitations
The rapid review process comes with several potential limitations, including the possibility that studies may have been missed (Tricco et al., 2017).This rapid review is not exhaustive, and as such, a search was not conducted on grey literature.The selected studies were from the academic research community and excluded evaluation methods within business and private organizations other than hospitals and universities.Articles included in the analysis were limited to English, full-text studies, which may bias studies from high-income English language countries such as the United States and the United Kingdom, thus missing studies written in different languages.Further, the data extraction has been performed on learning interventions from training programs and did not consider evaluation methods available from education platforms (e.g., Learning Management System).The focal point of this rapid review was on learning outcomes and articles that only reported on technology acceptance were excluded.Furthermore, the purpose of the rapid review was to summarize evidence rather than evaluate effects, the evaluation of reported quantitative data from the studies were not the primary focus.Lastly, a critical appraisal was not performed and inter-rater reliability selecting articles between the two reviewers was not measured, owing to the rapid nature of this review (Tricco et al., 2017).However, Table 3 includes a column that outlines the limitations mentioned within the identified studies.

Conclusion
This rapid review investigated the various methods and types of tools used to measure learning effectiveness for online education.The review included studies of online education within the discipline of healthcare and observed studies reporting positive effects of these training interventions.Education and development opportunities were identified as an important function that allows professionals to keep "up to date" with current practices.However, providing these opportunities within and across busy work schedules is complex.Although advancements in technology offer some alternatives on how professional development can be structured and organized, there is limited evidence to support what makes online teaching and learning effective.Many of the studies identified in this review suggest that professional development should provide support over a sustained period to achieve the most effective outcomes.However, due to constraints like funding, time, organizational structure, and policy, this is not often the case.Finally, while education evaluation tools and methods are popular in assessing the effectiveness of the training programs, the evidence suggests that using any evaluation method in isolation is insufficient.Suggestions were to adopt previously validated frameworks (not limited to the Kirkpatrick model) and appropriately implement them within the course curriculum.Developing a framework which identifies 'best practices' in the organization, development, delivery and evaluation of training can help support effective and sustainable education programs.Evaluations tools did not identify how students used the online platform and how they engaged with the activities.Long-term impact of knowledge acquisition was also not measured.
The differences between the two methods-online and face-to-face were discussed as limitations for comparing the findings.Evaluate student use of the teaching method.

Table 2
Search Method and Number of Results per Database

Table 3
Summary of Evaluation Methods and Their Limitations DeclarationsAuthors declare no known conflicts of interest.This research was supported by the Digital Health CRC.Digital Health CRC Limited is funded under the Commonwealth Government's Cooperative Research Centres Program.The authors would like to thank Dr Melanie Haines, Education Manager, Digital Health CRC.