The Role of Evaluation Methods in Health-Related E-learning: A Rapid Review




Assessment tools, evaluation method, learning impact measure, outcome measure, e-learning, online education, online learning, health education, healthcare


Training and development programs are increasingly delivered online with numerous studies reporting no differences in learning outcomes between online and traditional learning. However, there are no established standardized methods to evaluate the effectiveness of online learning. This review aims to map the state of research around health-related education to determine what e-learning evaluation methods are being used, the strengths or deficiencies of these methods, and which are appropriate for measuring the effectiveness of online education. Databases searched were PubMed, ProQuest, Education Resources Information Centre, Cumulative Index to Nursing and Allied Health Literature, Scopus, PsychInfo, and Medline. Studies were included if they were published between 2011 and 2021, reported health-related online education and included an evaluation component. Thirty studies were obtained from numerous countries with varied methodologies and designs. Participants ranged from undergraduate students to medical professionals. Evaluation methods included student participation, students’ reaction to the training program, self-efficacy, knowledge assessment, long-term performance, and the Kirkpatrick Evaluation Framework. The review identified that course evaluations, such as measuring student satisfaction scores alone, are insufficient when used to quantify learning effectiveness for online education. This was particularly important as studies are reporting these single metrics as positive effects of training interventions without justification. Suggestions within the reviewed papers were to adopt and implement an appropriate validated method within the course curriculum to evaluate learning outcomes.

Author Biographies

Urooj R Khan, La Trobe University

Senior Lecturer (Digital Health), Department of Public Health, La Trobe University, Melbourne, Australia

James H Boyd, La Trobe University

Professor Public Health (Digital Health), Department of Public Health, La Trobe University, Melbourne, Australia


Adwan, J. (2016). Dynamic online peer evaluations to improve group assignments in nursing e-learning environment. Nurse Education Today, 41, 67-72.

Ahmed, H., Allaf, M., & Elghazaly, H. (2020). COVID-19 and medical education. The Lancet Infectious Diseases, 20(7), 777-778.

Annan, R. A., Aduku, L. N. E., Kyei-Boateng, S., Yeun, H. M., Pickup, T., Pulman, A., Monroy-Valle, M., Ashworth-Hill, A., Jackson, A. A., & Choi, S. (2020). Implementing effective e-Learning for scaling up global capacity building: Findings from the malnutrition elearning course evaluation in Ghana. Global Health Action, 13(1).

Aper, L., Reniers, J., Koole, S., Valcke, M., & Derese, A. (2012). Impact of three alternative consultation training formats on self-efficacy and consultation skills of medical students. Medical Teacher, 34(7), e500-507.

Australian Health Practitioner Regulation Agency. (2019, December 3). Continuing professional development.

Backhouse, M., Fitzpatrick, M., Hutchinson, J., Thandi, C. S., & Keenan, I. D. (2017). Improvements in anatomy knowledge when utilizing a novel cyclical "Observe-Reflect-Draw-Edit-Repeat" learning process. Anatomical Sciences Education, 10(1), 7-22.

Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W.H. Freeman.

Barteit, S., Guzek, D., Jahn, A., Bärnighausen, T., Jorge, M. M., & Neuhann, F. (2020). Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Computers & Education, 145, 103726.

Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347.

Campbell, K., Taylor, V., & Douglas, S. (2019). Effectiveness of online cancer education for nurses and allied health professionals: A systematic review using Kirkpatrick evaluation framework. Journal of Cancer Education, 34(2), 339-356.

Carrizosa, J., Braga, P., Albuquerque, M., Bogacz, A., Burneo, J., Coan, A. C., Contreras, G., Guilhoto, L., Izquierdo, A., Ladino, L., Lin, K., Manreza, M. L., Morales, L., Rios, L., Solarte, R., Valente, K., Venegas, V., Uscategui-Daccarett, A., & Yacubian, E. M. (2018). Epilepsy for primary health care: A cost-effective Latin American E-learning initiative. Epileptic Disorders, 20(5), 386-395.

Dhawan, S. (2020). Online learning: A panacea in the time of COVID-19 crisis. Journal of Educational Technology Systems, 49(1), 5-22.

dos Reis, T. M., Baldoni, A. d. O., Campos, A. M., Girotto, E., Guidoni, C. M., Obreli-Neto, P. R., & Pereira, L. R. L. (2019). A distance-learning course to improve drug-dispensing behaviors among Brazilian community pharmacists. American Journal of Pharmaceutical Education, 83(8), 1703-1711.

Elzainy, A., El Sadik, A., & Al Abdulmonem, W. (2020). Experience of e-learning and online assessment during the COVID-19 pandemic at the College of Medicine, Qassim University. Journal of Taibah University Medical Sciences, 15(6), 456-462.

Figuccio, M. J. (2020). Examining the efficacy of e-service-learning. Frontiers in Education, 5.

Gagnon, J., Gagnon, M. P., Buteau, R. A., Azizah, G. M., JettÉ, S., Lampron, A., Simonyan, D., Asua, J., & Reviriego, E. (2015). Adaptation and evaluation of online self-learning modules to teach critical appraisal and evidence-based practice in nursing: An international collaboration. CIN: Computers, Informatics, Nursing, 33(7), 285-294.

Garrett, B. M., MacPhee, M., & Jackson, C. (2013). Evaluation of an eportfolio for the assessment of clinical competence in a baccalaureate nursing program. Nurse Education Today, 33(10), 1207-1213.

Gorchs-Molist, M., Sola-Munoz, S., Enjo-Perez, I., Querol-Gil, M., Carrera-Giraldo, D., Nicolas-Arfelis, J. M., Jimenez-Fabrega, F. X., & Perez de la Ossa, N. (2020). An online training intervention on prehospital stroke codes in Catalonia to improve the knowledge, pre-notification compliance and time performance of emergency medical services professionals. International Journal of Environmental Research and Public Health, 17(17), 6183.

Harzing, A. W. (2007). Publish or Perish. (Version 7)

Hegerius, A., Caduff-Janosa, P., Savage, R., & Ellenius, J. (2020). E-Learning in pharmacovigilance: An evaluation of microlearning-based modules developed by Uppsala Monitoring Centre. Drug Safety, 43(11), 1171-1180.

Heuberger, R., & Clark, W. A. (2019). Synchronous delivery of online graduate education in clinical nutrition: An inquiry into student perceptions and preferences. Journal of Allied Health, 48(1), 61-66.

Huang, X., Lin, J., & Demner-Fushman, D. (2006). Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annual Symposium proceedings. 359-363.

Hughes, A. M., Gregory, M. E., Joseph, D. L., Sonesh, S. C., Marlow, S. L., Lacerenza, C. N., Benishek, L. E., King, H. B., & Salas, E. (2016). Saving lives: A meta-analysis of team training in healthcare. Journal of Applied Psychology, 101(9), 1266-1304.

Jayakumar, N., Brunckhorst, O., Dasgupta, P., Khan, M. S., & Ahmed, K. (2015). e-Learning in surgical education: A systematic review. Journal of Surgical Education, 72(6), 1145-1157.

Kemper, K. J. (2017). Brief online mindfulness training: Immediate impact. Journal of Evidence-Based Complementary and Alternative Medicine, 22(1), 75-80.

Khan, R. A. G., Khan, F. A., & Khan, M. A. (2011). Impact of training and development on organizational performance. Global Journal of Management and Business Research, 11(7).

Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels (Vol. 1). Berrett-Koehler Publishers.

Kirkpatrick, J. D., & Kirkpatrick, D. (2006). Evaluating training programs: The four levels. Berrett-Koehler Publishers.

Le Marne, F. A., McGinness, H., Slade, R., Cardamone, M., Balbir Singh, S., Connolly, A. M., & Bye, A. M. (2016). Evaluation of an e-learning resource on approach to the first unprovoked seizure. Journal of Paediatrics and Child Health, 52(9), 896-900.

Lee, S. J., Park, J., Lee, Y. J., Lee, S., Kim, W. H., & Yoon, H. B. (2020). The feasibility and satisfaction of an online global health education course at a single medical school: a retrospective study. Korean Journal of Medical Education, 32(4), 307-315.

Liaw, S. Y., Wong, L. F., Lim, E. Y., Ang, S. B., Mujumdar, S., Ho, J. T., Mordiffi, S. Z., & Ang, E. N. (2016). Effectiveness of a web-based simulation in improving nurses' workplace practice with deteriorating ward patients: A pre- and post-intervention study. Journal of Medical Internet Research, 18(2), e37.

Lima, M. S., Tonial, F. G., Basei, E., Brew, M. C., Grossmann, E., Haddad, A. E., Rivaldo, E. G., Vargas, I. A., & Bavaresco, C. S. (2019). Effectiveness of the distance learning strategy applied to orthodontics education: A systematic literature review. Telemedicine Journal and e-Health, 25(12), 1134-1143.

Martinez, C., Castellano, Y., Andres, A., Fu, M., Feliu, A., Anton, L., Ballbe, M., Fernandez, P., Cabrera, S., Riccobene, A., Gavilan, E., Baena, A., Margalef, M., Tigova, O., Quiros, N., Guillen, O., Company, A., & Fernandez, E. (2019). Impact of an online training program in smoking cessation interventions in hospitals. Journal of Nursing Scholarship, 51(4), 449-458.

Moehead, A., DeSouza, K., Walsh, K., & Pit, S. W. (2020). A web-based dementia education program and its application to an Australian web-based dementia care competency and training network: Integrative systematic review. Journal of Medical Internet Research, 22(1), e16808.

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71.

Peterson, J. L. (2016). Formative evaluations in online classes. Journal of Educators Online, 13(1).

Pokhrel, S., & Chhetri, R. (2021). A literature review on impact of COVID-19 pandemic on teaching and learning. Higher Education for the Future, 8(1), 133-141.

Prosser, M., Stephenson, T., Mathur, J., Enayati, H., Kadie, A., Abdi, M. M., Handuleh, J. I. M., & Keynejad, R. C. (2021). Reflective practice and transcultural psychiatry peer e-learning between Somaliland and the UK: A qualitative evaluation. BMC Medical Education, 21(1), 58.

Reese, D., Dolansky, M. A., Moore, S. M., Bolden, H., & Singh, M. K. (2021). Quality improvement education innovation: evaluation of Coursera MOOC ‘Take the Lead on Healthcare Quality Improvement’. Journal of Research in Nursing, 26(1-2), 62-78.

Reviriego, E., Cidoncha, M. A., Asua, J., Gagnon, M. P., Mateos, M., Garate, L., de Lorenzo, E., & Gonzalez, R. M. (2014). Online training course on critical appraisal for nurses: Adaptation and assessment. BMC Medical Education, 14, 136.

Rouleau, G., Gagnon, M. P., Cote, J., Payne-Gagnon, J., Hudson, E., Dubois, C. A., & Bouix-Picasso, J. (2019). Effects of e-learning in a continuing education context on nursing care: Systematic review of systematic qualitative, quantitative, and mixed-studies reviews. Journal of Medical Internet Research, 21(10), e15118.

Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2), 74-101.

Salinas, J., Munoz, C., Albagli, A., Araya, G., & Vio, F. (2017). The contribution of distance education to health promotion in Chile. Health Promotion International, 32(5), 913-921.

Salter, S. M., Karia, A., Sanfilippo, F. M., & Clifford, R. M. (2014). Effectiveness of e-learning in pharmacy education. American Journal of Pharmaceutical Education, 78(4), 83.

Schulz-Quach, C., Wenzel-Meyburg, U., & Fetz, K. (2018). Can elearning be used to teach palliative care? - medical students' acceptance, knowledge, and self-estimation of competence in palliative care after elearning. BMC Medical Education, 18(1), 82.

Simonsen, B. O., Daehlin, G. K., Johansson, I., & Farup, P. G. (2014). Improvement of drug dose calculations by classroom teaching or e-learning: a randomised controlled trial in nurses. BMJ Open, 4(10), e006025.

Tannenbaum, C., & van Hoof, K. (2018). Effectiveness of online learning on health researcher capacity to appropriately integrate sex, gender, or both in grant proposals. Biology of Sex Differences, 9(1), 39.

Tricco, A. C., Langlois, E. V., & Straus, S. E. (2017). Rapid reviews to strengthen health policy and systems: A practical guide. World Health Organization. Geneva, Switzerland.

Uden-Holman, T., Bedet, J., Walkner, L., & Abd-Hamid, N. H. (2014). Adaptive scenarios: A training model for today's public health workforce. Journal of Public Health Management and Practice, 20 Suppl 5, S44-48.

United Nations. (2020). Policy brief: Education during COVID-19 and beyond.

Vaona, A., Banzi, R., Kwag, K. H., Rigon, G., Cereda, D., Pecoraro, V., Tramacere, I., & Moja, L. (2018). E-learning for health professionals. Cochrane Database of Systematic Reviews, 2018(8).

Veritas Health Innovation. (2019). Covidence systematic review software.

Whitt, K. J., Macri, C., O'Brien, T. J., & Wright, S. (2016). Improving nurse practitioners' competence with genetics: Effectiveness of an online course. Journal of the American Association of Nurse Practitioners, 28(3), 151-159.

Willman, B., Grankvist, K., & Bolenius, K. (2018). Evaluation of the clinical implementation of a large-scale online e-learning program on venous blood specimen collection guideline practices. Clinical Chemistry and Laboratory Medicine, 56(11), 1870-1877.

Wlodarczyk, D., Chylinska, J., Lazarewicz, M., Rzadkiewicz, M., Jaworski, M., Adamus, M., Haugan, G., Lillefjell, M., & Espnes, G. A. (2017). Enhancing doctors' competencies in communication with and activation of older patients: The Promoting Active Aging (PRACTA) computer-based intervention study. Journal of Medical Internet Research, 19(2), e45.

Zafar, S., Safdar, S., & Zafar, A. N. (2014). Evaluation of use of e-Learning in undergraduate radiology education: A review. European Journal of Radiology, 83(12), 2277-2287.






Section II