The Role of Evaluation Methods in Health-Related E-learning: A Rapid Review

Jason D Stemp, Debannita Ghosh, Urooj R Khan, James H Boyd

Abstract


Training and development programs are increasingly delivered online with numerous studies reporting no differences in learning outcomes between online and traditional learning. However, there are no established standardized methods to evaluate the effectiveness of online learning. This review aims to map the state of research around health-related education to determine what e-learning evaluation methods are being used, the strengths or deficiencies of these methods, and which are appropriate for measuring the effectiveness of online education. Databases searched were PubMed, ProQuest, Education Resources Information Centre, Cumulative Index to Nursing and Allied Health Literature, Scopus, PsychInfo, and Medline. Studies were included if they were published between 2011 and 2021, reported health-related online education and included an evaluation component. Thirty studies were obtained from numerous countries with varied methodologies and designs. Participants ranged from undergraduate students to medical professionals. Evaluation methods included student participation, students’ reaction to the training program, self-efficacy, knowledge assessment, long-term performance, and the Kirkpatrick Evaluation Framework. The review identified that course evaluations, such as measuring student satisfaction scores alone, are insufficient when used to quantify learning effectiveness for online education. This was particularly important as studies are reporting these single metrics as positive effects of training interventions without justification. Suggestions within the reviewed papers were to adopt and implement an appropriate validated method within the course curriculum to evaluate learning outcomes.


Keywords


Assessment tools; evaluation method; learning impact measure; outcome measure; e-learning; online education; online learning; health education; healthcare

Full Text:

PDF

References


Adwan, J. (2016). Dynamic online peer evaluations to improve group assignments in nursing e-learning environment. Nurse Education Today, 41, 67-72. https://doi.org/10.1016/j.nedt.2016.03.028

Ahmed, H., Allaf, M., & Elghazaly, H. (2020). COVID-19 and medical education. The Lancet Infectious Diseases, 20(7), 777-778. https://doi.org/10.1016/S1473-3099(20)30226-7

Annan, R. A., Aduku, L. N. E., Kyei-Boateng, S., Yeun, H. M., Pickup, T., Pulman, A., Monroy-Valle, M., Ashworth-Hill, A., Jackson, A. A., & Choi, S. (2020). Implementing effective e-Learning for scaling up global capacity building: Findings from the malnutrition elearning course evaluation in Ghana. Global Health Action, 13(1). https://doi.org/10.1080/16549716.2020.1831794

Aper, L., Reniers, J., Koole, S., Valcke, M., & Derese, A. (2012). Impact of three alternative consultation training formats on self-efficacy and consultation skills of medical students. Medical Teacher, 34(7), e500-507. https://doi.org/10.3109/0142159X.2012.668627

Australian Health Practitioner Regulation Agency. (2019, December 3). Continuing professional development. https://www.ahpra.gov.au/Registration/Registration-Standards/CPD.aspx

Backhouse, M., Fitzpatrick, M., Hutchinson, J., Thandi, C. S., & Keenan, I. D. (2017). Improvements in anatomy knowledge when utilizing a novel cyclical "Observe-Reflect-Draw-Edit-Repeat" learning process. Anatomical Sciences Education, 10(1), 7-22. https://doi.org/10.1002/ase.1616

Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W.H. Freeman.

Barteit, S., Guzek, D., Jahn, A., Bärnighausen, T., Jorge, M. M., & Neuhann, F. (2020). Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Computers & Education, 145, 103726. https://doi.org/10.1016/j.compedu.2019.103726

Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. https://doi.org/10.1016/j.evalprogplan.2004.04.011

Campbell, K., Taylor, V., & Douglas, S. (2019). Effectiveness of online cancer education for nurses and allied health professionals: A systematic review using Kirkpatrick evaluation framework. Journal of Cancer Education, 34(2), 339-356. https://doi.org/10.1007/s13187-017-1308-2

Carrizosa, J., Braga, P., Albuquerque, M., Bogacz, A., Burneo, J., Coan, A. C., Contreras, G., Guilhoto, L., Izquierdo, A., Ladino, L., Lin, K., Manreza, M. L., Morales, L., Rios, L., Solarte, R., Valente, K., Venegas, V., Uscategui-Daccarett, A., & Yacubian, E. M. (2018). Epilepsy for primary health care: A cost-effective Latin American E-learning initiative. Epileptic Disorders, 20(5), 386-395. https://doi.org/10.1684/epd.2018.0997

Dhawan, S. (2020). Online learning: A panacea in the time of COVID-19 crisis. Journal of Educational Technology Systems, 49(1), 5-22. https://doi.org/10.1177/0047239520934018

dos Reis, T. M., Baldoni, A. d. O., Campos, A. M., Girotto, E., Guidoni, C. M., Obreli-Neto, P. R., & Pereira, L. R. L. (2019). A distance-learning course to improve drug-dispensing behaviors among Brazilian community pharmacists. American Journal of Pharmaceutical Education, 83(8), 1703-1711. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6900818/

Elzainy, A., El Sadik, A., & Al Abdulmonem, W. (2020). Experience of e-learning and online assessment during the COVID-19 pandemic at the College of Medicine, Qassim University. Journal of Taibah University Medical Sciences, 15(6), 456-462. https://doi.org/10.1016/j.jtumed.2020.09.005

Figuccio, M. J. (2020). Examining the efficacy of e-service-learning. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.606451

Gagnon, J., Gagnon, M. P., Buteau, R. A., Azizah, G. M., JettÉ, S., Lampron, A., Simonyan, D., Asua, J., & Reviriego, E. (2015). Adaptation and evaluation of online self-learning modules to teach critical appraisal and evidence-based practice in nursing: An international collaboration. CIN: Computers, Informatics, Nursing, 33(7), 285-294. https://doi.org/10.1097/CIN.0000000000000156

Garrett, B. M., MacPhee, M., & Jackson, C. (2013). Evaluation of an eportfolio for the assessment of clinical competence in a baccalaureate nursing program. Nurse Education Today, 33(10), 1207-1213. https://doi.org/10.1016/j.nedt.2012.06.015

Gorchs-Molist, M., Sola-Munoz, S., Enjo-Perez, I., Querol-Gil, M., Carrera-Giraldo, D., Nicolas-Arfelis, J. M., Jimenez-Fabrega, F. X., & Perez de la Ossa, N. (2020). An online training intervention on prehospital stroke codes in Catalonia to improve the knowledge, pre-notification compliance and time performance of emergency medical services professionals. International Journal of Environmental Research and Public Health, 17(17), 6183. https://doi.org/10.3390/ijerph17176183

Harzing, A. W. (2007). Publish or Perish. (Version 7) https://harzing.com/resources/publish-or-perish

Hegerius, A., Caduff-Janosa, P., Savage, R., & Ellenius, J. (2020). E-Learning in pharmacovigilance: An evaluation of microlearning-based modules developed by Uppsala Monitoring Centre. Drug Safety, 43(11), 1171-1180. https://doi.org/10.1007/s40264-020-00981-w

Heuberger, R., & Clark, W. A. (2019). Synchronous delivery of online graduate education in clinical nutrition: An inquiry into student perceptions and preferences. Journal of Allied Health, 48(1), 61-66. https://pubmed.ncbi.nlm.nih.gov/30826832

Huang, X., Lin, J., & Demner-Fushman, D. (2006). Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annual Symposium proceedings. 359-363. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1839740/

Hughes, A. M., Gregory, M. E., Joseph, D. L., Sonesh, S. C., Marlow, S. L., Lacerenza, C. N., Benishek, L. E., King, H. B., & Salas, E. (2016). Saving lives: A meta-analysis of team training in healthcare. Journal of Applied Psychology, 101(9), 1266-1304. https://doi.org/10.1037/apl0000120

Jayakumar, N., Brunckhorst, O., Dasgupta, P., Khan, M. S., & Ahmed, K. (2015). e-Learning in surgical education: A systematic review. Journal of Surgical Education, 72(6), 1145-1157. https://doi.org/10.1016/j.jsurg.2015.05.008

Kemper, K. J. (2017). Brief online mindfulness training: Immediate impact. Journal of Evidence-Based Complementary and Alternative Medicine, 22(1), 75-80. https://doi.org/10.1177/2156587216639199

Khan, R. A. G., Khan, F. A., & Khan, M. A. (2011). Impact of training and development on organizational performance. Global Journal of Management and Business Research, 11(7). https://swsu.ru/sbornik-statey/pdf/8-Impact-of-Training-and-Development-on-Organizational-Performance.pdf

Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels (Vol. 1). Berrett-Koehler Publishers.

Kirkpatrick, J. D., & Kirkpatrick, D. (2006). Evaluating training programs: The four levels. Berrett-Koehler Publishers. https://search.alexanderstreet.com/view/work/bibliographic_entity%7Cbibliographic_details%7C2341903

Le Marne, F. A., McGinness, H., Slade, R., Cardamone, M., Balbir Singh, S., Connolly, A. M., & Bye, A. M. (2016). Evaluation of an e-learning resource on approach to the first unprovoked seizure. Journal of Paediatrics and Child Health, 52(9), 896-900. https://doi.org/10.1111/jpc.13277

Lee, S. J., Park, J., Lee, Y. J., Lee, S., Kim, W. H., & Yoon, H. B. (2020). The feasibility and satisfaction of an online global health education course at a single medical school: a retrospective study. Korean Journal of Medical Education, 32(4), 307-315. https://doi.org/10.3946/kjme.2020.178

Liaw, S. Y., Wong, L. F., Lim, E. Y., Ang, S. B., Mujumdar, S., Ho, J. T., Mordiffi, S. Z., & Ang, E. N. (2016). Effectiveness of a web-based simulation in improving nurses' workplace practice with deteriorating ward patients: A pre- and post-intervention study. Journal of Medical Internet Research, 18(2), e37. https://doi.org/10.2196/jmir.5294

Lima, M. S., Tonial, F. G., Basei, E., Brew, M. C., Grossmann, E., Haddad, A. E., Rivaldo, E. G., Vargas, I. A., & Bavaresco, C. S. (2019). Effectiveness of the distance learning strategy applied to orthodontics education: A systematic literature review. Telemedicine Journal and e-Health, 25(12), 1134-1143. https://doi.org/10.1089/tmj.2018.0286

Martinez, C., Castellano, Y., Andres, A., Fu, M., Feliu, A., Anton, L., Ballbe, M., Fernandez, P., Cabrera, S., Riccobene, A., Gavilan, E., Baena, A., Margalef, M., Tigova, O., Quiros, N., Guillen, O., Company, A., & Fernandez, E. (2019). Impact of an online training program in smoking cessation interventions in hospitals. Journal of Nursing Scholarship, 51(4), 449-458. https://doi.org/10.1111/jnu.12469

Moehead, A., DeSouza, K., Walsh, K., & Pit, S. W. (2020). A web-based dementia education program and its application to an Australian web-based dementia care competency and training network: Integrative systematic review. Journal of Medical Internet Research, 22(1), e16808. https://doi.org/10.2196/16808

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71

Peterson, J. L. (2016). Formative evaluations in online classes. Journal of Educators Online, 13(1). https://doi.org/10.9743/JEO.2016.1.8

Pokhrel, S., & Chhetri, R. (2021). A literature review on impact of COVID-19 pandemic on teaching and learning. Higher Education for the Future, 8(1), 133-141. https://doi.org/10.1177/2347631120983481

Prosser, M., Stephenson, T., Mathur, J., Enayati, H., Kadie, A., Abdi, M. M., Handuleh, J. I. M., & Keynejad, R. C. (2021). Reflective practice and transcultural psychiatry peer e-learning between Somaliland and the UK: A qualitative evaluation. BMC Medical Education, 21(1), 58. https://doi.org/10.1186/s12909-020-02465-y

Reese, D., Dolansky, M. A., Moore, S. M., Bolden, H., & Singh, M. K. (2021). Quality improvement education innovation: evaluation of Coursera MOOC ‘Take the Lead on Healthcare Quality Improvement’. Journal of Research in Nursing, 26(1-2), 62-78. https://doi.org/10.1177/1744987120982644

Reviriego, E., Cidoncha, M. A., Asua, J., Gagnon, M. P., Mateos, M., Garate, L., de Lorenzo, E., & Gonzalez, R. M. (2014). Online training course on critical appraisal for nurses: Adaptation and assessment. BMC Medical Education, 14, 136. https://doi.org/10.1186/1472-6920-14-136

Rouleau, G., Gagnon, M. P., Cote, J., Payne-Gagnon, J., Hudson, E., Dubois, C. A., & Bouix-Picasso, J. (2019). Effects of e-learning in a continuing education context on nursing care: Systematic review of systematic qualitative, quantitative, and mixed-studies reviews. Journal of Medical Internet Research, 21(10), e15118. https://doi.org/10.2196/15118

Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2), 74-101. https://doi.org/10.1177/1529100612436661

Salinas, J., Munoz, C., Albagli, A., Araya, G., & Vio, F. (2017). The contribution of distance education to health promotion in Chile. Health Promotion International, 32(5), 913-921. https://doi.org/10.1093/heapro/daw023

Salter, S. M., Karia, A., Sanfilippo, F. M., & Clifford, R. M. (2014). Effectiveness of e-learning in pharmacy education. American Journal of Pharmaceutical Education, 78(4), 83. https://doi.org/10.5688/ajpe78483

Schulz-Quach, C., Wenzel-Meyburg, U., & Fetz, K. (2018). Can elearning be used to teach palliative care? - medical students' acceptance, knowledge, and self-estimation of competence in palliative care after elearning. BMC Medical Education, 18(1), 82. https://doi.org/10.1186/s12909-018-1186-2

Simonsen, B. O., Daehlin, G. K., Johansson, I., & Farup, P. G. (2014). Improvement of drug dose calculations by classroom teaching or e-learning: a randomised controlled trial in nurses. BMJ Open, 4(10), e006025. https://doi.org/10.1136/bmjopen-2014-006025

Tannenbaum, C., & van Hoof, K. (2018). Effectiveness of online learning on health researcher capacity to appropriately integrate sex, gender, or both in grant proposals. Biology of Sex Differences, 9(1), 39. https://doi.org/10.1186/s13293-018-0197-3

Tricco, A. C., Langlois, E. V., & Straus, S. E. (2017). Rapid reviews to strengthen health policy and systems: A practical guide. World Health Organization. Geneva, Switzerland. https://www.who.int/alliance-hpsr/resources/publications/rapid-review-guide/en/

Uden-Holman, T., Bedet, J., Walkner, L., & Abd-Hamid, N. H. (2014). Adaptive scenarios: A training model for today's public health workforce. Journal of Public Health Management and Practice, 20 Suppl 5, S44-48. https://doi.org/10.1097/PHH.0000000000000067

United Nations. (2020). Policy brief: Education during COVID-19 and beyond. https://www.un.org/development/desa/dspd/wpcontent/uploads/sites/22/2020/08/sg_policy_brief_covid-19_and_education_august_2020.pdf

Vaona, A., Banzi, R., Kwag, K. H., Rigon, G., Cereda, D., Pecoraro, V., Tramacere, I., & Moja, L. (2018). E-learning for health professionals. Cochrane Database of Systematic Reviews, 2018(8). https://doi.org/10.1002/14651858.CD011736.pub2

Veritas Health Innovation. (2019). Covidence systematic review software. https://www.covidence.org

Whitt, K. J., Macri, C., O'Brien, T. J., & Wright, S. (2016). Improving nurse practitioners' competence with genetics: Effectiveness of an online course. Journal of the American Association of Nurse Practitioners, 28(3), 151-159. https://doi.org/10.1002/2327-6924.12282

Willman, B., Grankvist, K., & Bolenius, K. (2018). Evaluation of the clinical implementation of a large-scale online e-learning program on venous blood specimen collection guideline practices. Clinical Chemistry and Laboratory Medicine, 56(11), 1870-1877. https://doi.org/10.1515/cclm-2018-0051

Wlodarczyk, D., Chylinska, J., Lazarewicz, M., Rzadkiewicz, M., Jaworski, M., Adamus, M., Haugan, G., Lillefjell, M., & Espnes, G. A. (2017). Enhancing doctors' competencies in communication with and activation of older patients: The Promoting Active Aging (PRACTA) computer-based intervention study. Journal of Medical Internet Research, 19(2), e45. https://doi.org/10.2196/jmir.6948

Zafar, S., Safdar, S., & Zafar, A. N. (2014). Evaluation of use of e-Learning in undergraduate radiology education: A review. European Journal of Radiology, 83(12), 2277-2287. https://doi.org/10.1016/j.ejrad.2014.08.017




DOI: http://dx.doi.org/10.24059/olj.v26i4.3115



Copyright (c) 2022 Jason D Stemp, Debannita Ghosh, Urooj R Khan, James H Boyd

License URL: https://creativecommons.org/licenses/by/4.0/