The Role of Evaluation Methods in Health-Related E-learning: A Rapid Review
DOI:
https://doi.org/10.24059/olj.v26i4.3115Keywords:
Assessment tools, evaluation method, learning impact measure, outcome measure, e-learning, online education, online learning, health education, healthcareAbstract
Training and development programs are increasingly delivered online with numerous studies reporting no differences in learning outcomes between online and traditional learning. However, there are no established standardized methods to evaluate the effectiveness of online learning. This review aims to map the state of research around health-related education to determine what e-learning evaluation methods are being used, the strengths or deficiencies of these methods, and which are appropriate for measuring the effectiveness of online education. Databases searched were PubMed, ProQuest, Education Resources Information Centre, Cumulative Index to Nursing and Allied Health Literature, Scopus, PsychInfo, and Medline. Studies were included if they were published between 2011 and 2021, reported health-related online education and included an evaluation component. Thirty studies were obtained from numerous countries with varied methodologies and designs. Participants ranged from undergraduate students to medical professionals. Evaluation methods included student participation, students’ reaction to the training program, self-efficacy, knowledge assessment, long-term performance, and the Kirkpatrick Evaluation Framework. The review identified that course evaluations, such as measuring student satisfaction scores alone, are insufficient when used to quantify learning effectiveness for online education. This was particularly important as studies are reporting these single metrics as positive effects of training interventions without justification. Suggestions within the reviewed papers were to adopt and implement an appropriate validated method within the course curriculum to evaluate learning outcomes.
References
Adwan, J. (2016). Dynamic online peer evaluations to improve group assignments in nursing e-learning environment. Nurse Education Today, 41, 67-72. https://doi.org/10.1016/j.nedt.2016.03.028
Ahmed, H., Allaf, M., & Elghazaly, H. (2020). COVID-19 and medical education. The Lancet Infectious Diseases, 20(7), 777-778. https://doi.org/10.1016/S1473-3099(20)30226-7
Annan, R. A., Aduku, L. N. E., Kyei-Boateng, S., Yeun, H. M., Pickup, T., Pulman, A., Monroy-Valle, M., Ashworth-Hill, A., Jackson, A. A., & Choi, S. (2020). Implementing effective e-Learning for scaling up global capacity building: Findings from the malnutrition elearning course evaluation in Ghana. Global Health Action, 13(1). https://doi.org/10.1080/16549716.2020.1831794
Aper, L., Reniers, J., Koole, S., Valcke, M., & Derese, A. (2012). Impact of three alternative consultation training formats on self-efficacy and consultation skills of medical students. Medical Teacher, 34(7), e500-507. https://doi.org/10.3109/0142159X.2012.668627
Australian Health Practitioner Regulation Agency. (2019, December 3). Continuing professional development. https://www.ahpra.gov.au/Registration/Registration-Standards/CPD.aspx
Backhouse, M., Fitzpatrick, M., Hutchinson, J., Thandi, C. S., & Keenan, I. D. (2017). Improvements in anatomy knowledge when utilizing a novel cyclical "Observe-Reflect-Draw-Edit-Repeat" learning process. Anatomical Sciences Education, 10(1), 7-22. https://doi.org/10.1002/ase.1616
Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W.H. Freeman.
Barteit, S., Guzek, D., Jahn, A., Bärnighausen, T., Jorge, M. M., & Neuhann, F. (2020). Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Computers & Education, 145, 103726. https://doi.org/10.1016/j.compedu.2019.103726
Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. https://doi.org/10.1016/j.evalprogplan.2004.04.011
Campbell, K., Taylor, V., & Douglas, S. (2019). Effectiveness of online cancer education for nurses and allied health professionals: A systematic review using Kirkpatrick evaluation framework. Journal of Cancer Education, 34(2), 339-356. https://doi.org/10.1007/s13187-017-1308-2
Carrizosa, J., Braga, P., Albuquerque, M., Bogacz, A., Burneo, J., Coan, A. C., Contreras, G., Guilhoto, L., Izquierdo, A., Ladino, L., Lin, K., Manreza, M. L., Morales, L., Rios, L., Solarte, R., Valente, K., Venegas, V., Uscategui-Daccarett, A., & Yacubian, E. M. (2018). Epilepsy for primary health care: A cost-effective Latin American E-learning initiative. Epileptic Disorders, 20(5), 386-395. https://doi.org/10.1684/epd.2018.0997
Dhawan, S. (2020). Online learning: A panacea in the time of COVID-19 crisis. Journal of Educational Technology Systems, 49(1), 5-22. https://doi.org/10.1177/0047239520934018
dos Reis, T. M., Baldoni, A. d. O., Campos, A. M., Girotto, E., Guidoni, C. M., Obreli-Neto, P. R., & Pereira, L. R. L. (2019). A distance-learning course to improve drug-dispensing behaviors among Brazilian community pharmacists. American Journal of Pharmaceutical Education, 83(8), 1703-1711. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6900818/
Elzainy, A., El Sadik, A., & Al Abdulmonem, W. (2020). Experience of e-learning and online assessment during the COVID-19 pandemic at the College of Medicine, Qassim University. Journal of Taibah University Medical Sciences, 15(6), 456-462. https://doi.org/10.1016/j.jtumed.2020.09.005
Figuccio, M. J. (2020). Examining the efficacy of e-service-learning. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.606451
Gagnon, J., Gagnon, M. P., Buteau, R. A., Azizah, G. M., JettÉ, S., Lampron, A., Simonyan, D., Asua, J., & Reviriego, E. (2015). Adaptation and evaluation of online self-learning modules to teach critical appraisal and evidence-based practice in nursing: An international collaboration. CIN: Computers, Informatics, Nursing, 33(7), 285-294. https://doi.org/10.1097/CIN.0000000000000156
Garrett, B. M., MacPhee, M., & Jackson, C. (2013). Evaluation of an eportfolio for the assessment of clinical competence in a baccalaureate nursing program. Nurse Education Today, 33(10), 1207-1213. https://doi.org/10.1016/j.nedt.2012.06.015
Gorchs-Molist, M., Sola-Munoz, S., Enjo-Perez, I., Querol-Gil, M., Carrera-Giraldo, D., Nicolas-Arfelis, J. M., Jimenez-Fabrega, F. X., & Perez de la Ossa, N. (2020). An online training intervention on prehospital stroke codes in Catalonia to improve the knowledge, pre-notification compliance and time performance of emergency medical services professionals. International Journal of Environmental Research and Public Health, 17(17), 6183. https://doi.org/10.3390/ijerph17176183
Harzing, A. W. (2007). Publish or Perish. (Version 7) https://harzing.com/resources/publish-or-perish
Hegerius, A., Caduff-Janosa, P., Savage, R., & Ellenius, J. (2020). E-Learning in pharmacovigilance: An evaluation of microlearning-based modules developed by Uppsala Monitoring Centre. Drug Safety, 43(11), 1171-1180. https://doi.org/10.1007/s40264-020-00981-w
Heuberger, R., & Clark, W. A. (2019). Synchronous delivery of online graduate education in clinical nutrition: An inquiry into student perceptions and preferences. Journal of Allied Health, 48(1), 61-66. https://pubmed.ncbi.nlm.nih.gov/30826832
Huang, X., Lin, J., & Demner-Fushman, D. (2006). Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annual Symposium proceedings. 359-363. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1839740/
Hughes, A. M., Gregory, M. E., Joseph, D. L., Sonesh, S. C., Marlow, S. L., Lacerenza, C. N., Benishek, L. E., King, H. B., & Salas, E. (2016). Saving lives: A meta-analysis of team training in healthcare. Journal of Applied Psychology, 101(9), 1266-1304. https://doi.org/10.1037/apl0000120
Jayakumar, N., Brunckhorst, O., Dasgupta, P., Khan, M. S., & Ahmed, K. (2015). e-Learning in surgical education: A systematic review. Journal of Surgical Education, 72(6), 1145-1157. https://doi.org/10.1016/j.jsurg.2015.05.008
Kemper, K. J. (2017). Brief online mindfulness training: Immediate impact. Journal of Evidence-Based Complementary and Alternative Medicine, 22(1), 75-80. https://doi.org/10.1177/2156587216639199
Khan, R. A. G., Khan, F. A., & Khan, M. A. (2011). Impact of training and development on organizational performance. Global Journal of Management and Business Research, 11(7). https://swsu.ru/sbornik-statey/pdf/8-Impact-of-Training-and-Development-on-Organizational-Performance.pdf
Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels (Vol. 1). Berrett-Koehler Publishers.
Kirkpatrick, J. D., & Kirkpatrick, D. (2006). Evaluating training programs: The four levels. Berrett-Koehler Publishers. https://search.alexanderstreet.com/view/work/bibliographic_entity%7Cbibliographic_details%7C2341903
Le Marne, F. A., McGinness, H., Slade, R., Cardamone, M., Balbir Singh, S., Connolly, A. M., & Bye, A. M. (2016). Evaluation of an e-learning resource on approach to the first unprovoked seizure. Journal of Paediatrics and Child Health, 52(9), 896-900. https://doi.org/10.1111/jpc.13277
Lee, S. J., Park, J., Lee, Y. J., Lee, S., Kim, W. H., & Yoon, H. B. (2020). The feasibility and satisfaction of an online global health education course at a single medical school: a retrospective study. Korean Journal of Medical Education, 32(4), 307-315. https://doi.org/10.3946/kjme.2020.178
Liaw, S. Y., Wong, L. F., Lim, E. Y., Ang, S. B., Mujumdar, S., Ho, J. T., Mordiffi, S. Z., & Ang, E. N. (2016). Effectiveness of a web-based simulation in improving nurses' workplace practice with deteriorating ward patients: A pre- and post-intervention study. Journal of Medical Internet Research, 18(2), e37. https://doi.org/10.2196/jmir.5294
Lima, M. S., Tonial, F. G., Basei, E., Brew, M. C., Grossmann, E., Haddad, A. E., Rivaldo, E. G., Vargas, I. A., & Bavaresco, C. S. (2019). Effectiveness of the distance learning strategy applied to orthodontics education: A systematic literature review. Telemedicine Journal and e-Health, 25(12), 1134-1143. https://doi.org/10.1089/tmj.2018.0286
Martinez, C., Castellano, Y., Andres, A., Fu, M., Feliu, A., Anton, L., Ballbe, M., Fernandez, P., Cabrera, S., Riccobene, A., Gavilan, E., Baena, A., Margalef, M., Tigova, O., Quiros, N., Guillen, O., Company, A., & Fernandez, E. (2019). Impact of an online training program in smoking cessation interventions in hospitals. Journal of Nursing Scholarship, 51(4), 449-458. https://doi.org/10.1111/jnu.12469
Moehead, A., DeSouza, K., Walsh, K., & Pit, S. W. (2020). A web-based dementia education program and its application to an Australian web-based dementia care competency and training network: Integrative systematic review. Journal of Medical Internet Research, 22(1), e16808. https://doi.org/10.2196/16808
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Peterson, J. L. (2016). Formative evaluations in online classes. Journal of Educators Online, 13(1). https://doi.org/10.9743/JEO.2016.1.8
Pokhrel, S., & Chhetri, R. (2021). A literature review on impact of COVID-19 pandemic on teaching and learning. Higher Education for the Future, 8(1), 133-141. https://doi.org/10.1177/2347631120983481
Prosser, M., Stephenson, T., Mathur, J., Enayati, H., Kadie, A., Abdi, M. M., Handuleh, J. I. M., & Keynejad, R. C. (2021). Reflective practice and transcultural psychiatry peer e-learning between Somaliland and the UK: A qualitative evaluation. BMC Medical Education, 21(1), 58. https://doi.org/10.1186/s12909-020-02465-y
Reese, D., Dolansky, M. A., Moore, S. M., Bolden, H., & Singh, M. K. (2021). Quality improvement education innovation: evaluation of Coursera MOOC ‘Take the Lead on Healthcare Quality Improvement’. Journal of Research in Nursing, 26(1-2), 62-78. https://doi.org/10.1177/1744987120982644
Reviriego, E., Cidoncha, M. A., Asua, J., Gagnon, M. P., Mateos, M., Garate, L., de Lorenzo, E., & Gonzalez, R. M. (2014). Online training course on critical appraisal for nurses: Adaptation and assessment. BMC Medical Education, 14, 136. https://doi.org/10.1186/1472-6920-14-136
Rouleau, G., Gagnon, M. P., Cote, J., Payne-Gagnon, J., Hudson, E., Dubois, C. A., & Bouix-Picasso, J. (2019). Effects of e-learning in a continuing education context on nursing care: Systematic review of systematic qualitative, quantitative, and mixed-studies reviews. Journal of Medical Internet Research, 21(10), e15118. https://doi.org/10.2196/15118
Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2), 74-101. https://doi.org/10.1177/1529100612436661
Salinas, J., Munoz, C., Albagli, A., Araya, G., & Vio, F. (2017). The contribution of distance education to health promotion in Chile. Health Promotion International, 32(5), 913-921. https://doi.org/10.1093/heapro/daw023
Salter, S. M., Karia, A., Sanfilippo, F. M., & Clifford, R. M. (2014). Effectiveness of e-learning in pharmacy education. American Journal of Pharmaceutical Education, 78(4), 83. https://doi.org/10.5688/ajpe78483
Schulz-Quach, C., Wenzel-Meyburg, U., & Fetz, K. (2018). Can elearning be used to teach palliative care? - medical students' acceptance, knowledge, and self-estimation of competence in palliative care after elearning. BMC Medical Education, 18(1), 82. https://doi.org/10.1186/s12909-018-1186-2
Simonsen, B. O., Daehlin, G. K., Johansson, I., & Farup, P. G. (2014). Improvement of drug dose calculations by classroom teaching or e-learning: a randomised controlled trial in nurses. BMJ Open, 4(10), e006025. https://doi.org/10.1136/bmjopen-2014-006025
Tannenbaum, C., & van Hoof, K. (2018). Effectiveness of online learning on health researcher capacity to appropriately integrate sex, gender, or both in grant proposals. Biology of Sex Differences, 9(1), 39. https://doi.org/10.1186/s13293-018-0197-3
Tricco, A. C., Langlois, E. V., & Straus, S. E. (2017). Rapid reviews to strengthen health policy and systems: A practical guide. World Health Organization. Geneva, Switzerland. https://www.who.int/alliance-hpsr/resources/publications/rapid-review-guide/en/
Uden-Holman, T., Bedet, J., Walkner, L., & Abd-Hamid, N. H. (2014). Adaptive scenarios: A training model for today's public health workforce. Journal of Public Health Management and Practice, 20 Suppl 5, S44-48. https://doi.org/10.1097/PHH.0000000000000067
United Nations. (2020). Policy brief: Education during COVID-19 and beyond. https://www.un.org/development/desa/dspd/wpcontent/uploads/sites/22/2020/08/sg_policy_brief_covid-19_and_education_august_2020.pdf
Vaona, A., Banzi, R., Kwag, K. H., Rigon, G., Cereda, D., Pecoraro, V., Tramacere, I., & Moja, L. (2018). E-learning for health professionals. Cochrane Database of Systematic Reviews, 2018(8). https://doi.org/10.1002/14651858.CD011736.pub2
Veritas Health Innovation. (2019). Covidence systematic review software. https://www.covidence.org
Whitt, K. J., Macri, C., O'Brien, T. J., & Wright, S. (2016). Improving nurse practitioners' competence with genetics: Effectiveness of an online course. Journal of the American Association of Nurse Practitioners, 28(3), 151-159. https://doi.org/10.1002/2327-6924.12282
Willman, B., Grankvist, K., & Bolenius, K. (2018). Evaluation of the clinical implementation of a large-scale online e-learning program on venous blood specimen collection guideline practices. Clinical Chemistry and Laboratory Medicine, 56(11), 1870-1877. https://doi.org/10.1515/cclm-2018-0051
Wlodarczyk, D., Chylinska, J., Lazarewicz, M., Rzadkiewicz, M., Jaworski, M., Adamus, M., Haugan, G., Lillefjell, M., & Espnes, G. A. (2017). Enhancing doctors' competencies in communication with and activation of older patients: The Promoting Active Aging (PRACTA) computer-based intervention study. Journal of Medical Internet Research, 19(2), e45. https://doi.org/10.2196/jmir.6948
Zafar, S., Safdar, S., & Zafar, A. N. (2014). Evaluation of use of e-Learning in undergraduate radiology education: A review. European Journal of Radiology, 83(12), 2277-2287. https://doi.org/10.1016/j.ejrad.2014.08.017
Downloads
Published
Issue
Section
License
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
