Using AI to Evaluate a Competency-Based Online Writing Course in Nursing
DOI:
https://doi.org/10.24059/olj.v27i3.3974Abstract
Nursing education is transitioning from traditional teaching to competency-based education. Additionally, more nursing courses and programs are now offered online. Scholarly writing is a powerful strategy to teach effective communication and critical thinking, both core competencies for safe and effective nursing practice. However, teaching writing online to nursing students is challenging due to a lack of research evaluating best practices, faculty time constraints, and inconsistent writing assessment. Automated essay scoring systems using artificial intelligence (AI) provide new opportunities for efficient, reliable, and valid assessment of writing skills. We used a quasi-experimental design to investigate the impact of a 14-week fully online competency-based writing course on students’ self-efficacy, task value, and writing performance. The participants were master’s nursing students enrolled in an existing one-semester online competency-based writing course for healthcare professionals. An AI-powered writing assessment, IntelliMetric®, and the SAWSES self-efficacy survey were administered pre- and post-intervention. The results showed statistically significant gains in self-efficacy and writing performance with large effect sizes. This study addresses the gap in nursing education regarding the assessment of online, research-based writing interventions on students’ scholarly writing capacity. Recommendations include implementing a required scholarly writing course in all graduate-level nursing programs, scaffolding students’ competency development with the cognitive apprenticeship model, using best practices from composition research to inform online instruction, and employing AI-powered automated essay scoring to evaluate students’ writing progress and instructional efficacy.
References
Abbott, M. R. B., & Shaw, P. (2019). Concordance within an RN to BSN program: Standardized writing assessment rubrics. Nursing Education Perspectives, 40(6), 372–373. https://doi.org/10.1097/01.NEP.0000000000000365
American Association of Colleges of Nursing. (2021a). The essentials: Core competencies for professional nursing education. https://www.aacnnursing.org/Portals/42/AcademicNursing/pdf/Essentials-2021.pdf
American Association of Colleges of Nursing. (2021b). Data spotlight: Distance education. https://www.aacnnursing.org/News-Information/News/View/ArticleId/24969/data-spotlight-distance-education
American Nursing Association. (2015). Scope and standards of practice (3rd ed.). Nursebooks.
Andre, J. A., & Graves, R. (2013). Writing requirements across nursing programs in Canada. Journal of Nursing Education, 52, 91–97. https://doi.org/10.3928/01484834- 20130114-02
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall.
Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman.
Baleghizadeh, S., & Gordani, Y. (2012). Academic writing and grammatical accuracy: The role of corrective feedback. Gist Education and Learning Research Journal, 6, 159–176.
Ball, E., Franks, H., Jenkins, J. E., McGrath, M. L., & Leigh, J. A. (2009). Annotation is a valuable tool to enhance learning and assessment in student essays. Nurse Education Today, 29(3), 284–291. https://doi.org/10.1016/j.nedt.2008.10.005
Bean, J. (2011). Engaging ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom (2nd ed.). Jossey-Bass.
Behrens, S., Johnson, A., Allard, M., & Caroli, A. (2016). I know it when I see it: Uncovering student and educator expectations about academic writing in higher education. Writing and Pedagogy, 8(2). https://doi.org/10.1558/wap.24108
Benner, P., Hughes, R. G., and Sutphen, M. (2008). Clinical reasoning, decision making, and action: Thinking critically and clinically. In R. G. Hughes (Ed.), Patient safety and quality: An evidence-based handbook for nurses. Agency for Healthcare Research and Quality.
Bernstein, D., & Greenhoot, A. F. (2014). Team-designed improvement of writing and critical thinking in large undergraduate courses. Teaching and Learning Inquiry, 2(1), 39–61.
Bickes, J. T., & Schim, S. M. (2010). Righting writing: Strategies for improving nursing student
papers. International Journal of Nursing Education Scholarship, 7, 8. https://doi.org/10.2202/1548-923X.1964
Billings, D., & Kowalski, K. (2006). Journaling: A strategy for developing reflective practitioners. The Journal of Continuing Education, 37(3), 104–105. https://doi.org/10.3928/00220124-20060301-02
Binding, L. L., Morck, A. C., & Moules, N. J. (2010). Learning to see the other: A vehicle of reflection. Nurse Education Today, 30(6), 591–594. https://doi.org/10.1016/j.nedt.2009.12.014
Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16.
Borglin, G. (2012). Promoting critical thinking and academic writing skills in nurse education. Nurse Education Today, 32(5), 611–613. https://doi.org/10.1016/j.nedt.2011.06.009
Borglin, G., & Fagerstrom, C. (2012). Nursing students’ understanding of critical thinking and appraisal and academic writing: A descriptive, qualitative study. Nurse Education in Practice, 12(6), 356–360. https://doi.org/10.1016/j.nepr.2012.04.009
Brannon, L., Courtney, J. P., Urbanski, C. P., Woodward, S. V., Reynolds, J. M., Iannone, A. E., Haag, K. D., Mach, K., Manship, L. A., & Kendrick, M. (2008). The five-paragraph essay and the deficit model of education. The English Journal, 98(2), 16–21. https://www.jstor.org/stable/40503377
Brennan, T. A., Leape, L. L., Laird, N. M., Hebert, L., Localio, A. R., Lawthers, A. G., Newhouse, J. P., Weiler, P. C., & Hiatt, H. (2004). Incidence of adverse events and negligence in hospitalized patients: Results of the Harvard Medical Practice Study I, 1991. Quality & Safety in Health Care, 13, 145–151.
Brown, V. S., Lewis, D., & Toussaint, M. (2018). Students’ perceptions of quality across four course development models. Online Learning Journal, 22(2). https://doi.org/10.24059/olj.v22i2.1213
Carter, L. (2008). Perceptions of writing confidence, critical thinking, and writing competence among registered nurse-learners studying online. Canadian Journal of University Continuing Education, 34(2). https://doi.org/10.21225/d5r592
Carter, R., Salamonson, Y., Ramjan, L. M., & Halcomb, E. (2018). Students use of exemplars to support academic writing in higher education: An integrative review. Nurse Education Today, 65, 87–93. https://doi.org/10.1016/j.nedt.2018.02.038
Chandler, G., Roberts, S. J., & DeMarco, R. F. (2005). Developing nursing voice
through writing in a group. Annual Review of Nursing Education, 3, 359–374.
Chaudoir, S., Lasiuk, G., & Trepanier, K. (2016). Writing assignments: A relatively emotional experience of learning to write in one baccalaureate nursing program. Quality Advancement in Nursing Education, 2, 1–21.
Clay, M. (1991). Becoming literate: The construction of inner control (2nd ed.). Heinemann.
Clay, M. (2005). Literacy lessons designed for individuals. Heinemann.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
Collins, A., Brown, J. S., & Newman, S. E. (1987). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Cognition and instruction: Issues and agendas. Lawrence Erlbaum Associates.
Ding, H. (2008). The use of cognitive and social apprenticeship to teach a disciplinary genre: Initiation of graduate students into NIH grant writing. Written Communication, 25(1), 3–52.
Eccles, J. (1983). Expectancies, values and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives: Psychological and sociological approaches (pp. 75–146). Freeman.
Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44, 350–383. https://doi.org/10.2307/2666999
Elliot, S. M. (2003). IntelliMetric®: From here to validity. In M. D. Shermis, & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 71–86). Lawrence Erlbaum.
Gazza, E. A., & Hunker, D. F. (2012). Facilitating scholarly writer development: The writing
scaffold. Nursing Forum, 47, 278–285. https://doi.org/10.1111/j.1744-6198.2012.00275.x
Gazza, E. A., Shellenbarger, T., & Hunker, D. F. (2018). MSN students’ self-assessed use of the knowledge, skills, and attitudes of scholarly writing. Nursing Education Perspectives, 39(6), 350–354. https://doi.org/10.1097/01.nep.0000000000000363
Gimenez, J. (2012). Disciplinary epistemologies, generic attributes and undergraduate academic writing in nursing and midwifery. Higher Education, 63, 401–419. https://doi.org/10.1007/s10734-011-9447-6
Goodman, B. (2011). The sociological imagination, provocative pedagogy and scholarship: Revaluing thinking and writing in nursing education. Nurse Education Today, 31(5), 427–428. https://doi.org/10.1016/j.nedt.2011.03.015
Grigoryan, A. (2017). Feedback 2.0 in online writing instruction: Combining audio-visual and text-based commentary to enhance student revision and writing competency. Journal of Computing in Higher Education, 29, 451–476.
Haisfield, L., Schultz, M., & McCann Associates (2012). California Common Assessment—Writing: Developed for the C6 Consortium. McCann Associates.
Harvard College Writing Program (2007). A brief guide to responding to student writing. https://writingproject.fas.harvard.edu/files/hwp/files/bg_responding_to_student_writing.pdf
Hathaway, J. (2015). Developing that voice: Locating academic writing tuition in the mainstream of higher education, Teaching in Higher Education, 20(5), 506–517. https://doi.org/10.1080/13562517.2015.1026891
Hawisher, G. E., & Pemberton, M. A. (2019). Writing across the curriculum encounters asynchronous learning networks or WAC meets up with ALN. Online Learning, 1(1). https://doi.org/10.24059/olj.v1i1.1940
Hawks, S. J., Turner, K. M., Derouin, A. L., Hueckel, R. M., Leonardelli, A. K., & Oermann, M. H. (2016). Writing across the curriculum: Strategies to improve the writing skills of nursing students. Nursing Forum, 51(4), 261–267. https://doi.org/10.1111/nuf.12151
Hussein M. A., Hassan, H., & Nasser, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science, 5, e208. http://doi.org/10.7717/peerj-cs.208
Ivanic, R. (1998). Writing and identity: The discoursal construction of identity in academic writing. John Benjamins.
Jefferies, D. M., McNally, S., Roberts, K., Stunden, A., D’Souza, S., & Glew, P. (2018). The importance of academic literacy for undergraduate nursing students and its relationship to future professional clinical practice: A systematic review. Nurse Education Today, 60, 84–91. https://doi.org/10.1016/j.nedt.2017.09.020
Klucevsek, K. M., & Brungard, A. B. (2016). Information literacy in science writing: how students find, identify, and use scientific literature. International Journal of Science Education, 38(17), 2573–2595.
Kozulin, A. (2002). Sociocultural theory and the mediated learning experience. School Psychology International, 23(1), 7–35.
Kwak, S. (2017). Approaches reflected in academic writing MOOCs. The International Review of Research in Open and Distributed Learning, 18. https://doi.org/10.19173/irrodl.v18i3.2845
Lavelle, E., Ball, S. C., & Maliszewski, G. (2013). Writing approaches of nursing students. Nurse Education Today, 33, 60–63. https://doi.org/10.1016/j.nedt.2011.10.021
Lea, M. R., & Street, B. V. (2006). The “academic literacies” model: Theory and applications. Theory Into Practice, 45(4), 368–377.
Leaver, C. A., Stanley, J. M., & Veenema, T. G. (2022). Impact of COVID-19 pandemic on the future of nursing education. Academic Medicine, 97(3), S82–S89. https://doi.org/10.1097/ACM.0000000000004528
Luthy, K. E., Peterson, N. E., Lassetter, J. H., & Callister, L. C. (2009). Successfully incorporating writing across the curriculum with advanced writing in nursing. Journal of Nursing Education, 48(1), 54–59. https://doi.org/10.3928/01484834-20090101-07
Martin, J. (2019). Building relationships and increasing engagement in the virtual classroom: Practical tools for the online instructor. Journal of Educators Online, 16(1). https://doi.org/10.9743/jeo.2019.16.1.9
McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299–325. https://doi.org/10.1007/bf01464076
McLeod, S., & Soven, M. (2000). Writing across the curriculum: A guide to developing programs. Sage. https://wac.colostate.edu/books/mcleod_soven/mcleod_soven.pdf
McMillan, L. R., & Raines, K. (2010). Headed in the “write” direction: Nursing student publication and health promotion in the community. Journal of Nursing Education, 49(7), 418–421. https://doi.org/10.3928/014834-20100430-05
McQuerrey, L. (2017). The importance of writing as a nurse practitioner. Chron. http://work.chron.com/importance-writing-nurse-practitioner-18156.html
Minnich, M., Kirkpatrick, A. J., Goodman, J. T., Whittaker, A., Chapple, H. S., Schoening, A. M., & Khanna, M. M. (2018). Writing across the curriculum: Reliability testing of a standardized rubric. Journal Of Nursing Education, 57(6), 366–370. https://doi.org/10.3928/01484834-20180522-08
Mitchell, K. M. (2018). Constructing writing practices in nursing. Journal of Nursing Education, 57(7) 399–407. https://doi.org/10.3928/01484834-20180618-04
Mitchell, K. M., Harrigan, T., Stefansson, T., & Setlack, H. (2017). Exploring self-efficacy and anxiety in first-year nursing students enrolled in a discipline-specific scholarly writing course. Quality Advancement in Nursing Education, 3(1). https://doi.org/10.17483/2368-6669.1084
Mitchell, K. M., McMillan, D. E., Lobchuk, M. M., & Nickel, N. (2021). Development and validation of the Situated Academic Writing Self-Efficacy Scale (SAWSES). Assessing Writing, 48(2), 100524. https://doi.org/10.1016/j.asw.2021.100524
Naber, J., & Wyatt, T. H. (2014). The effect of reflective writing interventions on the critical thinking skills and dispositions of baccalaureate nursing students. Nurse Education Today, 34(1), 67–72. https://doi.org/10.1016/j.nedt.2013.04.002
National Council of State Boards of Nursing. (Winter 2019). Next generation NCLEX news. https://www.ncsbn.org/NGN_Winter19.pdf
National Council of State Boards of Nursing. (2023). NCSBN Launches Next Generation NCLEX Exam. https://www.ncsbn.org/news/ncsbn-launches-next-generation-nclex-exam
Oermann, M. H. (2013). Enhancing writing in online education. In K. H. Frith & D. Clark (Eds.) Distance education in nursing (3rd ed., pp. 145–162). Springer.
Oermann, M. H., Leonardelli, A. K., Turner, K. M., Hawks, S. J., Derouin, A. L., & Hueckel, R. M. (2015). Systematic review of educational programs and strategies for developing students’ and nurses’ writing skills. Journal of Nursing Education, 54, 28–34. https://doi.org/10.3928/01484834-20141224-01
Peterson’s. (n.d.). Online schools. Retrieved July 16, 2023, from https://www.petersons.com/online-schools.aspx
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire. Technical report no. 91-B-004. The Regents of The University of Michigan.
Prestia, A. S. (2019). Reflection: A powerful leadership tool. Nurse Leader, 17(5), 465–467. https://doi.org/10.1016/j.mnl.2019.01.004
Quality Matters. (n.d.). Course design rubric standards. https://www.qualitymatters.org/qa-resources/rubric-standards/higher-ed-rubric
Ray, D. (2009). An experimental study of assignment sheets and assignment models in composition classrooms. Research in the Teaching of English, 43, 163–188.
Roberts, S. T., & Goss, G. (2009). Use of an online writing tutorial to improve writing skills in
nursing courses. Nurse Educator, 34, 262–265. https://doi.org/10.1097/NNE.0b013e3181bc740d
Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of the IntelliMetric® essay scoring system. The Journal of Technology, Learning, and Assessment, 4(4).
Seaman, J. E., Allen, I. E., & Seaman, J. (2018). Grade increase: Tracking distance education in the United States. Babson. https://www.bayviewanalytics.com/reports/gradeincrease.pdf
Sibiya, M. N. (2018). Effective communication in nursing. Intechopen. https://doi.org/10.5772/intechopen.74995
Smith, L. J. (2008) Grading written projects: What approaches do students find most helpful? Journal of Education for Business, 83(6), 325–330.
https://doi.org/10.3200/JOEB.83.6.325-330
Stewart, M. K. (2021). Social presence in online writing instruction: Distinguishing between presence, comfort, attitudes, and learning. Computers and Composition, 62. https://doi.org/10.1016/j.compcom.2021.102669
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295–312.
Troxler, H., Vann, J. C. J., & Oermann, M. H. (2011). How baccalaureate nursing programs teach writing. Nursing Forum, 46(4), 280–288. https://doi.org/10.1111/j.1744-6198.2011.00242.x
Tyndall, D. E., & Scott, E. S. (2017). Writing development in associate degree in nursing-to-baccalaureate degree in nursing students: Moving out of the comfort zone. Journal of Nursing Education, 56(3), 182–185. https://doi.org/10.3928/01484834-20170222-11
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
Wardle, E., & Roozen, K. (2012). Addressing the complexity of writing development: Toward an ecological model of assessment. Assessing Writing, 17(2), 106–119. https://doi.org/10.1016/j.asw.2012.01.001
Warnock, S. (2009). Teaching writing online: How and why. National Council of Teachers of English.
Warren, K. K., & Glass, G. V. (2010). Is assignment guidance effective? A systematic review of the literature on a widely used writing pedagogy. The Journal of Educational Research, 104, 8–22.
Wengel, K., & Fager, J. (2008). Does assignment guidance make a difference? An experimental study of effective of assignment sheets on student writing. Composition Studies, 36, 85–100.
Whitehead, D. (2002). The academic writing experiences of a group of student nurses: A
phenomenological study. Journal of Advanced Nursing, 38(5), 498–506. https://doi.org/10.1046/j.1365-2648.2002.02211.x
Wiggins, G., & McTighe, J. (1998). Understanding by design. ASCD.
Wolf, A. (2022). Adapting nursing programs to competency-based education. Nursing 2022, 52(2), 12–13. https://doi.org/10.1097/01.NURSE.0000806200.13094.90
Wolf, A., Carbonero, S. M., & Wolf, R. R. (2022). Empowering metacognitive learning in a competency-based curriculum in health professions through comprehensive learner record. Intersection: A Journal at the Intersection of Assessment and Learning, 3(2). https://aalhe.scholasticahq.com/article/36299-empowering-metacognitive-learning-in-a-competency-based-curriculum-through-comprehensive-learner-record
Wolf, R. R. & Wolf, A. (2022). Using a hierarchy of writing concerns to guide feedback and revision. Journal of Nursing Education, 61(7), 424–424. https://doi.org/10.3928/01484834-20220613-11
Wolters Kluwer. (2020). Forecast for the future: Technology trends in nursing education. https://www.wolterskluwer.com/en/solutions/lippincott-nursing-faculty/dean-survey
World Health Organization (2016). Nurse educator core competencies. WHO Press. https://www.who.int/publications/i/item/nurse-educator-core-competencies
Worrall-Carter, L., & Snell, R. (2003). Nurse academics meeting the challenges of scholarship and research. Contemporary Nurse, 16, 40–50.
Zhang, X., Tai, D., Pforsich, H., Lin, V. W. (2018). United States registered nurse workforce report card and shortage forecast: A revisit. American Journal of Medical Quality, 33(3), 229–236. https://doi.org/10.1177/1062860617738328
Zimmerman, W., Altman, B., Simunich, B., Shattuck, K., & Burch, B. (2020). Evaluating online course quality: A study on implementation of course quality standards. Online Learning, 24(4), 147–163. https://doi.org/10.24059/olj.v24i4.2325
Zygmont, D. M., & Schaefer, K. M. (2006). Writing across the nursing curriculum project. Annual Review of Nursing Education, 4, 275–290.
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Rebecca Red Wolf, Dr. Andrew B. Wolf
This work is licensed under a Creative Commons Attribution 4.0 International License.
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
