Using AI to Evaluate a Competency-Based Online Writing Course in Nursing




Nursing education is transitioning from traditional teaching to competency-based education. Additionally, more nursing courses and programs are now offered online. Scholarly writing is a powerful strategy to teach effective communication and critical thinking, both core competencies for safe and effective nursing practice. However, teaching writing online to nursing students is challenging due to a lack of research evaluating best practices, faculty time constraints, and inconsistent writing assessment. Automated essay scoring systems using artificial intelligence (AI) provide new opportunities for efficient, reliable, and valid assessment of writing skills. We used a quasi-experimental design to investigate the impact of a 14-week fully online competency-based writing course on students’ self-efficacy, task value, and writing performance. The participants were master’s nursing students enrolled in an existing one-semester online competency-based writing course for healthcare professionals. An AI-powered writing assessment, IntelliMetric®, and the SAWSES self-efficacy survey were administered pre- and post-intervention. The results showed statistically significant gains in self-efficacy and writing performance with large effect sizes. This study addresses the gap in nursing education regarding the assessment of online, research-based writing interventions on students’ scholarly writing capacity. Recommendations include implementing a required scholarly writing course in all graduate-level nursing programs, scaffolding students’ competency development with the cognitive apprenticeship model, using best practices from composition research to inform online instruction, and employing AI-powered automated essay scoring to evaluate students’ writing progress and instructional efficacy.

Author Biography

Dr. Andrew B. Wolf, St. John Fisher University

Assistant Dean of Online Learning 

Wegman's School of Pharmacy


Abbott, M. R. B., & Shaw, P. (2019). Concordance within an RN to BSN program: Standardized writing assessment rubrics. Nursing Education Perspectives, 40(6), 372–373.

American Association of Colleges of Nursing. (2021a). The essentials: Core competencies for professional nursing education.

American Association of Colleges of Nursing. (2021b). Data spotlight: Distance education.

American Nursing Association. (2015). Scope and standards of practice (3rd ed.). Nursebooks.

Andre, J. A., & Graves, R. (2013). Writing requirements across nursing programs in Canada. Journal of Nursing Education, 52, 91–97. 20130114-02

Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall.

Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman.

Baleghizadeh, S., & Gordani, Y. (2012). Academic writing and grammatical accuracy: The role of corrective feedback. Gist Education and Learning Research Journal, 6, 159–176.

Ball, E., Franks, H., Jenkins, J. E., McGrath, M. L., & Leigh, J. A. (2009). Annotation is a valuable tool to enhance learning and assessment in student essays. Nurse Education Today, 29(3), 284–291.

Bean, J. (2011). Engaging ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom (2nd ed.). Jossey-Bass.

Behrens, S., Johnson, A., Allard, M., & Caroli, A. (2016). I know it when I see it: Uncovering student and educator expectations about academic writing in higher education. Writing and Pedagogy, 8(2).

Benner, P., Hughes, R. G., and Sutphen, M. (2008). Clinical reasoning, decision making, and action: Thinking critically and clinically. In R. G. Hughes (Ed.), Patient safety and quality: An evidence-based handbook for nurses. Agency for Healthcare Research and Quality.

Bernstein, D., & Greenhoot, A. F. (2014). Team-designed improvement of writing and critical thinking in large undergraduate courses. Teaching and Learning Inquiry, 2(1), 39–61.

Bickes, J. T., & Schim, S. M. (2010). Righting writing: Strategies for improving nursing student

papers. International Journal of Nursing Education Scholarship, 7, 8.

Billings, D., & Kowalski, K. (2006). Journaling: A strategy for developing reflective practitioners. The Journal of Continuing Education, 37(3), 104–105.

Binding, L. L., Morck, A. C., & Moules, N. J. (2010). Learning to see the other: A vehicle of reflection. Nurse Education Today, 30(6), 591–594.

Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16.

Borglin, G. (2012). Promoting critical thinking and academic writing skills in nurse education. Nurse Education Today, 32(5), 611–613.

Borglin, G., & Fagerstrom, C. (2012). Nursing students’ understanding of critical thinking and appraisal and academic writing: A descriptive, qualitative study. Nurse Education in Practice, 12(6), 356–360.

Brannon, L., Courtney, J. P., Urbanski, C. P., Woodward, S. V., Reynolds, J. M., Iannone, A. E., Haag, K. D., Mach, K., Manship, L. A., & Kendrick, M. (2008). The five-paragraph essay and the deficit model of education. The English Journal, 98(2), 16–21.

Brennan, T. A., Leape, L. L., Laird, N. M., Hebert, L., Localio, A. R., Lawthers, A. G., Newhouse, J. P., Weiler, P. C., & Hiatt, H. (2004). Incidence of adverse events and negligence in hospitalized patients: Results of the Harvard Medical Practice Study I, 1991. Quality & Safety in Health Care, 13, 145–151.

Brown, V. S., Lewis, D., & Toussaint, M. (2018). Students’ perceptions of quality across four course development models. Online Learning Journal, 22(2).

Carter, L. (2008). Perceptions of writing confidence, critical thinking, and writing competence among registered nurse-learners studying online. Canadian Journal of University Continuing Education, 34(2).

Carter, R., Salamonson, Y., Ramjan, L. M., & Halcomb, E. (2018). Students use of exemplars to support academic writing in higher education: An integrative review. Nurse Education Today, 65, 87–93.

Chandler, G., Roberts, S. J., & DeMarco, R. F. (2005). Developing nursing voice

through writing in a group. Annual Review of Nursing Education, 3, 359–374.

Chaudoir, S., Lasiuk, G., & Trepanier, K. (2016). Writing assignments: A relatively emotional experience of learning to write in one baccalaureate nursing program. Quality Advancement in Nursing Education, 2, 1–21.

Clay, M. (1991). Becoming literate: The construction of inner control (2nd ed.). Heinemann.

Clay, M. (2005). Literacy lessons designed for individuals. Heinemann.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Collins, A., Brown, J. S., & Newman, S. E. (1987). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Cognition and instruction: Issues and agendas. Lawrence Erlbaum Associates.

Ding, H. (2008). The use of cognitive and social apprenticeship to teach a disciplinary genre: Initiation of graduate students into NIH grant writing. Written Communication, 25(1), 3–52.

Eccles, J. (1983). Expectancies, values and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives: Psychological and sociological approaches (pp. 75–146). Freeman.

Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44, 350–383.

Elliot, S. M. (2003). IntelliMetric®: From here to validity. In M. D. Shermis, & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 71–86). Lawrence Erlbaum.

Gazza, E. A., & Hunker, D. F. (2012). Facilitating scholarly writer development: The writing

scaffold. Nursing Forum, 47, 278–285.

Gazza, E. A., Shellenbarger, T., & Hunker, D. F. (2018). MSN students’ self-assessed use of the knowledge, skills, and attitudes of scholarly writing. Nursing Education Perspectives, 39(6), 350–354.

Gimenez, J. (2012). Disciplinary epistemologies, generic attributes and undergraduate academic writing in nursing and midwifery. Higher Education, 63, 401–419.

Goodman, B. (2011). The sociological imagination, provocative pedagogy and scholarship: Revaluing thinking and writing in nursing education. Nurse Education Today, 31(5), 427–428.

Grigoryan, A. (2017). Feedback 2.0 in online writing instruction: Combining audio-visual and text-based commentary to enhance student revision and writing competency. Journal of Computing in Higher Education, 29, 451–476.

Haisfield, L., Schultz, M., & McCann Associates (2012). California Common Assessment—Writing: Developed for the C6 Consortium. McCann Associates.

Harvard College Writing Program (2007). A brief guide to responding to student writing.

Hathaway, J. (2015). Developing that voice: Locating academic writing tuition in the mainstream of higher education, Teaching in Higher Education, 20(5), 506–517.

Hawisher, G. E., & Pemberton, M. A. (2019). Writing across the curriculum encounters asynchronous learning networks or WAC meets up with ALN. Online Learning, 1(1).

Hawks, S. J., Turner, K. M., Derouin, A. L., Hueckel, R. M., Leonardelli, A. K., & Oermann, M. H. (2016). Writing across the curriculum: Strategies to improve the writing skills of nursing students. Nursing Forum, 51(4), 261–267.

Hussein M. A., Hassan, H., & Nasser, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science, 5, e208.

Ivanic, R. (1998). Writing and identity: The discoursal construction of identity in academic writing. John Benjamins.

Jefferies, D. M., McNally, S., Roberts, K., Stunden, A., D’Souza, S., & Glew, P. (2018). The importance of academic literacy for undergraduate nursing students and its relationship to future professional clinical practice: A systematic review. Nurse Education Today, 60, 84–91.

Klucevsek, K. M., & Brungard, A. B. (2016). Information literacy in science writing: how students find, identify, and use scientific literature. International Journal of Science Education, 38(17), 2573–2595.

Kozulin, A. (2002). Sociocultural theory and the mediated learning experience. School Psychology International, 23(1), 7–35.

Kwak, S. (2017). Approaches reflected in academic writing MOOCs. The International Review of Research in Open and Distributed Learning, 18.

Lavelle, E., Ball, S. C., & Maliszewski, G. (2013). Writing approaches of nursing students. Nurse Education Today, 33, 60–63.

Lea, M. R., & Street, B. V. (2006). The “academic literacies” model: Theory and applications. Theory Into Practice, 45(4), 368–377.

Leaver, C. A., Stanley, J. M., & Veenema, T. G. (2022). Impact of COVID-19 pandemic on the future of nursing education. Academic Medicine, 97(3), S82–S89.

Luthy, K. E., Peterson, N. E., Lassetter, J. H., & Callister, L. C. (2009). Successfully incorporating writing across the curriculum with advanced writing in nursing. Journal of Nursing Education, 48(1), 54–59.

Martin, J. (2019). Building relationships and increasing engagement in the virtual classroom: Practical tools for the online instructor. Journal of Educators Online, 16(1).

McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299–325.

McLeod, S., & Soven, M. (2000). Writing across the curriculum: A guide to developing programs. Sage.

McMillan, L. R., & Raines, K. (2010). Headed in the “write” direction: Nursing student publication and health promotion in the community. Journal of Nursing Education, 49(7), 418–421.

McQuerrey, L. (2017). The importance of writing as a nurse practitioner. Chron.

Minnich, M., Kirkpatrick, A. J., Goodman, J. T., Whittaker, A., Chapple, H. S., Schoening, A. M., & Khanna, M. M. (2018). Writing across the curriculum: Reliability testing of a standardized rubric. Journal Of Nursing Education, 57(6), 366–370.

Mitchell, K. M. (2018). Constructing writing practices in nursing. Journal of Nursing Education, 57(7) 399–407.

Mitchell, K. M., Harrigan, T., Stefansson, T., & Setlack, H. (2017). Exploring self-efficacy and anxiety in first-year nursing students enrolled in a discipline-specific scholarly writing course. Quality Advancement in Nursing Education, 3(1).

Mitchell, K. M., McMillan, D. E., Lobchuk, M. M., & Nickel, N. (2021). Development and validation of the Situated Academic Writing Self-Efficacy Scale (SAWSES). Assessing Writing, 48(2), 100524.

Naber, J., & Wyatt, T. H. (2014). The effect of reflective writing interventions on the critical thinking skills and dispositions of baccalaureate nursing students. Nurse Education Today, 34(1), 67–72.

National Council of State Boards of Nursing. (Winter 2019). Next generation NCLEX news.

National Council of State Boards of Nursing. (2023). NCSBN Launches Next Generation NCLEX Exam.

Oermann, M. H. (2013). Enhancing writing in online education. In K. H. Frith & D. Clark (Eds.) Distance education in nursing (3rd ed., pp. 145–162). Springer.

Oermann, M. H., Leonardelli, A. K., Turner, K. M., Hawks, S. J., Derouin, A. L., & Hueckel, R. M. (2015). Systematic review of educational programs and strategies for developing students’ and nurses’ writing skills. Journal of Nursing Education, 54, 28–34.

Peterson’s. (n.d.). Online schools. Retrieved July 16, 2023, from

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire. Technical report no. 91-B-004. The Regents of The University of Michigan.

Prestia, A. S. (2019). Reflection: A powerful leadership tool. Nurse Leader, 17(5), 465–467.

Quality Matters. (n.d.). Course design rubric standards.

Ray, D. (2009). An experimental study of assignment sheets and assignment models in composition classrooms. Research in the Teaching of English, 43, 163–188.

Roberts, S. T., & Goss, G. (2009). Use of an online writing tutorial to improve writing skills in

nursing courses. Nurse Educator, 34, 262–265.

Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of the IntelliMetric® essay scoring system. The Journal of Technology, Learning, and Assessment, 4(4).

Seaman, J. E., Allen, I. E., & Seaman, J. (2018). Grade increase: Tracking distance education in the United States. Babson.

Sibiya, M. N. (2018). Effective communication in nursing. Intechopen.

Smith, L. J. (2008) Grading written projects: What approaches do students find most helpful? Journal of Education for Business, 83(6), 325–330.

Stewart, M. K. (2021). Social presence in online writing instruction: Distinguishing between presence, comfort, attitudes, and learning. Computers and Composition, 62.

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295–312.

Troxler, H., Vann, J. C. J., & Oermann, M. H. (2011). How baccalaureate nursing programs teach writing. Nursing Forum, 46(4), 280–288.

Tyndall, D. E., & Scott, E. S. (2017). Writing development in associate degree in nursing-to-baccalaureate degree in nursing students: Moving out of the comfort zone. Journal of Nursing Education, 56(3), 182–185.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.

Wardle, E., & Roozen, K. (2012). Addressing the complexity of writing development: Toward an ecological model of assessment. Assessing Writing, 17(2), 106–119.

Warnock, S. (2009). Teaching writing online: How and why. National Council of Teachers of English.

Warren, K. K., & Glass, G. V. (2010). Is assignment guidance effective? A systematic review of the literature on a widely used writing pedagogy. The Journal of Educational Research, 104, 8–22.

Wengel, K., & Fager, J. (2008). Does assignment guidance make a difference? An experimental study of effective of assignment sheets on student writing. Composition Studies, 36, 85–100.

Whitehead, D. (2002). The academic writing experiences of a group of student nurses: A

phenomenological study. Journal of Advanced Nursing, 38(5), 498–506.

Wiggins, G., & McTighe, J. (1998). Understanding by design. ASCD.

Wolf, A. (2022). Adapting nursing programs to competency-based education. Nursing 2022, 52(2), 12–13.

Wolf, A., Carbonero, S. M., & Wolf, R. R. (2022). Empowering metacognitive learning in a competency-based curriculum in health professions through comprehensive learner record. Intersection: A Journal at the Intersection of Assessment and Learning, 3(2).

Wolf, R. R. & Wolf, A. (2022). Using a hierarchy of writing concerns to guide feedback and revision. Journal of Nursing Education, 61(7), 424–424.

Wolters Kluwer. (2020). Forecast for the future: Technology trends in nursing education.

World Health Organization (2016). Nurse educator core competencies. WHO Press.

Worrall-Carter, L., & Snell, R. (2003). Nurse academics meeting the challenges of scholarship and research. Contemporary Nurse, 16, 40–50.

Zhang, X., Tai, D., Pforsich, H., Lin, V. W. (2018). United States registered nurse workforce report card and shortage forecast: A revisit. American Journal of Medical Quality, 33(3), 229–236.

Zimmerman, W., Altman, B., Simunich, B., Shattuck, K., & Burch, B. (2020). Evaluating online course quality: A study on implementation of course quality standards. Online Learning, 24(4), 147–163.

Zygmont, D. M., & Schaefer, K. M. (2006). Writing across the nursing curriculum project. Annual Review of Nursing Education, 4, 275–290.






2023 OLC Conference Special Issue