Automatic Detection of Metacognitive Language and Student Achievement in an Online STEM College Course

Hannah Valdiviejas; Renato Ferreira Leitão Azevedo; Philip Nigel Bosch; Michelle Perry

doi:10.24059/olj.v28i3.4127

Authors

Hannah Valdiviejas University of Illinois Urbana Champaign
Renato Ferreira Leitão Azevedo University of Illinois Urbana Champaign
Nigel Bosch
Michelle Perry

DOI:

https://doi.org/10.24059/olj.v28i3.4127

Keywords:

metacognition, online, college, STEM, underrepresentation

Abstract

Metacognition is a valuable tool for learning, due to its role in self-regulated learning. However, online learning settings bring new challenges for engaging in metacognition given the unique opportunities and challenges presented by the online space, especially for diverse populations and students underrepresented in STEM (UR-STEM). Thus, we investigated whether a relation existed between college STEM students’ metacognition—measured by their spontaneously produced metacognitive phrases in online course discussions forums—and their success in an online STEM college course—measured by their final course grade. Using Bayesian generalized linear models, we examined whether this relation differed for UR-STEM compared to non-UR-STEM students and whether related course behaviors (i.e., engagement and verbosity) and prior knowledge predicted variance in course grade. Metacognition plausibly predicted course grade and we found no plausible differences between UR- and non-UR-STEM students, suggesting that the online space could afford students from diverse groups the capacity to engage equally in a critical aspect of self-regulated learning: metacognition. Implications of the results for teaching and learning STEM content in the online space are discussed.

References

Ahif, M., & McNeil, S. (2023). A systematic review of research on moderators in asynchronous online discussions. Online Learning Journal, 27(1), 219–262. https://doi.org/10.24059/olj.v27i1.3381

Akyol, Z., & Garrison, D. R. (2011). Assessing metacognition in an online Community of Inquiry. Internet and Higher Education, 14(3), 183–190. https://doi.org/10.1016/j.iheduc.2011.01.005

Al-Gaseem, M., Bakkar, B., & Al-Zoubi, S. (2020). Metacognitive thinking skills among talented science education students. Journal for the Education of Gifted Young Scientists, 8(2), 897–904. https://doi.org/10.17478/jegys.707205

Allen, E. I., Seaman, J., Poulin, R., & Taylor Straut, T. (2016). Online report card: Tracking online education in the United States. Babson Survey Research Group and Quahog Research Group.

Anyichie, A. C., Butler, D. L. (2017). A culturally responsive self-regulated learning framework. In P. Chen (Chair), Examining synergistic relationships among self-regulated learning and motivational variables. Annual meeting of the American Educational Research Association, San Antonio, TX, United States.

Andrzejewski, C. E., Davis, H. A., Shalter Bruening, P., & Poirier, R. R. (2016). Can a self-regulated strategy intervention close the achievement gap? Exploring a classroom-based intervention in 9th grade earth science. Learning and Individual Differences, 49, 85–99. https://doi.org/10.1016/j.lindif.2016.05.013

Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of Educational Psychology, 96, 523–535. https://doi.org/10.1037/0022-0663.96.3.523

Azevedo, R., Mudrick, N. V., Taub, M., & Bradbury, A. E. (2019). Self-regulation in computer-assisted learning systems. In J. Dunlosky & K. A. Rawson (Eds.), The Cambridge handbook of cognition and education (pp. 587–618). Cambridge University Press. http://doi.org/10.1017/9781108235631.024

Baumeister, R. F., & Vohs, K. D. (Eds.) (2007). Encyclopedia of social psychology. Sage. https://dx.doi.org/10.4135/9781412956253

Bernacki, M. L., Aguilar, A. C., & Byrnes, J. P. (2011). Self-regulated learning and technology-enhanced learning environments: An opportunity-propensity analysis. In G. Dettori & D. Persico (Eds.), Fostering self-regulated learning through ICT (pp. 1–26). IGI Global. http://doi.org/10.4018/978-1-61692-901-5.ch001

Bernacki, M. L., Vosicka, L., & Utz, J. C. (2020). Can a brief, digital skill training intervention help undergraduates “learn to learn” and improve their STEM achievement? Journal of Educational Psychology, 112(4), 765–781. https://doi.org/10.1037/edu0000405

Bürkner, P.-C. (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80(1), 1–28. https://doi.org/10.18637/jss.v080.i01

Bürkner, P.-C., & Vuorre, M. (2019). Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Sciences, 2(1), 77–101. https://doi.org/10.1177/2515245918823199

Callaway, C., Campbell, G., Dzikovska, M., Fallow, E., Moore, J., & Steinhauser, N. (2009). Metacognitive awareness versus linguistic politeness: Expressions of confusion in tutorial dialogues. Proceedings of the Annual Meeting of the Cognitive Science Society, 31. https://escholarship.org/uc/item/3xw2x09f

Cardelle-Elawar, M. (1995). Effects of metacognitive instruction on low achievers in mathematics problems. Teaching & Teacher Education, 11, 81–95. https://doi.org/10.1016/0742-051x(94)00019-3

Cardinale, J. A., & Johnson, B. C. (2017). Metacognition modules: A scaffolded series of online assignments designed to improve students’ study skills. Journal of Microbiology & Biology Education, 18(1), 1–8. https://doi.org/10.1128/jmbe.v18i1.1212

Carlone, H. B., Haun-Frank, J., & Webb, A. (2011). Assessing equity beyond knowledge- and skills- based outcomes: A comparative ethnography of two fourth-grade reform-based science classrooms. Journal of Research in Science Teaching, 48(5), 459–485. https://doi.org/10.1002/tea.20413

Carlone, H. B., & Johnson, A. (2007). Understanding the science experiences of successful women of color: Science identity as an analytic lens. Journal of Research in Science Teaching, 44,1187–1218. https://doi.org/10.1002/tea.20413

Castaño-Garrido, C., Garay, U., & Maiz, I. (2017). Factores de éxito académico en la integración de los MOOC en el aula universitaria. Revista Española de Pedagogía, 75(266), 65–82. https://doi.org/10.22550/rep75-1-2017-11

Chen, J., & Bogachenko, T. (2022). Online community building in distance education: The case of social presence in the Blackboard discussion board versus multimodal VoiceThread interaction. Educational Technology & Society, 25(2), 62–75. https://www.jstor.org/stable/10.2307/48660124

Chen, C., & Zhang, L. J. (2017). An intercultural analysis of the use of hedging by Chinese and Anglophone academic English writers. Applied Linguistics Review, 8(1), 1–34. https://doi.org/10.1515/applirev-2016-2009

Crues, R. W., Henricks, G. M., Perry, M., Bhat, S., Anderson, C. J., Shaik, N., & Angrave, L. (2018). How do gender, learning goals, and forum participation predict persistence in a computer science MOOC? ACM Transactions on Computing Education, 18(4). https://doi.org/10.1145/3152892

Dent, A., & Koenka, A. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8

Efklides, A., Schwartz, B. L., & Brown, V. (2018). Motivation and affect in self-regulated learning: Does metacognition play a role? In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). Routledge.

Fielding, I., Winters, Greene, J. A. & Costich., C. M. (2008). Self-regulation of learning within computer-based learning environments: A critical analysis. Educational Psychology Review, 20(4), 429–444. https://doi.org/10.1007/s10648-008-9080-9

Fry, R., Kennedy, B., & Funk, C. (2021). STEM jobs see uneven progress in increasing gender, racial and ethnic diversity. Pew Research Center. https://www.pewresearch.org/science/2021/04/01/stem-jobs-see-uneven-progress-in-increasing-gender-racial-and-ethnic-diversity/

Garrison, D. R. (2022). Shared metacognition in a Community of Inquiry. Online Learning Journal, 26(1), 6–18. https://doi.org/10.24059/olj.v26i1.3023

Garrison, D. R. & Akyol, Z. (2013). Toward the development of a metacognition construct for communities of inquiry. The Internet and Higher Education, 17, 84–89. https://doi.org/10.1016/j.iheduc.2012.11.005

Greene, J., Plumley, R., Urban, C., Bernacki, M., Gates, K., Hogan, K., Panter, A., & Demetriou, C. (2019). Modeling temporal self-regulatory processing in a higher education biology course. Learning and Instruction, 72, Article 101201. https://doi.org/10.1016/j.learninstruc.2019.04.002

Guo, L. (2022). Using metacognitive prompts to enhance self‐regulated learning and learning outcomes: A meta‐analysis of experimental studies in computer‐based learning environments. Journal of Computer Assisted Learning, 38(3), 811–832. https://doi.org/10.1111/jcal.12650

Harrison, G. M., & Vallin, L. M. (2018). Evaluating the metacognitive awareness inventory using empirical factor-structure evidence. Metacognition and Learning, 13(1), 15–38. https://doi.org/10.1007/s11409-017-9176-z

Hadie, S., Mohd I., Hassan, A., Ismail, H., Talip, S., & Rahim, A. (2018). Empowering students’ minds through a cognitive load theory-based lecture model: A metacognitive approach. Innovations in Education and Teaching International, 55, 398–407. https://doi.org/10.1080/14703297.2016.1252685

Hart, J. T. (1965). Memory and the feeling-of-knowing experience. Journal of Educational Psychology, 56(4), 208–216. https://doi.org/10.1037/h0022263

Heil, J., & Ifenthaler, D. (2023). Online assessment in higher education: A systematic review. Online Learning Journal, 27(1), 187–218. https://doi.org/10.24059/olj.v27i1.3398

Henricks, G. M., Bhat, S., & Perry, M. (2021). Gender and gendered discourse in two online STEM college courses. Computer-Based Learning in Context, 3(1), 1–16. https://www.upenn.edu/learninganalytics/CBLC/issue-3-1/CBLC-2021-3-1-1.pdf

Huang, E., Valdiviejas, H., & Bosch, N. (2019). I’m sure! Automatic detection of metacognition in online course discussion forums. Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII 2019), 241–247. https://doi.org/10.1109/ACII.2019.8925506

Lehmann, T., Haehnlein, I., & Ifenthaler, D. (2014). Cognitive, metacognitive and motivational perspectives on preflection in self-regulated online learning. Computers in Human Behavior, 32, 313–323. https://doi.org/10.1016/j.chb.2013.07.051.

Mata, A., Ferreira, M. B., & Sherman, S. J. (2013). The metacognitive advantage of deliberative thinkers: A dual-process perspective on overconfidence. Journal of Personality and Social Psychology, 105(3), 353–373. https://doi.org/10.1037/a0033640

McCarthy, K. S., Likens, A. D., Johnson, A. M., Guerrero, T. A., & McNamara, D. S. (2018). Metacognitive overload!: Positive and negative effects of metacognitive prompts in an intelligent tutoring system. International Journal of Artificial Intelligence in Education, 28, 420–438. https://doi.org/10.1007/s40593-018-0164-5

Means, B., & Neisler, J. (2023). Bridging theory and measurement of student engagement: A practical approach. Online Learning Journal. 27(4), 26–47. https://doi.org/10.24059/olj.v27i4.4034

Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. U.S. Department of Education.

Nacu, D. C., Martin, C. K., Sandherr, J., & Pinkard, N. (2015). Encouraging online contributions in underrepresented populations. 2015 Research on Equity and Sustained Participation in Engineering, Computing, and Technology, RESPECT 2015. https://ieeexplore.ieee.org/document/7296503

National Center for Science and Engineering Statistics (NCSES). (2023). Diversity and STEM: Women, Minorities, and Persons with Disabilities 2023. Special Report NSF 23-315. National Science Foundation. https://ncses.nsf.gov/wmpd

Nealy, S., & Orgill, M. (2019). Postsecondary underrepresented minority STEM students’ perceptions of their science identity. Journal of Negro Education 88(3), 249–268. https://muse.jhu.edu/article/802592

Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. The Journal of General Psychology. 135(2), 151–165. https://doi.org/10.3200/GENP.135.2.151-166

Nietfeld, J. L., Cao, L., & Osborne, J. W. (2005). Metacognitive monitoring accuracy and student performance in the postsecondary classroom. The Journal of Experimental Education, 74, 7–28. https://www.jstor.org/stable/20157410

Nietfeld, J. L., & Schraw, G. (2002). The effect of knowledge and strategy training on monitoring accuracy. The Journal of Educational Research, 95(3), 131–142. https://doi.org/10.1080/00220670209596583

Nieuwoudt, J. E. (2020). Investigating synchronous and asynchronous class attendance as predictors of academic success in online education. Australasian Journal of Educational Technology, 36(3), 15–25. https://doi.org/10.14742/ajet.5137

Nelson, T. O., & Narens, L. (1994). Why investigate metacognition? In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 1–25). MIT Press. https://doi.org/10.7551/mitpress/4561.003.0003

Parkes, M., Stein, S., & Reading, C. (2015). Student preparedness for university e-learning environments. Internet and Higher Education, 25, 1–10. https://doi.org/10.1016/j.iheduc.2014.10.002

Perry, M., & Lewis, J. (1999). Verbal imprecision as a tool for understanding knowledge in transition. Developmental Psychology, 35, 749–759. https://doi.org/10.1037/0012-1649.35.3.749

Phan, T., McNeil, S. G., & Robin, B. R. (2016). Students’ patterns of engagement and course performance in a Massive Open Online Course. Computers & Education, 95, 36–44. https://doi.org/10.1016/j.compedu.2015.11.015

R Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

Reder, L. M. (Ed.). (1996). Implicit memory and metacognition. Erlbaum.

Rioch, K. E., & Tharp, J. L. (2022). Student engagement practices and GPA among RN-BSN students. Online Learning Journal, 26(1), 198–217. https://doi.org/10.24059/olj.v26i2.2680

Ross, M. E., Green, S. B., Salisbury-Glennon, J. D., & Tollefson, N. (2006). College students’ study strategies as a function of testing: An investigation into metacognitive self-regulation. Innovative Higher Education, 30(5), 361–375. https://doi.org/10.1007/s10755-005-9004-2

Rovai, A. P., & Downey, J. R. (2010) Why some distance education programs fail while others succeed in a global environment. The Internet and Higher Education, 13(3), 141–147, https://doi.org/10.1016/j.iheduc.2009.07.001.

Rovers, S. F., Clarebout, G., Savelberg, H. H., de Bruin, A. B., & van Merriënboer, J. J. (2019). Granularity matters: Comparing different ways of measuring self-regulated learning. Metacognition and Learning, 14, 1–19. https://doi.org/10.1007/s11409-019-09188-6

Sadaf A., Kim, S. Y., & Olesova, L. (2022). Relationship between metacognition and online Community of Inquiry in an online case-based course. Online Learning Journal, 26(4), 78–93. https://doi.org/10.24059/olj.v26i4.3474

Schuster, C., Wirth, J., Stebner, F., Geukes, S., Jansen, M., & Leutner, D. (2023). The effects of direct and indirect training in metacognitive learning strategies on near and far transfer in self-regulated learning. Learning and Instruction, 83.

https://doi.org/10.1016/j.learninstruc.2022.101708

Sharma, B., Nand, R., Naseem, M., & Reddy, E. V. (2020). Effectiveness of online presence in a blended higher learning environment in the Pacific. Studies in Higher Education, 45(8),1547–1565. https://doi.org/10.1080/03075079.2019.1602756

Siegel, A. L. M., & Castel, A. D. (2019). Age-related differences in metacognition for memory capacity and selectivity. Memory, 27(9), 1236–1249. https://doi.org/10.1080/09658211.2019.1645859

Snyder, M. M., & Dringus, L. P. (2014). An exploration of metacognition in asynchronous student-led discussions: A qualitative inquiry. Journal of Asynchronous Learning Networks, 18(2), 1–19. https://doi.org/10.24059/olj.v18i2.418

Stanton, J. D., Neider, X. N., Gallegos, I. J., & Clark, N. C. (2015). Differences in metacognitive regulation in introductory biology students: When prompts are not enough. CBE Life Sciences Education, 14(2). https://doi.org/10.1187/cbe.14-08-0135

Stanton, J. D., Sebesta, A. J., & Dunlosky, J. (2021). Fostering metacognition to support student learning and performance. CBE Life Sciences Education, 20(2), fe3. https://doi.org/10.1187/cbe.20-12-0289

Tsai, Y.-H., Lin, C.-H., Hong, J.-C., & Tai, K.-H. (2018). The effects of metacognition on online learning interest and continuance to learn with MOOCs. Computers and Education, 121, 18–29. https://doi.org/10.1016/j.compedu.2018.02.011

Vally Essa, F., Andrews, G., Mendelowitz, B., Reed, Y., Fouche, I. (2023). Humanising online pedagogy through asynchronous discussion forums: An analysis of student dialogic interactions at a South African university. Online Learning Journal, 27(4), 508–529. https://doi.org/10.24059/olj.v27i4.3652

Veenman, M. V. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3–14. https://doi.org/10.1007/s11409-006-6893-0

Vrugt, A., & Oort, F. J. (2008). Metacognition, achievement goals, study strategies and academic achievement: Pathways to achievement. Metacognition and Learning, 3, 123–146. https://doi.org/10.1007/s11409-008-9022-4

Waters, J., & Gasson, S. (2015). Supporting metacognition in online, professional graduate courses. In 2015 48th Hawaii International Conference on System Sciences (pp. 91–100). IEEE. https://doi.org/10.1109/hicss.2015.21

Weinstein, C. E., Acee, T. W., & Jung, J. (2011). Self-regulation and learning strategies. New Directions for Teaching and Learning, 126, 45–53. https://doi.org/10.1002/tl.443

Winne, P. H., Hadwin, A. F., & Gress, C. (2010). The learning kit project: Software tools for supporting and researching regulation of collaborative learning. Computers in Human Behavior, 26, 787–793. https://doi.org/10.1016/j.chb.2007.09.009

Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self-reports about study tactics and achievement. Contemporary Educational Psychology, 27, 551–572.

https://doi.org/10.1016/s0361-476x(02)00006-1

Winne, P. H., Jamieson-Noel, D., & Muis, K. R. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement, Volume 12: New directions in measures and methods (pp. 121–155). Elsevier.

Wladis, C., Conway, K. M., & Hachey, A. C. (2015). The online STEM classroom—who succeeds? An exploration of the impact of ethnicity, gender, and non-traditional student characteristics in the community college context. Community College Review, 43(2), 142–164. https://doi.org/10.1177/0091552115571729

Xu, D., & Jaggars, S. (2011). The effectiveness of distance education across Virginia’s community colleges: Evidence from introductory college-level math and English courses. Educational Evaluation and Policy Analysis, 33, 360–377. https://doi.org/10.3102/0162373711413814

Xu, D., & Jaggars, S. (2014). Performance gaps between online and face-to-face courses: Differences across types of students and academic subject areas. Journal of Higher Education, 85, 633–659. https://doi.org/10.1353/jhe.2014.0028

Zhang, Z., Xu, Q., Koehler, A. A., & Newby, T. (2023). Comparing blended and online learners’ self-efficacy, self-regulation, and actual learning in the context of educational technology. Online Learning Journal, 27(4), 295–314. https://doi.org/10.24059/olj.v27i4.4039

Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. (2011). Enhancing self- reflection and mathematics achievement of at-risk urban technical college students. Psychological Test and Assessment Modeling, 53, 141–160.