Predicting Cognitive Presence in At-Scale Online Learning: MOOC and For-Credit Online Course Environments

Jeonghyun Lee, Farahnaz Soleimani, India Irish, John Hosmer, IV, Meryem Yilmaz Soylu, Roy Finkelberg, Saurabh Chatterjee

Abstract


In this study, we work towards a strategy to measure and enhance the quality of interactions in discussion forums at scale. We present a machine learning (ML) model which identifies the phase of cognitive presence exhibited by a student’s post and suggest future applications of such a model to help online students develop higher-order thinking. We collect discussion forum transcript data from two online courses: CS1301 (an introductory computer programming MOOC) offered by edX and CS6601 (a graduate course on artificial intelligence) which uses the Piazza online discussion tool. We manually code a random sample of students’ posts based on the Community of Inquiry coding scheme and explore trends in cognitive presence within and across the courses. We further use this coded data to analyze the relationship between students’ observed cognitive presence and course grades. In terms of testing and building an ML model, we use a Bidirectional Encoder Representations from Transformers model that uses a deep learning technique to train large text corpus and fine-tune the language model. Our results suggest that deeper cognitive engagement with course concepts, as expressed by higher cognitive presence, are associated with better learning outcomes for students in both course settings. Our ML approach achieves 92.5% accuracy on the classification task, motivating the use of ML for instructional interventions in online courses. We expect that our research study will not only contribute to extending the literature on cognitive presence but also have a beneficial impact on online instructors or curriculum developers in higher education.


Keywords


cognitive presence; discussion forums; machine learning; higher education

Full Text:

PDF

References


Al-Shabandar, R., Hussain, A. J., Liatsis, P., & Keight, R. (2019). Detecting at-risk students with early interventions using machine learning techniques. IEEE Access, 7, 149464-149478. https://doi.org/10.1109/ACCESS.2019.2943351

Amemado, D., & Manca, S. (2017). Learning from decades of online distance education: MOOCs and the Community of Inquiry Framework. Journal of e-learning and Knowledge Society, 13(2). https://www.learntechlib.org/p/180225/

An, H., Shin, S, & Lim, K. (2009). The effects of different instructor facilitation approaches on students’ interactions during asynchronous online discussions. Computers & Education, 53(3), 749-760. https://doi.org/10.1016/j.compedu.2009.04.015

Arisoy, E., Sainath, T. N., Kingsbury, B., & Ramabhadran, B. (2012, June). Deep neural network language models. In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT (pp. 20-28). Association for Computational Linguistics. https://dl.acm.org/doi/abs/10.5555/2390940.2390943

Askeroth, J.H., & Richardson, J.C. (2019). Instructor perceptions of quality learning in MOOCs they teach. Online Learning, 23(4), 135-159. https://doi.org/doi:10.24059/olj.v23i4.2043

Baglione, S., & Nastanski, M. (2007). The superiority of online discussion: Faculty perceptions. The Quarterly Review of Distance Education, 8(2), 139-150.

Baran, E., & Correia, A. (2009). Student-led facilitation strategies in online discussions. Distance Education, 30(3), 339-361. https://doi.org/10.1080/01587910903236510

Bliuc, A., Ellis, R., Goodyear, P., & Piggott, L. (2009). Learning through face-to-face and online discussions: Associations between students’ conceptions, approaches and academic performance in political science. British Journal of Educational Technology, 41(3), 512-524. https://doi.org/10.1111/j.1467-8535.2009.00966.x

Chapman, D., Storberg-Walker, J., & Stone, S. (2008). Hitting reply: A qualitative study to understand student decisions to respond to online discussion postings. E-Learning and Digital Media, 5(1), 29-39. https://doi.org/10.2304%2Felea.2008.5.1.29

Chen, B., Chang, Y. H., Ouyang, F., & Zhou, W. (2018). Fostering student engagement in online discussion through social learning analytics. The Internet and Higher Education, 37, 21-30. https://doi.org/10.1016/j.iheduc.2017.12.002

Cheung, W., Hew, K., & Ng, C. (2008). Toward an understanding of why students contribute in asynchronous online discussions. Journal of Educational Computing Research, 38(1), 29-50. https://doi.org/10.2190%2FEC.38.1.b

Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219-243. https://doi.org/10.1080/00461520.2014.965823

Darabi, A., Arrastia, M., Nelson, D., Cornille, T. and Liang, X. (2011). Cognitive presence in asynchronous online learning: A comparison of four discussion strategies. Journal of Computer Assisted Learning, 27(3), 216-227. https://doi.org/10.1111/j.1365-2729.2010.00392.x

deNoyelles, A., Zydney, J., & Chen, B. (2014). Strategies for creating a community of inquiry through online asynchronous discussions. Journal of Online Learning and Teaching, 10(1), 153-165.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. https://arxiv.org/abs/1810.04805

Galikyan, I., Admiraal, W., & Kester, L. (2021). MOOC discussion forums: The interplay of the cognitive and the social. Computers & Education, 165, 104133. https://doi.org/10.1016/j.compedu.2021.104133

Gao, F., Zhang, T., & Franklin, T. (2013). Designing asynchronous online discussion environments: Recent progress and possible future directions. British Journal of Educational Technology, 44(3), 469-483. https://doi.org/10.1111/j.1467-8535.2012.01330

Garrison, D. R., & Akyol, Z. (2015). Toward the development of a metacognition construct for communities of inquiry. The Internet and Higher Education, 24, 66–71. https://doi.org/10.1016/j.iheduc.2014.10.001

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23. https://doi.org/10.1080/08923640109527071

Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of inquiry framework: A retrospective. The Internet and Higher Education, 13(1-2), 5-9. https://doi.org/10.1016/j.iheduc.2009.10.003

Guo, P., Saab, N., Wu, L., & Admiraal, W. (2021). The Community of Inquiry perspective on students' social presence, cognitive presence, and academic performance in online project-based learning. Journal of Computer Assisted Learning, 37(5), 1479–1493. https://doi. org/10.1111/jcal.12586

Hayati H., Idrissi M. K., & Bennani S. (2020) Automatic classification for cognitive engagement in online discussion forums: Text mining and machine learning approach. In Bittencourt, I., Cukurova, M., Muldner, K., Luckin, R., & Millán, E. (Eds.), Artificial Intelligence in Education. AIED 2020. Lecture Notes in Computer Science, vol 12164 (pp. 114-118). Springer, Cham. https://doi.org/10.1007/978-3-030-52240-7_21

Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001

Hew, K. F., Hu, X., Qiao, C., & Tang, Y. (2020). What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach. Computers & Education, 145, 103724. https://doi.org/10.1016/j.compedu.2019.103724

Hu, Y., Donald, C., Giacaman, N., & Zhu, Z. (2020, March). Towards automated analysis of cognitive presence in MOOC discussions: A manual classification study. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 135-140). https://doi.org/10.1145/3375462.3375473

Irish, I., Finkelberg, R., Nkemelu, D., Gujrania, S., Padiyath, A., Raman, S., Tailor, C., Arriaga, R., & Starner, T. (2020, August). PARQR: Automatic Post Suggestion in the Piazza Online Forum to Support Degree Seeking Online Masters Students. In Proceedings of the Seventh ACM Conference on Learning@ Scale (pp. 125-134). https://doi.org/10.1145/3386527.3405914

Kilis, S., & Yildirim, Z. (2019). Posting patterns of students’ social presence, cognitive presence, and teaching presence in online learning. Online Learning, 23(2), 179-195. https://doi.org/10.24059/olj.v23i2.1460

Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016, April). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 15-24). https://doi.org/10.1145/2883851.2883950

Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—A literature review. In A. Peña-Ayala (Ed.), Learning analytics: fundaments, applications, and trends (pp. 1-23), Springer. https://doi.org/10.1007/978-3-319-52977-6_1

Mazzolini, M., & Maddison, S. (2007). When to jump in: The role of the instructor in online discussion forums. Computers & Education., 49(2), 193-213. https://doi.org/10.1016/j.compedu.2005.06.011

Nanzi, D., Hamilton, M., & Harland, J. (2012). Evaluating the quality of interaction in asynchronous discussion forums in fully online courses. Distance Education, 33(1), 5-30. https://doi.org/10.1080/01587919.2012.667957

Neto, V., Rolim, V., Cavalcanti, A. P., Lins, R. D., Gasevic, D., & Ferreiramello, R. (2021). Automatic Content Analysis of Online Discussions for Cognitive Presence: A Study of the Generalizability across Educational Contexts. IEEE Transactions on Learning Technologies, 14(3), 299-312. https://doi.org/10.1109/TLT.2021.3083178

Pelánek, R. (2020, March). Learning analytics challenges: Trade-offs, methodology, scalability. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 554-558). https://doi.org/10.1145/3375462.3375463

Quintana, R. M., Pinto, J. D., & Tan, Y. (2021). What we learned when we compared discussion posts from one MOOC hosted on two platforms. Online Learning, 25(4), 7-24. https://doi.org/10.24059/olj.v25i4.2897

Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8, 842-866. https://doi.org/10.1162/tacl_a_00349

Sadaf, A., Kim, S. Y., & Wang, Y. (2021). A comparison of cognitive presence, learning, satisfaction, and academic performance in case-based and non-case-based online discussions. American Journal of Distance Education, 35(3), 214-227. https://doi.org/10.1080/08923647.2021.1888667

Shea, P., & Bidjerano, T. (2009). Community of inquiry as a theoretical framework to foster “epistemic engagement” and “cognitive presence” in online education. Computers & Education, 52(3), 543- 553. https://doi.org/10.1016/j.compedu.2008.10.007

Sadaf, A., & Olesova, L. (2017). Enhancing cognitive presence in online case discussions with questions based on the practical inquiry model. American Journal of Distance Education, 31(1), 56-69. https://doi.org/10.1080/08923647.2017.1267525

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30-32.

Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98-110. https://doi.org/10.1016/j.chb.2018.07.027

Zhu, M., Bonk, C. J., & Sari, A.R. (2018). Instructor experiences designing MOOCs in higher education: Pedagogical, resource, and logistical considerations and challenges. Online Learning, 22(4), 203-241. https://doi.org/10.24059/olj.v22i4.1495




DOI: http://dx.doi.org/10.24059/olj.v26i1.3060



Copyright (c) 2022 Jeonghyun Lee, Farahnaz Soleimani, India Irish, John Hosmer, IV, Meryem Yilmaz Soylu, Roy Finkelberg, Saurabh Chatterjee

License URL: https://creativecommons.org/licenses/by/4.0/