Predicting Cognitive Presence in At-Scale Online Learning: MOOC and For-Credit Online Course Environments
DOI:
https://doi.org/10.24059/olj.v26i1.3060Keywords:
cognitive presence, discussion forums, machine learning, higher educationAbstract
In this study, we work towards a strategy to measure and enhance the quality of interactions in discussion forums at scale. We present a machine learning (ML) model which identifies the phase of cognitive presence exhibited by a student’s post and suggest future applications of such a model to help online students develop higher-order thinking. We collect discussion forum transcript data from two online courses: CS1301 (an introductory computer programming MOOC) offered by edX and CS6601 (a graduate course on artificial intelligence) which uses the Piazza online discussion tool. We manually code a random sample of students’ posts based on the Community of Inquiry coding scheme and explore trends in cognitive presence within and across the courses. We further use this coded data to analyze the relationship between students’ observed cognitive presence and course grades. In terms of testing and building an ML model, we use a Bidirectional Encoder Representations from Transformers model that uses a deep learning technique to train large text corpus and fine-tune the language model. Our results suggest that deeper cognitive engagement with course concepts, as expressed by higher cognitive presence, are associated with better learning outcomes for students in both course settings. Our ML approach achieves 92.5% accuracy on the classification task, motivating the use of ML for instructional interventions in online courses. We expect that our research study will not only contribute to extending the literature on cognitive presence but also have a beneficial impact on online instructors or curriculum developers in higher education.
References
Al-Shabandar, R., Hussain, A. J., Liatsis, P., & Keight, R. (2019). Detecting at-risk students with early interventions using machine learning techniques. IEEE Access, 7, 149464-149478. https://doi.org/10.1109/ACCESS.2019.2943351
Amemado, D., & Manca, S. (2017). Learning from decades of online distance education: MOOCs and the Community of Inquiry Framework. Journal of e-learning and Knowledge Society, 13(2). https://www.learntechlib.org/p/180225/
An, H., Shin, S, & Lim, K. (2009). The effects of different instructor facilitation approaches on students’ interactions during asynchronous online discussions. Computers & Education, 53(3), 749-760. https://doi.org/10.1016/j.compedu.2009.04.015
Arisoy, E., Sainath, T. N., Kingsbury, B., & Ramabhadran, B. (2012, June). Deep neural network language models. In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT (pp. 20-28). Association for Computational Linguistics. https://dl.acm.org/doi/abs/10.5555/2390940.2390943
Askeroth, J.H., & Richardson, J.C. (2019). Instructor perceptions of quality learning in MOOCs they teach. Online Learning, 23(4), 135-159. https://doi.org/doi:10.24059/olj.v23i4.2043
Baglione, S., & Nastanski, M. (2007). The superiority of online discussion: Faculty perceptions. The Quarterly Review of Distance Education, 8(2), 139-150.
Baran, E., & Correia, A. (2009). Student-led facilitation strategies in online discussions. Distance Education, 30(3), 339-361. https://doi.org/10.1080/01587910903236510
Bliuc, A., Ellis, R., Goodyear, P., & Piggott, L. (2009). Learning through face-to-face and online discussions: Associations between students’ conceptions, approaches and academic performance in political science. British Journal of Educational Technology, 41(3), 512-524. https://doi.org/10.1111/j.1467-8535.2009.00966.x
Chapman, D., Storberg-Walker, J., & Stone, S. (2008). Hitting reply: A qualitative study to understand student decisions to respond to online discussion postings. E-Learning and Digital Media, 5(1), 29-39. https://doi.org/10.2304%2Felea.2008.5.1.29
Chen, B., Chang, Y. H., Ouyang, F., & Zhou, W. (2018). Fostering student engagement in online discussion through social learning analytics. The Internet and Higher Education, 37, 21-30. https://doi.org/10.1016/j.iheduc.2017.12.002
Cheung, W., Hew, K., & Ng, C. (2008). Toward an understanding of why students contribute in asynchronous online discussions. Journal of Educational Computing Research, 38(1), 29-50. https://doi.org/10.2190%2FEC.38.1.b
Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219-243. https://doi.org/10.1080/00461520.2014.965823
Darabi, A., Arrastia, M., Nelson, D., Cornille, T. and Liang, X. (2011). Cognitive presence in asynchronous online learning: A comparison of four discussion strategies. Journal of Computer Assisted Learning, 27(3), 216-227. https://doi.org/10.1111/j.1365-2729.2010.00392.x
deNoyelles, A., Zydney, J., & Chen, B. (2014). Strategies for creating a community of inquiry through online asynchronous discussions. Journal of Online Learning and Teaching, 10(1), 153-165.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. https://arxiv.org/abs/1810.04805
Galikyan, I., Admiraal, W., & Kester, L. (2021). MOOC discussion forums: The interplay of the cognitive and the social. Computers & Education, 165, 104133. https://doi.org/10.1016/j.compedu.2021.104133
Gao, F., Zhang, T., & Franklin, T. (2013). Designing asynchronous online discussion environments: Recent progress and possible future directions. British Journal of Educational Technology, 44(3), 469-483. https://doi.org/10.1111/j.1467-8535.2012.01330
Garrison, D. R., & Akyol, Z. (2015). Toward the development of a metacognition construct for communities of inquiry. The Internet and Higher Education, 24, 66–71. https://doi.org/10.1016/j.iheduc.2014.10.001
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23. https://doi.org/10.1080/08923640109527071
Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of inquiry framework: A retrospective. The Internet and Higher Education, 13(1-2), 5-9. https://doi.org/10.1016/j.iheduc.2009.10.003
Guo, P., Saab, N., Wu, L., & Admiraal, W. (2021). The Community of Inquiry perspective on students' social presence, cognitive presence, and academic performance in online project-based learning. Journal of Computer Assisted Learning, 37(5), 1479–1493. https://doi. org/10.1111/jcal.12586
Hayati H., Idrissi M. K., & Bennani S. (2020) Automatic classification for cognitive engagement in online discussion forums: Text mining and machine learning approach. In Bittencourt, I., Cukurova, M., Muldner, K., Luckin, R., & Millán, E. (Eds.), Artificial Intelligence in Education. AIED 2020. Lecture Notes in Computer Science, vol 12164 (pp. 114-118). Springer, Cham. https://doi.org/10.1007/978-3-030-52240-7_21
Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001
Hew, K. F., Hu, X., Qiao, C., & Tang, Y. (2020). What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach. Computers & Education, 145, 103724. https://doi.org/10.1016/j.compedu.2019.103724
Hu, Y., Donald, C., Giacaman, N., & Zhu, Z. (2020, March). Towards automated analysis of cognitive presence in MOOC discussions: A manual classification study. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 135-140). https://doi.org/10.1145/3375462.3375473
Irish, I., Finkelberg, R., Nkemelu, D., Gujrania, S., Padiyath, A., Raman, S., Tailor, C., Arriaga, R., & Starner, T. (2020, August). PARQR: Automatic Post Suggestion in the Piazza Online Forum to Support Degree Seeking Online Masters Students. In Proceedings of the Seventh ACM Conference on Learning@ Scale (pp. 125-134). https://doi.org/10.1145/3386527.3405914
Kilis, S., & Yildirim, Z. (2019). Posting patterns of students’ social presence, cognitive presence, and teaching presence in online learning. Online Learning, 23(2), 179-195. https://doi.org/10.24059/olj.v23i2.1460
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016, April). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 15-24). https://doi.org/10.1145/2883851.2883950
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—A literature review. In A. Peña-Ayala (Ed.), Learning analytics: fundaments, applications, and trends (pp. 1-23), Springer. https://doi.org/10.1007/978-3-319-52977-6_1
Mazzolini, M., & Maddison, S. (2007). When to jump in: The role of the instructor in online discussion forums. Computers & Education., 49(2), 193-213. https://doi.org/10.1016/j.compedu.2005.06.011
Nanzi, D., Hamilton, M., & Harland, J. (2012). Evaluating the quality of interaction in asynchronous discussion forums in fully online courses. Distance Education, 33(1), 5-30. https://doi.org/10.1080/01587919.2012.667957
Neto, V., Rolim, V., Cavalcanti, A. P., Lins, R. D., Gasevic, D., & Ferreiramello, R. (2021). Automatic Content Analysis of Online Discussions for Cognitive Presence: A Study of the Generalizability across Educational Contexts. IEEE Transactions on Learning Technologies, 14(3), 299-312. https://doi.org/10.1109/TLT.2021.3083178
Pelánek, R. (2020, March). Learning analytics challenges: Trade-offs, methodology, scalability. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 554-558). https://doi.org/10.1145/3375462.3375463
Quintana, R. M., Pinto, J. D., & Tan, Y. (2021). What we learned when we compared discussion posts from one MOOC hosted on two platforms. Online Learning, 25(4), 7-24. https://doi.org/10.24059/olj.v25i4.2897
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8, 842-866. https://doi.org/10.1162/tacl_a_00349
Sadaf, A., Kim, S. Y., & Wang, Y. (2021). A comparison of cognitive presence, learning, satisfaction, and academic performance in case-based and non-case-based online discussions. American Journal of Distance Education, 35(3), 214-227. https://doi.org/10.1080/08923647.2021.1888667
Shea, P., & Bidjerano, T. (2009). Community of inquiry as a theoretical framework to foster “epistemic engagement” and “cognitive presence” in online education. Computers & Education, 52(3), 543- 553. https://doi.org/10.1016/j.compedu.2008.10.007
Sadaf, A., & Olesova, L. (2017). Enhancing cognitive presence in online case discussions with questions based on the practical inquiry model. American Journal of Distance Education, 31(1), 56-69. https://doi.org/10.1080/08923647.2017.1267525
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30-32.
Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98-110. https://doi.org/10.1016/j.chb.2018.07.027
Zhu, M., Bonk, C. J., & Sari, A.R. (2018). Instructor experiences designing MOOCs in higher education: Pedagogical, resource, and logistical considerations and challenges. Online Learning, 22(4), 203-241. https://doi.org/10.24059/olj.v22i4.1495
Downloads
Published
Issue
Section
License
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions