Quantifying social presence in online-based learning: a statistical and didactical analysis of indicators from social network analysis

Lisa-Maria Norz; Éva Kaczkó; Michael Netzer; Elske Ammenwerth

doi:10.24059/olj.v28i4.4134

Authors

Lisa-Maria Norz Institute of Medical Informatics, UMIT TIROL – Private University for Health Sciences and Health Technology, 6060 Hall i. Tirol, Austria https://orcid.org/0000-0003-2711-3227
Éva Kaczkó Department of Organization and Learning (Business Education), University of Innsbruck, 6020 Innsbruck, Austria
Michael Netzer Institute of Medical Informatics, UMIT TIROL – Private University for Health Sciences and Health Technology, 6060 Hall i. Tirol, Austria https://orcid.org/0000-0001-7536-9257
Elske Ammenwerth Institute of Medical Informatics, UMIT TIROL – Private University for Health Sciences and Health Technology, 6060 Hall i. Tirol, Austria https://orcid.org/0000-0002-3244-6918

DOI:

https://doi.org/10.24059/olj.v28i4.4134

Keywords:

Community of Inquiry, social presence, social network analysis, higher education, learning communities, distance education

Abstract

Social presence is one key factor for successful learning in socio-constructivist learning theories. Teachers need easily interpretable and didactically relevant information to monitor social presence and to intervene if required while a course is running. Social network analysis is a promising method to provide this information, yet it is unclear which indicators are most appropriate for this aim. This study aimed to identify and evaluate indicators derived from social network analysis, which could quantify social presence and provide didactically relevant information to teachers. 3,546 postings from different modules and study groups (n=49) of an online-based Master course were manually coded. Egocentric measures were calculated from social network analysis. Path models were developed to analyze which indicators from social network analysis were statistically significant measures of social presence. The course of these indicators was analyzed across the modules and statistically evaluated using the Friedman test. Out of 13 possible indicators, six indicators were found to be statistically and didactically relevant (Ties, Density, Efficiency, nBroke, Out- & In-closeness). These indicators showed high regression weights in the path models and the development across multiple time points was identified to be statistically significant (p < .001). Literature analysis and didactical considerations showed that these indicators might provide an appropriate and real-time overview of students' social presence. We identified indicators that allow to measure social presence and provide didactically relevant information for teachers in real time by using log data from learning management systems. Further research will now focus on implementing the findings in a teacher dashboard.

References

Akyol, Z., & Garrison, D. R. (2008). The development of a community of inquiry over time in an online course: Understanding the progression and integration of social, cognitive and teaching presence. Online Learning, 12(3). https://doi.org/10.24059/olj.v12i3.72

Ammenwerth, E., Hackl, W. O., Felderer, M., & Hoerbst, A. (2017). Developing and evaluating collaborative online-based instructional designs in health information management. Studies in Health Technology and Informatics, 243, 8–12. https://doi.org/10.3233/978-1-61499-808-2-8

André, M., Ferreira, R., Nascimento, A., Lins, R., & Gasevic, D. (2021). Toward automatic classification of online discussion messages for social presence. IEEE Transactions on Learning Technologies, 14(6), 802–816. https://doi.org/10.1109/TLT.2022.3150663

Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. Internet and Higher Education, 11(3–4), 133–136. https://doi.org/10.1016/j.iheduc.2008.06.003

Arbuckle, J. L. (2014). Amos (Version 27.0). IBM SPSS.

Boston, W., Ice, P., Díaz, S. R., Richardson, J., Gibson, A. M., & Swan, K. (2019). An exploration of the relationship between indicators of the community of inquiry framework and retention in online programs. Journal of Asynchronous Learning Networks, 14(1).

Borgatti, S., P., & Halgin, D. S. (2011). On network theory. Organization Science, 22(5), 1168–1181. https://doi.org/10.1287/orsc.1100.0641

Borgatti, S. P., Everett, M. G., & Freeman, L. C. (2020). Ucinet for Windows: Software for social network analysis. Analytic Technologies.

Carolan, B. V. (2014). Social Network Analysis and Education: Theory, Methods and Applications. SAGE Publications, Inc.

Caskurlu, S. (2018). Confirming the subdimensions of teaching, social, and cognitive presences: A construct validity study. Internet and Higher Education, 39, 1–12. https://doi.org/10.1016/j.iheduc.2018.05.002

Chen, B., & Poquet, O. (2022). Networks in learning analytics: Where theory, methodology, and practice intersect. Journal of Learning Analytics, 9(1), 1–12. https://doi.org/10.18608/jla.2022.7697

Chou, T.-L., Wu, J.-J., & Tsai, C.-C. (2019). research trends and features of critical thinking studies in e-learning environments: A review. Journal of Educational Computing Research, 57(4), 1038–1077. https://doi.org/10.1177/0735633118774350

Dado, M., & Bodemer, D. (2017). A review of methodological applications of social network analysis in computer-supported collaborative learning. Educational Research Review, 22, 159–180. 10.1016/j.edurev.2017.08.005.

De Laat, M., Lally, V., Lipponen, L., & Simons, R. J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: A role for Social Network Analysis. International Journal of Computer-Supported Collaborative Learning, 2(1), 87–103. https://doi.org/10.1007/s11412-007-9006-4

Dumford, A. D., & Miller, A. L. (2018). Online learning in higher education: exploring advantages and disadvantages for engagement. Journal of Computing in Higher Education, 30(3), 452–465. https://doi.org/10.1007/s12528-018-9179-z

European Commission, Eurostat (2022). Interest in online education grows in the EU. https://ec.europa.eu/eurostat/web/digital-economy-and-society/data/database

Ferreira, M., Rolim, V., Mello, R. F., Lins, R. D., Chen, G., & Gaševic, D. (2020). Towards automatic content analysis of social presence in transcripts of online discussions. ACM International Conference Proceeding Series, 141–150. https://doi.org/10.1145/3375462.3375495

Fiock, H. S. (2020). Designing a community of inquiry in online courses. International Review of Research in Open and Distance Learning, 21(1), 112–133. https://doi.org/10.19173/irrodl.v20i5.4383

Garrison, D. R. (2013). Theoretical foundations and epistemological insights of the community of inquiry. In Z. Akyol & D. R. Garrison (Eds.), Educational communities of inquiry: theoretical framework, research, and practice (pp. 1–11). IGI Global.

Garrison, D. R. (2017). E-Learning in the 21st century: A community of inquiry framework for research and practice (3rd ed.). Routledge.

Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23. https://doi.org/10.1080/08923640109527071

Greenhow, C., Graham, C. R., & Koehler, M. J. (2022). Foundations of online learning: Challenges and opportunities. Educational Psychologist, 57(3), 131–147. https://doi.org/10.1080/00461520.2022.2090364

Henrie, C. R., Bodily, R., Larsen, R., & Graham, C. R. (2018). Exploring the potential of LMS log data as a proxy measure of student engagement. Journal of Computing in Higher Education, 30(2), 344–362. https://doi.org/10.1007/s12528-017-9161-1

Huang, J., Dasgupta, A., Ghosh, A., Manning, J., & Sanders, M. (2014). Superposter behavior in MOOC forums. In Proceedings of the First ACM Conference on Learning @ Scale (L@S). https://doi.org/10.1145/2556325.2566249

IBM Corp. (2020).BM SPSS Statistics for Windows. Version 27.0. IBM Corp.

Jan, S. K., & Vlachopoulos, P. (2019). Social network analysis: A framework for identifying communities in higher education online learning. Technology, Knowledge and Learning, 24(4), 621–639. https://doi.org/10.1007/s10758-018-9375-y

Jan, S. K., Vlachopoulos, P., & Parsell, M. (2019). Social network analysis and learning communities in higher education online learning: A systematic literature review. Online Learning Journal, 23(1), 249–264. https://doi.org/10.24059/olj.v23i1.1398

Jimoyiannis, A., Tsiotakis, P., & Roussinos, D. (2012). Blogs in higher education: Analyzing students' participation and presence in a community of blogging. Proceedings of the IADIS International Conference E-Learning 2012, June 2016, 228–235.

Jimoyiannis, A., & Tsiotakis, P. (2017). Beyond students' perceptions: investigating learning presence in an educational bloggings community. Journal of Applied Research in Higher Education, 9(1), 129–146. https://doi.org/10.1108/JAHRE-06-2015-0046

Jo, I., Park, Y., & Lee, H. (2017). Three interaction patterns on asynchronous online discussion behaviours: A methodological comparison. Journal of Computer Assisted Learning, 33(2), 106–122. https://doi.org/10.1111/jcal.12168

Kaczkó, É., Norz, L.-M., Dornauer, V., & Ammenwerth, E. (2022). Community-of-Inquiry-rahmenwerk: deutschsprachiger kodierungsleitfaden zur inhaltsanalyse der kognitiven und sozialen präsenz. https://doi.org/10.13140/RG.2.2.15719.93605

Kim, G., & Gurvitch, R. (2020). Online education research adopting the community of inquiry framework: A systematic review. Quest, 72(4), 395–409. https://doi.org/10.1080/00336297.2020.1761843

Knight, S., & Buckingham, S. S. (2017). Theory and learning analytics. In Handbook of Learning Analytics. Society for Learning Analytics Research (SoLAR) (pp. 17–22).

Kovanovic, V., Joksimovic, S., Gasevic, D., & Hatala, M. (2014). What is the source of social capital? the association between social network position and social presence in Communities of Inquiry. CEUR Workshop Proceedings, 1183(July), 21–28.

Lim, J. (2023). Exploring the relationships between interaction measures and learning outcomes through social network analysis: the mediating role of social presence. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00384-8

Luo, L., Liu, Q., Zhang, N., & Xu, B. (2019). Investigating interactive behaviors in online community of inquiry using social network analysis. In 2019 International Joint Conference on Information, Media and Engineering (IJCIME). IEEE. https://doi.org/10.1109/ijcime49369.2019.00034

Lowenthal, P. R. (2009). Social presence. In Encyclopedia of distance learning (2nd ed., pp. 1900–1906). https://www.researchgate.net/publication/265375995

Lowenthal, P. R. (2012). Social presence: What is it? How do we measure it? (Doctoral dissertation). University of Colorado Denver, Denver, Colorado.

Maloney, S., Axelsen, M., Galligan, L., Turner, J., Redmond, P., Brown, A., Basson, M., & Lawrence, J. (2022). Using LMS log data to explore student engagement with coursework videos. Online Learning, 26(4). https://doi.org/10.24059/olj.v26i4.2998

Martin, F., & Borup, J. (2022). Online learner engagement: Conceptual definitions, research themes, and supportive practices. Educational Psychologist, 57(3), 162–177. https://doi.org/10.1080/00461520.2022.2089147

National Center for Education Statistics (2021). Number and percentage of students enrolled in degree-granting postsecondary institutions, by distance education participation, location of student, level of enrollment, and control and level of institution: Fall 2019 and fall 2020. Institute of Education Sciences, National Center for Education Statistics. https://nces.ed.gov/programs/digest/d21/tables/dt21_311.15.asp

Norz, L. M., Dornauer, V., Hackl, W. O., & Ammenwerth, E. (2023). Measuring social presence in online-based learning: An exploratory path analysis using log data and social network analysis. Internet and Higher Education, 56. https://doi.org/10.1016/j.iheduc.2022.100894

Papanikolaou, K., Tzelepi, M., Moundridou, M., & Petroulis, I. (2020). Employing social network analysis to enhance community learning. In V. Kumar & C. Troussas (Eds.), Intelligent tutoring systems. ITS 2020. Lecture notes in computer science, vol. 12149. Springer. https://doi.org/10.1007/978-3-030-49663-0_41

Prinsloo, P., & Slade, S. (2017). Ethics and learning analytics: Charting the (Un)Charted. In C. Lang, G. Siemens, A. Wise, & D. Gašević, (Eds.), Handbook of Learning Analytics (pp. 49–57). SOLAR.

R Core Team. (2014). A language and environment for statistical computing. A Language and Environment for Statistical Cimputing. R Foundation for Statistical Computing. http://www.r-project.org/

Richardson, J. C., Arbaugh, J. B., Cleveland-Innes, M., Ice, P., Swan, K. P., & Garrison, D. R. (2012). Using the community of inquiry framework to inform effective instructional design. In L. Moller, & J. B. Huett (Eds.), The next generation of distance education: Unconstrained learning (pp. 97–125). Springer.

Richardson, J. C., Maeda, Y., Lv, J., & Caskurlu, S. (2017). Social presence in relation to students' satisfaction and learning in the online environment: A meta-analysis. Computers in Human Behavior, 71, 402–417. https://doi.org/10.1016/j.chb.2017.02.001

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), 50–71.

Ruthotto, I., Kreth, Q., Steven, J., Tively, C., & Melkers, J. (2020). Lurking and participation in the virtual classroom: The effects of gender, race, and age among graduate students in computer science, Computers & Education, 151. https://doi.org/10.1016/j.compedu.2020.103854.

Saqr, M., Viberg, O., & Vartiainen, H. (2020). Capturing the participation and social dimensions of computer-supported collaborative learning through social network analysis: which method and measures matter? International Journal of Computer-Supported Collaborative Learning, 15(2), 227–248. https://doi.org/10.1007/s11412-020-09322-6

Satar, H. M., & Akcan, S. (2018). Pre-service EFL teachers' online participation, interaction, and social presence. Language Learning and Technology, 22(1), 157–184.

Shea, P., & Bidjerano, T. (2010). Learning presence: Towards a theory of self-efficacy, self-regulation, and the development of a communities of inquiry in online and blended learning environments. Computers & Education, 55(4), 1721–1731. https://doi.org/10.1016/j.compedu.2010.07.017

Shea, P., Hayes, S., Vickers, J., Gozza-Cohen, M., Uzuner, S., Mehta, R., Valchova, A., & Rangan, P. (2010). A re-examination of the community of inquiry framework: Social network and content analysis. Internet and Higher Education, 13(1–2), 10–21. https://doi.org/10.1016/j.iheduc.2009.11.002

Shea, P., Hayes, S., Smith, S., U., Vickers, J., Bidjerano, T., Gozza-Cohen, M., Jian, S.-B., Pickett, A., M., Wilde, J., & Tseng, C.-H. (2013). Online learner self-regulation: Learning presence viewed through quantitative content- and social network analysis. International Review of Research in Open and Distributed Learning, 14(3), 427–461. https://doi.org/10.19173/irrodl.v14i3.1466

Shea, P., Hayes, S., Uzuner-Smith, S., Gozza-Cohen, M., Vickers, J., & Bidjerano, T. (2014). Reconceptualizing the community of inquiry framework: An exploratory analysis. Internet and Higher Education, 23, 9–17. https://doi.org/10.1016/j.iheduc.2014.05.002

Shelton, B. E., Hung, J. L., & Lowenthal, P. R. (2017). Predicting student success by modeling student interaction in asynchronous online courses. Distance Education, 38(1), 59–69. https://doi.org/10.1080/01587919.2017.1299562

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/ 0002764213498851

Short, J., Williams, E., & Christie, B. (1976). The Social Psychology of Telecommunications. Wiley.

So, H.-J., & Brush, T. A. (2008). Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: Relationships and critical factors. Computers & Education, 51(1), 318–336. https://doi.org/10.1016/j.compedu.2007.05.009.

Stenbom, S. (2018). A systematic review of the community of inquiry survey. Internet and Higher Education, 39, 22–32. https://doi.org/10.1016/j.iheduc.2018.06.001

Swan, K., Shea, P., Richardson, J., Ice, P., Garrison, D. R., Cleveland-Innes, M., & Arbaugh, J. B. (2008). Validating a measurement tool of presence in online communities of inquiry. E-Mentor, 2(24), 1–12.

Tirado, R., Hernando, Á., & Aguaded, J. I. (2015). The effect of centralization and cohesion on the social construction of knowledge in discussion forums. Interactive Learning Environments, 23(3), 293–316. https://doi.org/10.1080/10494820.2012.7454374

Tirado-Morueta, R., Maraver, L., Pablo, P., R., Amor, M., & Hernando, Á. (2019). Exploring social network structure patterns suitable to the community of inquiry model moderated by the task. Journal of Educational Computing Research, 58. https://doi.org/10.1177/0735633119845695

VERBI Software. (2019). MAXQDA 2020. maxqda.com

Wang, X., Yang, D., Wen, M., Koedinger, K., & Ros´e, C. P. (2015). Investigating how student's cognitive behavior in MOOC discussion forums affect learning gains. In Proceedings of the 8th international conference on educational data mining (EDM'15) (pp. 226–233).

Wicks, D., Craft, B. B., Lee, D. D., Lumpe, A., Henrikson, R., Baliram, N., Bian, X., Mehlberg, S., & Wicks, K. (2015). An evaluation of low versus high collaboration in online learning. Online Learning Journal, 19(4). https://doi.org/10.24059/olj.v19i4.552

Ye, D., & Pennisi, S. (2022). Analyzing interactions in online discussions through social network analysis. Journal of Computer Assisted Learning, 1–13. https://doi.org/10.1111/jcal.12648

Yücel, Ü. A. I., & Usluel, Y. K. (2016). Knowledge building and the quantity, content and quality of the interaction and participation of students in an online collaborative learning environment. Computers and Education, 97, 31–48. https://doi.org/10.1016/j.compedu.2016.02.015

Zou, W., Pan, Z., Li, C. & Liu, M. (2021). Does Social Presence Play a Role in Learners’ Positions in MOOC Learner Network? A Machine Learning Approach to Analyze Social Presence in Discussion Forums. In Communications in computer and information science (S. 248–264). https://doi.org/10.1007/978-3-030-67788-6_17