Bridging Theory and Measurement of Student Engagement: A Practical Approach

Barbara Means; Julie Neisler

doi:10.24059/olj.v27i4.4034

Authors

Dr. Barbara Means Digital Promise
Dr. Julie Neisler Digital Promise

DOI:

https://doi.org/10.24059/olj.v27i4.4034

Abstract

Learner engagement is well-established as critical for learning online. Professional development for online instructors emphasizes techniques for engaging students, and learning technology products tout features intended to promote engagement (e.g., adaptive content, video, gamification). But the influence of particular instructor practices and of particular learning technology features on theory-based aspects of student engagement is infrequently tested empirically, and even more rarely with Black, Latine, and low-income students, who are more likely to face barriers to learning online. This paper first provides a research-based theoretical model of affective engagement developed in conjunction with ongoing studies of blended learning implementations of courseware designed to enhance learning and engagement among historically and systemically marginalized students. Next, the paper describes development of survey-based measures of four components of affective engagement and the use of responses from over 850 students in introductory statistics courses to evaluate the reliability and factor structure of those measures. We conclude with implications for use of the engagement measures in future improvement-oriented research and practice.

References

Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44(5), 427–445.

Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289. doi.org/10.3102/0034654309333844

Borup, J., Graham, C. R., West, R. E., Archambault, L., & Spring, K. J. (2020). Academic Communities of Engagement: An expansive lens for examining support structures in blended and online learning. Educational Technology Research and Development, 68(2), 807–832. https://doi.org/10.1007/s11423-020-09744-x

Bowden, J. L. H., Tickle, L., & Naumann, K. (2021). The four pillars of tertiary student engagement and success: a holistic measurement approach. Studies in Higher Education, 46(6), 1207–1224.

Carroll, J. (1963). A model of school learning. Teachers College Record, 64, 723–733.

Center for Innovation in Teaching and Learning. (2020, May 19). Building and maintaining student engagement in an online environment. University of Illinois Urbana-Champaign. https://citl.illinois.edu/citl-101/teaching-learning/resources/teaching-across-modalities/teaching-tips-articles/teaching-tips/2020/05/19/building-and-maintaining-student-engagement

Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243.

Cho, M. H., Park, S. W., & Lee, S. E. (2021). Student characteristics and learning and teaching factors predicting affective and motivational outcomes in flipped college classrooms. Studies in Higher Education, 46(3), 509–522.

Cohen, D. K., Raudenbush, S., & Ball, D. (2003). Resources, instruction, and research. Educational Evaluation and Policy, 25(2), 1–24.

D’Agostino, S. (2022, September 13). The needs and preferences of fully online learners. Inside Higher Education. https://www.insidehighered.com/news/2022/09/14/needs–and-preferences-fully-online-learners-survey

Dahl, B. (2015, November 12). Seven tips for increasing student engagement in online courses. D2L. https://www.d2l.com/blog/7-tips-for-increasing-student-engagement-in-online-courses/Digital Promise. (2022, October 25). Equity and Digital Learning Student Survey. Every Learner Everywhere. https://www.everylearnereverywhere.org/resources/equity-indigital-learning-student-survey/

Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-value theory: A developmental, social cognitive, and sociocultural perspective on motivation. Contemporary Educational Psychology, 61, Article 101859.

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59-109.

Fuller, K. A., et al. (2018). Development of a self-report instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem. PLoS ONE, 13(10).

Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105.

Greenhow, C., Graham, C. R., & Koehler, M. J. (2022). Foundations of online learning: Challenges and opportunities, Educational Psychologist, 57(3), 131–147.

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis. 6th edition. Pearson.

Halverson, L. R., & Graham, C. R. (2019). Learner engagement in blended learning environments: A conceptual framework. Online Learning, 23(2), 145–178. doi.org/10.24059/olj.v23i2.1481

Hammond, Z. L. (2014). Culturally responsive teaching and the brain. Corwin Press.

Handelsman, M. M., Briggs, W. L., Sullivan, N., & Towler, A. (2005). A measure of college student course engagement. Journal of Educational Research, 98, 184–191.

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118

Hulleman, C., & Harackiewicz, J. M. (2020). The utility-value intervention. In G. M. Walton & A. J. Crum (Eds.,) Handbook of Wise Interventions, 100–125. Routledge.

Hughes, R. & Pace, C.R. (2003). Using NSSE to study student retention and withdrawal. Assessment Update, 15(4), 1–2.

Ingram, D. (2012). College students’ sense of belonging: Dimensions and correlates. [Doctoral dissertation, Stanford University].

Ketonen E., at al. (2016). Am I in the right place? Academic engagement and study success during the first year at university. Learning and Individual Differences, 51, 141–148. doi:10.1016/j.lindif.2016.08.017

Kinsella, M., et al. (2022). Supporting students’ transition into higher education: Motivation enhancement strategies. Access: Contemporary Issues in Education, 42(1), 3–20.

Kirby, L. A. J., & Thomas, C. L. (2022) High-impact teaching practices foster a greater sense of belonging in the college classroom, Journal of Further and Higher Education, 46(3), 368–381. doi: 10.1080/0309877X.2021.1950659

Kucuk, S., & Richardson, J. C. (2019). A structural equation model of predictors of online learners' engagement and satisfaction. Online Learning, 23(2), 196–216.

Kuh, G. D. (2001). Assessing what really matters to student learning: Inside the National Survey of Student Engagement. Change: The Magazine of Higher Learning, 33(3) 10–17.

Ladson-Billings, G. J. (2023, May). Consequential issues for educators and education. Presidential panelist remarks at the annual meeting of the American Educational Research Association, Chicago.

Lave, J., & Packer, M. (2008). Towards a social ontogeny of learning. In K. Nielsen et al. (Eds.), A qualitative stance, pp. 17–46. Aarhus Universitetsforlag.

Lin, S-H., & Huang, Y-C. (2018). Assessing college student engagement: Development and validation of the Student Course Engagement Scale. Journal of Psychoeducational Assessment, 36(7), 694–708.

Mandernach, B. J. (2015). Assessment of student engagement in higher education: A synthesis of literature and assessment tools. International Journal of Learning, Teaching and Educational Research, 12(2).

Martin, F. & Bolliger, D. U. (2018). Engagement matters: Student perceptions on the importance of engagement strategies in the online learning environment. Online Learning, 22(1), 205–222. doi:10.24059/olj.v22i1.1092

Martin, F., & Borup, J. (2022). Online learner engagement: Conceptual definitions, research themes, and supportive practices. Educational Psychologist, 57(3), 162–177.

McCoach, D. B., Gable, R. K., & Madura, J. P. (2013). Instrument development in the affective domain (Vol. 10, pp. 978–971). Springer.

Means, B. (2022). Bringing together insights from educational psychology and educational technology: Time for a new approach. Educational Psychologist, 57(3), 226–230.

Means, B., & Neisler, J. (2020). Unmasking inequality: STEM course experiences during the COVID-19 pandemic. Digital Promise Global.

Means, B., & Neisler, J. (2021). Teaching and learning in the time of COVID: The student perspective. Online Learning Journal, 25(1), 8–27.

Means, B., Toyama, Y., Murphy, R., & Bakia, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115. doi:10.1177/016146811311500307.

Meriwether, B. K. X. (2019). The effect of stereotype threat in gateway courses on academic major choice of black students in predominantly white institutions. Unpublished doctoral dissertation, Oakland University.

National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners, contexts, and cultures. National Academies Press.

Ogunyemi, D., Clare, C., Astudillo, Y. M., Marseille, M., Manu, E., & Kim, S. (2020). Microaggressions in the learning environment: A systematic review. Journal of Diversity in Higher Education, 13(2), 97–119. doi.org/10.1037/dhe0000107

Osborne, J., Simon, S., & Collins, S. (2003). Attitudes toward science: A review of the literature and its implications. International Journal of Science Education, 25(9), 1049–1079.

Ouimet, J. A., & Smallwood, R.A. (2005). Assessment measures: CLASSE—The Class-Level Survey of Student Engagement. Assessment Update, 17(6).

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning.

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.

Ramirez, G., Shaw, S. T., & Maloney, E. A. (2018). Math anxiety: Past research, promising interventions, and a new interpretation framework. Educational Psychologist, 53(3), 145–164.

Renninger, K. A., & Hidi, S. (2011). Revisiting the conceptualization, measurement, and generation of interest. Educational Psychologist, 46(3), 168–184.

Renninger, K. A., & Hidi, S. (2020). To level the playing field, develop interest. Policy Insights from the Behavioral and Brain Sciences, 7(1), 10–18.

Renninger, K. A., and S. Hidi. 2022. “Interest: A Unique Affective and Cognitive Motivational Variable That Develops.” Advances in Motivation Science 9: 179–239. doi:10.1016/bs.adms.2021.12.004.

Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. doi.org/10.18637/jss.v048.i02.

Schaeffer, C. E., & Konetes, G. D. (2010). Impact of learner engagement on attrition rates and student success in online learning. International Journal of Instructional Technology & Distance Learning, 7(5), 3–9.

Schmader, T., Hall, W., & Croft, A. (2015). Stereotype threat in intergroup relations. In M. Mikulincer, P. R. Shaver, J. F. Dovidio, & J. A. Simpson (Eds.), APA handbook of personality and social psychology, Vol. 2. Group processes (pp. 447–471). American Psychological Association.

Shapiro, J. R., & Williams, A. N. (2011). The role of stereotype threats in undermining girls’ and women’s performance and interest in STEM. Feminist Forum.

Shernof, D. J., et al. (2017). Student engagement as a general factor of classroom experience: Associations with student practices and educational outcomes in a university gateway course. Frontiers in Psychology, 8, 994. doi:10.3389/fpsyg.2017.00994. PMID: 28663733; PMCID: PMC5471299.

Shinde, G. S. (2010). The relationship between students’ responses on the National Survey of Student Engagement (NSSE) and retention. Review of Higher Education & Self-Learning, 3(7), 54-67.

Steele, C. M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613–629. doi:10.1037/0003-066X.52.6.613.

Stevens, J. P. (1992). Applied multivariate statistics for the social sciences (2nd edition). Erlbaum.

Strayhorn, T. L. (2018). College students’ sense of belonging: A key to educational success for all students. 2nd edition. Routledge.

Suárez-Álvarez, J., Pedrosa, I., Lozano, L. M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in Likert scales: A questionable practice. Psicothema, 30(2), 149–158. doi:10.7334/psicothema2018.33. PMID: 29694314.

Venn, E., Park, J., Andersen, L. P., & Hejmadi, M. (2020). How do learning technologies impact on undergraduates’ emotional and cognitive engagement with their learning? Teaching in Higher Education, 1–18.

Walker, K., & Koralesky, K. (2021). Student and instructor perceptions of engagement after the rapid online transition of teaching due to COVID‐19. Natural Sciences Education, 50. doi:10.1002/nse2.20038.

Walton, G. M., & Cohen, G. L. (2011). A brief social-belonging intervention improves academic and health outcomes of minority students. Science, 331, 1447–1451. doi:10.1126/science.1198364 Medline

Walton, G. M., & Wilson, T. D. (2018). Wise interventions: Psychological remedies for social and personal problems. Psychological Review, 125(5), 617–655.

Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81. doi:10.1006/ceps.1999.1015

Xu, D., & Jaggars, S. S. (2013). The impact of online learning on students’ course outcomes: Evidence from a large community and technical college system. Economics of Education Review, 37, 46–57. https://doi.org/f5n2d6

Xu, D., & Jaggars, S. S. (2014). Performance gaps between online and face-to-face courses: Differences across types of students and academic subject areas. The Journal of Higher Education, 85(5), 633–659. https://doi.org/ gmdjdw

Yeager, D. S., Bryk, A., Muhich, J., Hausman, H., & Morales, L. (2013). Practical measurement. Carnegie Foundation for the Advancement of Teaching.