Student Ratings and Course Modalities
A Small Study in a Large Context
DOI:
https://doi.org/10.24059/olj.v27i3.4053Abstract
This article examines the impact of course modality on student evaluation of courses and professors. Data were collected through the Student Perception of Instruction end of course rating form at the University of Central Florida (UCF), which contains nine items and maintains student anonymity. The findings indicate that while course modality accounts for only 1% of the variance in student evaluations, there is strong internal consistency and reliability in the rating scale. The distribution of ratings showed a concentration of scores at the high end, resulting in a high variability coefficient. However, when the long tail of low ratings was removed, the mean increased and the distribution became more symmetric, affecting various psychometric indices. The correlation matrices among the items revealed a single factor solution for each modality, suggesting that students tend to rely on a general impression when rating their courses. The multidimensional scaling process identified underlying categories such as structure, course climate, engagement, and consideration, even though students did not explicitly differentiate these elements in their responses. The study concludes that course modality has minimal influence on overall student ratings, a finding consistent across different time periods, including the COVID-19 pandemic. Although a single factor captures students' general evaluations, underlying categories shape their responses. The article also presents a classification model that predicts student ratings based on the scale items. This research addresses the complex dynamics of student evaluations, highlighting the nuanced relationship between course modality, student perceptions, and the underlying factors influencing their ratings.
References
Abrami, P. C., & D’Apollonia, S. (1991). Multidimensional students’ evaluations of teaching effectiveness: Generalizability of “N = 1” research: Comment of Marsh. Journal of Educational Psychology, 83(3), 411–415.
Algozzine, B., Gretes, J., Flowers, C., Howley, L., Beattie, J., Spooner, F., et al. (2004). Student evaluation of college teaching: A practice in search of principles. College Teaching, 52(4), 134–141.
Andersen, K. (2018). Fantasyland: How America went haywire: A 500-year history. Random House.
Apodaca, P., & Grad, H. (2005). The dimensionality of student ratings of teaching: Integration of uni- and multidimensional models. Studies in Higher Education, 30(6), 723–748.
Badur, B., and Mardikyan, S. (2011). Analyzing teaching performance of instructors using data mining techniques. Inform. Educ. 10, pp. 245–257.
Bangert, A. W. (2006). Identifying factors underlying the quality of online teaching effectiveness: An exploratory study. Journal of Computing in Higher Education, 17(2), 79–99.
Benjamin, R. (2020). Race after technology: Abolitionist tools for the new jim code. Polity.
Boghosian, B. (2020). The inescapable casino. The Best Writing on Mathematics 2020. Princeton University Press.
Borg, I., & Groenen, P. J. F. (2005). Modern multidimensional scaling: Theory and applications. Springer.
Boring, A., Ottoboni, K., & Stark P. (2016). Student evaluations of teaching are not only unreliable, they are significantly biased against female instructors. LSE Impact Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2016/02/04/student-evaluations-of-teaching-gender-bias/
Boring, A., Ottoboni, K., & Stark, P. B. (2017). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research, (01), 1. https://www.scienceopen.com/hosted-document?doi=10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1
Brieman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. Chapman and Hall/CRC.
Brocato, B. R., Bonanno, A., & Ulbig, S. (2015). Student perceptions and instructional evaluations: A multivariate analysis of online and face-to-face classroom settings. Education and Information Technologies 20(1), 37–55.
Bryant, J. (2022, October 7). High school graduates are saying no to college: Here’s why. Best Colleges. https://www.bestcolleges.com/news/analysis/why-high-school-grads-are-saying-no-to-college/
Buser, W., Batz-Barbarich, C., & Hayter, J. (2022). Evaluation of women in economics: Evidence of gender bias following behavioral role violations. Sex Roles, Vol. 86. p. 695–710.
Bush, M. G. (2023). Why high school graduates are saying no to college and the consequences of their decisions.[Unpublished manuscript]. Rosen Foundation.
Capa-Aydin, Y. (2016). Student evaluation of instruction: comparison between in-class and online methods. Assessment & Evaluation in Higher Education, Vol. 41(1), 112–126.
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245–276. https://doi.org/10.1207/s15327906mbr0102_10
Chang, T. S. (2000, April). An application of regression models with student ratings in determining course effectiveness. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED455311)
ChatGPT. (2023, April 26). How many articles have been published on student ratings in higher education [Response to a question]. OpenAI. https://openai.com/
Chatman, J., Sharps, D., Mishra, S., Kray, L., & North, M. (2022). Agentic but not warm: Age-gender interactions and the consequences of stereotype incongruity perceptions for middle-aged professional women. Organizational Behavior and Human Decision Processes, Vol. 173.
Clayson, D. E. (1999). Students’ evaluation of teaching effectiveness: Some implications of stability. Journal of Marketing Education, 21(1), 68–75.
Cohen, E. H. (2005). Student evaluations of course and teacher: Factor analysis and SSA approaches. Assessment & Evaluation in Higher Education, 30(2), 123–136.
Crocker, L., & Algina, J. (1986). Introduction to classical & modern test theory. Holt, Rinehart, and Winston Inc.
Crumbley, L., Henry, B. K., & Kratchman, S. H. (2001). Students’ perceptions of the evaluation of college teaching. Quality Assurance in Education, 9(4), 197–207.
Davidovitch, N., & Soen, D. (2006). Using students’ assessments to improve instructors’ quality of teaching. Journal of Further and Higher Education, 30(4), 351–376.
Dziuban, C., & Shirkey, E. (1974). When is a correlation matrix appropriate for factor analysis? Psychological Bulletin, 81(6), 358–361.
Dziuban, C., & Shirkey, E. (1993). A sequential psychometric criterion for the dimension of a rescaled covariance matrix. Annual Meeting of the Florida Educational Research Association, November 10–13, 1993.
Dziuban, C., Moskal, P., Kramer, L. & Thompson, J. (2012). Student satisfaction with online learning in the presence of ambivalence: Looking for the will-o’-the-wisp. The Internet and Higher Education. doi: 10.1016/j.iheduc.2012.08.001
Dziuban, C., Graham, C. R., Moskal, P. D., Norberg, A., & Sicilia, N. (2018). Blended learning: The new normal and emerging technologies. International Journal of Educational Technology in Higher Education, 15(1). https://doi.org/10.1186/s41239-017-0087-5
Eiszler, C. F. (2002). College students’ evaluations of teaching and grade inflation. Research in Higher Education, 43(4), 483–501.
Ernst, D. (2014). Expectancy theory outcomes and student evaluations of teaching, Educational Research and Evaluation, 20(7–8), 536–556. https://doi.org/10.1080/13803611.2014.997138
Esarey, J., & Valdes, N. (2020). Unbiased, reliable, and valid student evaluations can still be unfair. Assessment & Evaluation in Higher Education, 2020(1), 1–15.
Eubanks, V. (2019). Automating inequality: How high-tech tools profile, police, and punish the poor. Picador.
Feldman, K. A. (1976). The superior college teacher from the students’ view. Research in Higher Education, 5(3), 243–288.
Filak, V. F., & Nicolini, K. M. (2018). Differentiations in motivation and need satisfaction based on course modality: A self-determination theory perspective. Educational Psychology, 38(6), 772–784. https://doi.org/10.1080/01443410.2018.1457776
Flaherty, C. (2018, May 22). Most institutions say they value teaching but how they assess it tells a different story. Inside Higher Ed. https://www.insidehighered.com/news/2018/05/22/most-institutions-say-they-value-teaching-how-they-assess-it-tells-different-story
Flaherty, C. (2019, May 20). Fighting gender bias in student evaluations of teaching, and tenure's effect on instruction. Inside Higher Ed. https://www.insidehighered.com/news/2019/05/20/fighting-gender-bias-student-evaluations-teaching-and-tenures-effect-instruction
Flaherty, C. (2022, January 19). Study: Grade satisfaction a major factor in student evals. Inside Higher Ed. https://www.insidehighered.com/news/2022/01/19/study-grade-satisfaction-major-factor-student-evals
Floden, J. (2017). The impact of student feedback on teaching in higher education. Assessment & Evaluation in Higher Education, 42(7), 1054–1068.
Floridi, L. (2014). The 4th revolution: How the infosphere is reshaping human reality. Oxford University Press.
Foster, M. (2023). Instructor name preference and student evaluations of instruction. PS: Political Science & Politics, 56(1), 143–149. doi:10.1017/S1049096522001068
Friedman, S., & Laurison, D. (2019). The class ceiling: Why it pays to be privileged. Policy Press.
Gelsinger, P. (2018). Mind-blowing to mundane: How tech is reshaping our expectations. MIT Technology Review, 121(2), 7.
Genetin, B., Chen, J., Kogan, V., & Kalish, A. (2021, December 1). Mitigating implicit bias in student evaluations: A randomized intervention. Wiley Online Library. https://onlinelibrary.wiley.com/doi/epdf/10.1002/aepp.13217?saml_referrer
Ginns, P., & Ellis, R. (2007). Quality in blended learning: Exploring the relationships between online and face-to-face teaching and learning. The Internet and Higher Education, 10(1), 53–64.
Ginns, P., Prosser, M., & Barrie, S. (2007). Students’ perceptions of teaching quality in higher education: The perspective of currently enrolled students. Studies in Higher Education, 32(5), 603–615.
Giridharadas, A. (2018). Winners take all: The elite charade of changing the world. Knopf. Golding, C., & Adam, L. (2016). Evaluate to improve: Useful approaches to student evaluation. Assessment & Evaluation in Higher Education, 41(1), 1–14, http://dx.doi.org/10.1080/02602938.2014.976810
Greenwald, A. G. (Ed.). (1997). Student ratings of professors [Current Issues]. American Psychologist, 52(11), 1182–1225.
Greenwald, A. G., & Gilmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52(11), 1209–1217.
Griffin, B. (2016). Perceived autonomy support, intrinsic motivation, and student ratings of instruction. Studies in Educational Evaluation. 51, 116–125.
Gumbel, A. (2020). Won’t lose this dream: How an upstart urban university rewrote the rules of a broken system. The New Press.
Gump, S. E. (2007). Student evaluations of teaching effectiveness and the leniency hypothesis: A literature review. Educational Research Quarterly, 30(3), 55–68.
Gunduz, N., & Fokoue, E. (2021). Understanding students’ evaluations of professors using non-negative matrix factorization. Journal of Applied Statistics, 48(1), 2961–2981.
Heffernan, T. (2021). Sexism, racism, prejudice, and bias: a literature review and synthesis of research surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2021.1888075
Hill, B. (2011). The sequential Kaiser-Meyer-Olkin procedure as an alternative for determining the number of factors in common-factor analysis: A monte carlo simulation. Graduate College of the Oklahoma State University.
Hornstein, H. A. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. Cogent Education, 4(1), 1304016.
Howell, A. J., & Symbaluk, D. G. (2001). Published student ratings of instruction: Revealing and reconciling the views of students and faculty. Journal of Educational Psychology, 93(4), 790–796.
Isenberg, N. (2017). White trash: The 400-year untold history of class in America. Penguin Books.
Johnson, S. (2011). Where good ideas come from: The natural history of innovation. Riverhead Books.
Kabudi, T., Pappas, I., & Olsen, D. H. (2021). AI-enabled Adaptive Learning Systems: A systematic mapping of the literature. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100017
Kaiser, H. (1968). A measure of the average intercorrelation. Educational and Psychological Measurement, 28, 245–247.
Kaiser, H. F., & Rice, J. (1974). Little jiffy, Mark IV. Educational and Psychological Measurement, 34(1), 111–117. https://doi.org/10.1177/001316447403400115
Kim, L. E., & MacCann, C. (2018). Instructor personality matters for student evaluations: Evidence from two subject areas at university. British Journal of Educational Psychology, 88, 584–605.
Kogan, V., Genetin, B., Chen, J., and Kalish, A. (2022). Students' grade satisfaction influences evaluations of teaching: Evidence from individual-level data and an experimental intervention. (EdWorkingPaper: 22-513). Annenberg Institute at Brown University. https://doi.org/10.26300/spsf-tc23
Kolitch, E., & Dean, A. V. (1999). Student ratings of instruction in the USA: Hidden assumptions and missing conceptions about ‘good’ teaching. Studies in Higher Education, 24(1), 27–42.
Kornell, N., & Hausman, H. (2016). Do the best teachers get the best ratings? Frontiers in Psychology, 7, 570. https://doi.org/10.3389/fpsyg.2016.00570
Lannutti, P. J., & Strauman, E. C. (2006). Classroom communication: The influence of instructor self-disclosure on student evaluations. Communication Quarterly, 54(1), 89–99.
Lowenthal, P., Bauer, C., & Chen, K. (2015). Student perceptions of online learning: An analysis of online course evaluations. American Journal of Distance Education, 29(2), 85–97.
Mandouit, L. (2018). Using student feedback to improve teaching. Educational Action Research, 26(5), 755–769.
Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187–1197.
Mather, M., & Sarkans, A. (2018). Student perceptions of online and face-to-face learning. International Journal of Curriculum and Instruction, 10(2), 61–76.
Maurer, T. W. (2006). Cognitive dissonance or revenge? Student grades and course evaluations. Teaching of Psychology, 33(3), 176–179.
McGhee, H. (2021). The sum of us: What racism costs everyone and how we can prosper together. One World.
Mengel, F., Sauermann, J., & Zolitz, U. (2019). Gender bias in teaching evaluations. Journal of the European Economic Association, 17(2), 535–566.
Mitchell, K. M., & Martin, J. (2018). Gender bias in student evaluations. PS: Political Science & Politics, 51(3), 648–652.
Mlodinow, L. (2009). The drunkard’s walk: How randomness rules our lives. Penguin Books.
Mukherjee, S. (2016). The gene: An intimate history. Scribner.
Muller, J. (2018). The tyranny of metrics. Princeton University Press.
Nasser, F., & Fresko, B. (2006). Predicting student ratings: The relationship between actual student ratings and instructors’ predictions. Assessment & Evaluation in Higher Education, 31(1), 1–18.
Oliver, R. L., & Sautter, E. P. (2005). Using course management systems to enhance the value of student evaluations of teaching. Journal of Education for Business, 80(4), 231–234.
O’Neil, C. (2018). Weapons of math destruction how big data increases inequality and threatens democracy. Penguin Books.
Page, S., (2009). Understanding complexity. The Great Courses.
Pirsig, R. (2006). Zen and the art of motorcycle maintenance: An inquiry into values. William Morrow Paperbacks.
Pounder, J. S. (2007). Is student evaluation of teaching worthwhile? An analytical framework for answering the question. Quality Assurance in Education, 15(2), 178–191.
Ray, B., Babb, J., & Wooten, C. A. (2018). Rethinking SETs: Retuning student evaluations of teaching for student agency. Composition Studies, 46(1), 34–194.
Read, W. J., Rama, D. V., & Raghunandan, K. (2001). The relationship between student evaluations of teaching and faculty evaluations. Journal of Education for Business, 76(4), 189–192.
Renaud, R. D., & Murray, H. G. (2005). Factorial validity of student ratings of instruction. Research in Higher Education, 46(8), 929–953.
Richardson, J. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6, 135–147.
Rinderman, H., & Schofield, N. (2001). Generalizability of multidimensional student ratings of university instruction across courses and teachers. Research in Higher Education, 42(4), 377–399.
Riniolo, T. C., Johnson, K. C., Sherman, T. R., & Misso, J. A. (2006). Hot or not: Do professors perceived as physically attractive receive higher student evaluations? Journal of General Psychology, 133(1), 19–35.
Rosch, Eleanor (Eleanor Heider). 1973. Natural Categories. Cognitive Psychology, 4, 328–350.
Royal, K. D., & Stockdale, M. R. (2015). Are teacher course evaluations biased against faculty that teach quantitative methods courses? International Journal of Higher Education, 4(1). https://doi.org/10.5430/ijhe.v4n1p217
Safir, S., & Dugan, J. (2021). Street data: A next-generation model for equity, pedagogy, and school transformation. Sage Publications Inc.
Scherer, R., & Gustafsson, J. E. (2015). Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: an application of multilevel bifactor structural equation modeling. Frontiers in Psychology, 6. http://dx.doi.org/10.3389/fpsyg.2015.01550
Searle, J. (1995). The construction of social reality. The Free Press.
Sellnow-Richmond, D., Strawser, M. G., & Sellnow, D. D. (2020). Student perceptions of teaching effectiveness and learning achievement: A comparative examination of online and hybrid course delivery format. Communication Teacher, 34(3), 248–263.
Sheehan, E. P., & DuPrey, T. (1999). Student evaluations of university teaching. Journal of Instructional Psychology, 26(3), 188–194.
Shevlin, M., Banyard, P., Davies, M., & Griffiths, M. (2000). The validity of student evaluation of teaching in higher education: Love me, love my lectures? Assessment & Evaluation in Higher Education, 25(4), 397–405.
Sloan, R. (2013). Mr. Penumbra’s 24-hour bookstore. Picador Paper.
Smith, G., & Anderson, K. J. (2005). Students’ ratings of professors: The teaching style contingency for Latino professors. Journal of Latinos and Education, 4(2), 115–136.
Snygg, D., & Combs, A. W. (1949). Individual behavior: A new frame of reference for psychology. Harper.
Stapleton, R. J., & Murkison, G. (2001). Optimizing the fairness of student evaluations: A study of correlations between instructor excellence, study production, learning production, and expected grades. Journal of Management Education, 25(3), 269–291.
Stark, P. B., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1
Stott, P. (2016). The perils of a lack of student engagement: Reflections of a “lonely, brave, and rather exposed” online instructor. British Journal of Educational Technology, 47(1), 51–64.
Stroebe, W. (2016). Why good teaching evaluations may reward bad teaching: On grade inflation and other unintended consequences of student evaluations. Perspectives on Psychological Science, 11(6), 800–816. https://doi.org/10.1177/1745691616650284
Taleb, N. N. (2005). Fooled by randomness: The hidden role of chance in life and in the markets. Random House.
Taleb, N. N. (2007). The black swan: The impact of the highly improbable. Random House.
Taleb, N. N. (2012). Antifragile: Things that gain from disorder. Random House.
Taleb, N. N. (2018). Skin in the game: Hidden asymmetries in daily life. Random House.
Tejeiro, R., Whitelock-Wainwright, A., Perez, A., Urbina-Garcia, M. A. (2018). The best-achieving online students are overrepresented in course ratings. European Journal of Open Education and E-learning Studies, 3(2), 43–58.
Turner, K. M., Hatton, D., & Theresa, M. (2018). Student evaluations of teachers and courses: Time to wake up and shake up. Nursing Education Perspectives, 39(3), 130–131.
Uttl, B., & Smibert, D. (2017). Student evaluations of teaching: teaching quantitative courses can be hazardous to one’s career. PeerJ. http://dx.doi.org/10.7717/peerj.3299
Wachtel, H. K. (1998). Student evaluation of college teaching effectiveness: A brief overview. Assessment & Evaluation in Higher Education, 23(2), 191–212.
Wang, M. C., Dziuban, C. D., Cook, I. J., Moskal, P. D. (2009). Dr. Fox Rocks: Using Data-mining Techniques to Examine Student Ratings of Instruction. In M. C. Shelley II, L. D. Yore, & B. Hand (Eds.), Quality research in literacy and science education: International perspectives and gold standards. Springer.
Wilkerson, I. (2020). Caste: The origins of our discontents. Random House.
Williams, R. G., & Ware, J. E., Jr. (1977). An extended visit with Dr. Fox: Validity of student satisfaction with instruction ratings after repeated exposures to a lecturer. American Educational Research Journal, 14(4), 449–457.
Yen, S.-C., Lo, Y., Lee, A., & Enriquez, J. M. (2018). Learning online, offline, and in-between: Comparing student academic outcomes and course satisfaction in face-to-face, online, and blended teaching modalities. Education and Information Technologies, 23(5), 2141–2153. https://doi.org/10.1007/s10639-018-9707-5
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Chuck Dziuban, Patsy Moskal, Annette Reiner, Adysen Cohen
This work is licensed under a Creative Commons Attribution 4.0 International License.
As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.
This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.
Author(s) hold copyright in their work, and retain publishing rights without restrictions
