Establishing a Student Evaluation of Online Teaching and Learning Framework Through Analysis of Existing Instruments

Authors

  • Ting Sun Northwestern University
  • Florence Martin North Carolina State University
  • Stella Kim University of North Carolina at Charlotte
  • Carl Westine University of North Carolina at Charlotte

DOI:

https://doi.org/10.24059/olj.v27i1.3228

Keywords:

student evaluation, online teaching and learning, instrument development and validation

Abstract

This study aims to establish a framework for student evaluation of online teaching and learning (SEOTL) through an analysis of SEOTL instruments implemented by universities and colleges. From a list of 131 R1 and 135 R2 institutions, we searched, reviewed, and identified 27 instruments for student evaluation of online teaching. A five-dimensional evaluation framework with 24 categories of elements was developed through an analysis of these instruments. There were 278 evaluation elements among the 27 instruments. We found that most instruments focus more on the Course and Instructor dimensions, with Instructor Facilitation and Learning Goals and Objectives elements occurring most frequently. However, Organization and Technology dimensions with Advising Availability and Adequacy, Registration Procedures, Support Services, and Online Help Desk elements were least included. This study has implications for administrators, instructors, instructional designers, and students.

References

Bangert. A. W. (2004). The seven principles of good practice: A framework for evaluating online teaching. The Internet and Higher Education, 7(3), 217-232. https://doi.org/10.1016/j.iheduc.2004.06.003

Bangert, A. W. (2005). Identifying factors underlying the quality of online teaching effectiveness: An exploratory study. Journal of Computing in Higher Education, 17(2), 79-99. https://doi.org/10.1007/bf03032699

Bangert, A. W. (2006). The development of an instrument for assessing online teaching effectiveness. Journal of Educational Computing Research, 35(3), 227-244. https://doi.org/10.2190/b3xp-5k61-7q07-u443

Bangert, A. W. (2008). The development and validation of the student evaluation of online teaching effectiveness. Computers in the Schools, 25(1-2), 25-47. https://doi.org/10.1080/07380560802157717

Benton, S. L., & Cashin, W. E. (2012). Idea paper# 50 student ratings of teaching: A summary of research and literature. The IDEA Center.

Berk, R. A. (2013). Face-to-face versus online course evaluations: A “consumer's guide” to seven strategies. Journal of Online Learning and Teaching, 9(1), 140.

Biner, P. M. (1993). The development of an instrument to measure student attitudes toward televised courses. American Journal of Distance Education, 7(1), 62-73. https://doi.org/10.1080/08923649309526811

Blackman, G., Pedersen, J., March, M., Reyes-Fournier, E., & Cumella, E. J. (2019). A comprehensive literature review of online teaching effectiveness: Reconstructing the conceptual framework. Unpublished manuscript.

Carle, A. C. (2009). Evaluating college students’ evaluations of a professor’s teaching effectiveness across time and instruction mode (online vs. face-to-face) using a multilevel growth modeling approach. Computers & Education, 53(2): 429–435.

https://doi.org/10.1016/j.compedu.2009.03.001

Carnegie Classification of Institutions of Higher Education, (2021). https://carnegieclassifications.iu.edu/

Centra, J. A. (1993). Reflective faculty evaluation. Jossey-Bass.

Cheung, D. (1998). Developing a student evaluation instrument for distance teaching. Distance Education, 19(1), 23-42. https://doi.org/10.1080/0158791980190104

Chickering, A. W., & Gamson, Z. F. (1989). Seven principles for good practice in undergraduate education. Biochemical Education, 17(3), 140–141. https://doi.org/10.1016/0307-4412(89)90094-0

Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51(3), 281-309. https://doi.org/10.3102/00346543051003281

Darwin, S. (2017). What contemporary work are student ratings actually doing in higher education? Studies in Educational Evaluation, 54, 13-21. https://doi.org/10.1016/j.stueduc.2016.08.002

Driscoll, M. (1998). Web-based training. Jossey-Bass/Pfeiffer.

Feistauer, D., & Richter, T. (2018). Validity of students’ evaluations of teaching: Biasing effects of likability and prior subject interest. Studies in Educational Evaluation, 59, 168-178.

Galbraith, C., Merrill, G., & Kline, D. (2012). Are student evaluations of teaching effectiveness valid for measuring student outcomes in business related classes? A neural network and Bayesian analyses. Research in Higher Education, 53, 353–374. https://doi.org/10.1007/s11162-011-9229-0

Gurley, L. E. (2018). Educators' preparation to teach, perceived teaching presence, and perceived teaching presence behaviors in blended and online learning environments. Online Learning, 22(2), 197-220. https://doi.org/10.24059/olj.v22i2.1255

Harris, D. N., Ingle, W. K., & Rutledge, S. A. (2014). How teacher evaluation methods matter for accountability: A comparative analysis of teacher effectiveness ratings by principals and teacher value-added measures. American Educational Research Journal, 51(1), 73-112. https://doi.org/10.3102/0002831213517130

Khan, B. H. (1997). Web-based instruction (WBI): What is it and why is it? In Web-based instruction, ed. B. H. Khan, 5–18. Educational Technology Publications.

Kogan, J. (2014, April). Student Course Evaluation. In Proceedings of the 6th International Conference on Computer Supported Education-Volume 2 (pp. 221-225).

Kreitzer, R. J., & Sweet-Cushman, J. (2021). Evaluating student evaluations of teaching: A review of measurement and equity bias in SETs and recommendations for ethical reform. Journal of Academic Ethics, 1-12. https://doi.org/10.1007/s10805-021-09400-w

Liu, Y. (2006). A comparison study of online versus traditional student evaluation of instruction. International Journal of Instructional Technology & Distance Learning, 3(4): 15–29.

Lowenthal, P., Bauer, C., & Chen, K. Z. (2015). Student perceptions of online learning: An analysis of online course evaluations. American Journal of Distance Education, 29(2), 85-97. https://doi.org/10.1080/08923647.2015.1023621

Marsh, H. W., Muthèn, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J. S., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students’ evaluations of university teaching. Structural Equation Modeling, 16, 439–476. https://doi.org/10.1080/10705510903008220

Martin, F., Bolliger, D. U., & Flowers, C. (2021). Design matters: Development and validation of the online course design elements (OCDE) instrument. The International Review of Research in Open and Distributed Learning, 22(2), 46-71. https://doi.org/10.19173/irrodl.v22i2.5187

Martin, F., Sun, T., & Westine, C. D. (2020). A systematic review of research on online teaching and learning from 2009 to 2018. Computers & Education, 159, 104009. https://doi.org/10.1016/j.compedu.2020.104009

Martin, F., Wang, C., & Sadaf, A. (2020). Facilitation matters: Instructor perception of helpfulness of facilitation strategies in online courses. Online Learning, 24(1), 28-49. https://doi.org/10.24059/olj.v24i1.1980

McMahon, T., Barrett, T., & O'Neill, G. (2007). Using observation of teaching to improve quality: Finding your way through the muddle of competing conceptions, confusion of practice and mutually exclusive intentions. Teaching in Higher Education, 12(4), 499-511. https://doi.org/10.1080/13562510701415607

Nasri, N. M., Husnin, H., Mahmud, S. N. D., & Halim, L. (2020). Mitigating the COVID-19 Pandemic: A snapshot from Malaysia into the coping strategies for pre-service teachers’ education. Journal of Education for Teaching, 46(4), 546--553. https://doi.org/10.1080/02607476.2020.1802582

Ndoye, A., & Martin, F. (2021). Examining student perceptions of important features in online courses: A study based on demographic and contextual characteristics. Journal of Educators Online, 18(2).

Onwuegbuzie, A. J., Daniel, L. G., & Collins, K. M. (2009). A meta-validation model for assessing the score-validity of student teaching evaluations. Quality & Quantity, 43(2), 197-209. https://doi.org/10.1007/s11135-007-9112-4

Raible, J., Bennett, L., & Bastedo, K. (2016). Writing measurable learning objectives to aid successful online course development. International Journal for the Scholarship of Technology Enhanced Learning, 1(1), 112-122.

Rapanta, C., Botturi, L., Goodyear, P., Guàrdia, L., & Koole, M. (2020). Online university teaching during and after the Covid-19 crisis: Refocusing teacher presence and learning activity. Postdigital Science and Education, 2(3), 923-945. https://doi.org/10.1007/s42438-020-00155-y

Reyes-Fournier, E., Cumella, E. J., Blackman, G., March, M., & Pedersen, J. (2020). Development and validation of the online teaching effectiveness scale. Online Learning, 24(2), 111-127. https://doi.org/10.24059/olj.v24i2.2071

Roberts, T. G., Irani, T. A., Telg, R. W., & Lundy, L. K. (2005). The development of an instrument to evaluate distance education courses using student attitudes. The American Journal of Distance Education, 19(1), 51-64. https://doi.org/10.1207/s15389286ajde1901_5

Rothman, T., Romeo, L., Brennan, M., & Mitchell, D. (2011). Criteria for assessing student satisfaction with online courses. International Journal for e-Learning Security, 1(1-2), 27-32. https://doi.org/10.20533/ijels.2046.4568.2011.0004

Seok, S., DaCosta, B., Kinsell, C., & Tung, C. K. (2010). Comparison of instructors' and students' perceptions of the effectiveness of online courses. Quarterly Review of Distance Education, 11(1), 25-36.

Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598-642. https://doi.org/10.3102/0034654313496870

Stewart, I., Hong, E., & Strudler, N. (2004). Development and validation of an instrument for student evaluation of the quality of web-based instruction. The American Journal of Distance Education, 18(3), 131-150. https://doi.org/10.1207/s15389286ajde1803_2

Thomas, J. E., & Graham, C. R. (2017). Common practices for evaluating post-secondary online instructors. Online Journal of Distance Learning Administration, 20(4).

Weinkle, L. J., Stratford, J. M., Lee, L. M. J. (2020). Voice in digital education: The Impact of Instructor’s perceived age and gender on student learning and evaluation. Anatomical Sciences Education, 13, 59–70. https://doi.org/10.1002/ase.1865

Downloads

Published

2023-03-01

Issue

Vol. 27 No. 1 (2023)

Section

Section II

License

As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.

This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.

Author(s) hold copyright in their work, and retain publishing rights without restrictions