Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects

Authors

  • Heng Luo John A. Dutton E-Education Institute, The Pennsylvania State University
  • Anthony C. Robinson Department of Geography, The Pennsylvania State University
  • Jae-Young Park John A. Dutton E-Education Institute, The Pennsylvania State University

DOI:

https://doi.org/10.24059/olj.v18i2.429

Keywords:

peer grading, MOOC, reliability, validity

Abstract

Peer grading affords a scalable and sustainable way of providing assessment and feedback to a massive student population, and has been used in massive open online courses (MOOCs) on the Coursera platform. However, currently there is little empirical evidence to support the credentials of peer grading as a learning assessment method in the MOOC context. To address this research need, this study examined 1825 peer grading assignments collected from a Coursera MOOC with the purpose of investigating the reliability and validity of peer grading as well as its perceived effects on students’ MOOC learning experience. The empirical findings proved that the aggregate ratings of student graders can provide peer grading scores that were fairly consistent and highly similar to the instructor grading scores. Student responses to a survey also show that the peer grading activity was well received as the majority of MOOC students believed it was fair, useful, beneficial, and would recommend it to be included in future MOOC offerings. Based on the empirical results, this study concludes with a set of principles for designing and implementing peer grading activities in the MOOC context.

Author Biographies

Heng Luo, John A. Dutton E-Education Institute, The Pennsylvania State University

Heng Luo is currently working as a Research Associate at the John A. Dutton e-Education Institute, Penn State University. He is well-trained in the fields of instructional design, educational technology, learning and instructional science, and educational evaluation, and has educational backgrounds in both computer technology and applied linguistics. He was a doctoral fellow at Syracuse University and has received years of research training in both quantitative and qualitative inquiries there. He has published in areas such as case-based online instruction, interactive and mobile learning design, and modelling-based science education.

Anthony C. Robinson, Department of Geography, The Pennsylvania State University

Anthony C. Robinson is the lead faculty member for Penn State's Postbaccalaureate GIS Certificate and Master of Geographic Information Systems programs in the John A. Dutton e-Education Institute. He is also an Assistant Director for the GeoVISTA Center in the Department of Geography. He is the instructor of one of Penn State's first MOOCs on Coursera - Maps and the Geospatial Revolution. His research focuses broadly on designing and evaluating geovisualization tools. He has done work in epidemiology, crisis management, and national security domains. His most recent efforts involve characterizing how users assemble and collect their analytical results, studying the use of visualization tools using eye-tracking, and exploring issues related to map symbol standardization in the context of emergency management.

Jae-Young Park, John A. Dutton E-Education Institute, The Pennsylvania State University

Jae-Young Park is a Ph.D. candidate in the Workforce Education and Development program at Pennsylvania State University. She earned her BA in Public Administration and MA in Human Resource Development for Women at Sookmyung Women’s University in Korea (South). Her primary research topic focuses on human capital policy/strategy and evaluation including women workforce development, professional/career development in higher education, and competency-based workforce development .

References

Billington, H. L. (1997). Poster presentations and peer assessment: novel forms of evaluation and assessment. Journal of Biological Education, 31(3), 218-220.

Bloom, B. S. (1956). Taxonomy of educational objectives: Vol. 1. Cognitive domain. New York: McKay.

Bostock, S. (2000). Student peer assessment. Keele, Staffordshire: Centre for Learning Technology, Keele University. Retrieved from http://www.reading.ac.uk/web/FILES/engageinassessment/Student_peer_assessment_-_Stephen_Bostock.pdf

Bouzidi, L., & Jaillet, A. (2009). Can Online Peer Assessment be Trusted? Educational Technology & Society, 12 (4), 257–268.

Brown, S., Race, P., & Rust, C. (1995). Using and experiencing assessment, in P. Knight (Ed.) Assessment for Learning in Higher Education (pp.75-85). London: Kogan Page/SEDA.

Brown, S., Rust, C., & Gibbs, G. (1994). Strategies for diversifying assessment in higher education. Oxford: Oxford Centre for Staff Development.

Butcher, A. C., Stefani, L. A. J., & Tariq, V. N. (1995). Analysis of peer-, self- and staff-assessment in group project work. Assessment in Education, 2(2), 165-185.

Cheng, W., & Warren, M. (1999). Peer and teacher assessment of the oral and written tasks of a group project. Assessment & Evaluation in Higher Education, 24, 301–314.

Cho, K., Schunn, C., & Wilson, R. (2006). Validity and Reliability of Scaffolded Peer Assessment of Writing from Instructor and Student Perspectives. Journal of Educational Psychology, 98 (4), 891-901.

Coursera. (n.d.). Pedagogical Foundations. Retrieved from https://www.coursera.org/about/pedagogy

Coursera. (2014, March 12) How will my grade be determined? Retrieved from. http://help.coursera.org/customer/portal/articles/1163304-how-will-my-grade-be-determined-

Dancey, C. P., & Reidy, J. (2002). Statistics without maths for psychology (2nd ed). London: Prentice Hall.

Falchikov, N. (1994). Learning from peer feedback marking: student and teacher perspectives. In H. C. Foot, C. J. Howe, A. Anderson, A. K. Tolmie, & D. A. Warden (Eds.), Group and interactive learning (pp. 411-416). Southampton and Boston: Computational Mechanics Publications.

Falchikov, N., & Goldfinch, J. (2000). Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research, 70 (3), 287-322.

Freeman, M. (1995). Peer assessment by groups of group work, Assessment and Evaluation in Higher Education, 20(3), 289-300.

Fry, S. A. (1990). Implementation and evaluation of peer marking in higher education. Assessment and Evaluation in Higher Education, 15(3), 177-189.

Gay, L. R., & Airasian, P. (2003). Educational research: Competencies for analysis and application (7th ed.). Columbus, OH: Merrill, Prentice Hall.

Haaga, D. A. F. (1993). Peer review of term papers in graduate psychology courses. Teaching of Psychology, 20(1), 28–32.

Hammond, K. R., & Kern, F. (1959). Teaching comprehensive medical care: a psychological study of a change in medical education. Cambridge, MA: Harvard University Press.

Kaimann, R. A. (1974). The coincidence of student evaluation by professor and peer group using rank correlation. The Journal of Educational Research, 68(4), 152-153.

Korman, M., & Stubblefield, R. L. (1971). Medical school evaluation and internship performance. Journal of Medical Education, 46, 670-673.

Lu, R., & Bol, L. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback. Journal of Interactive Online Learning, 6(2), 100-115.

Lu, J., & Law, N. (2012). Online peer assessment: effects of cognitive and affective feedback. Instructional Science, 40(2), 257-275.

Magin, D. (1993). Should student peer ratings be used as part of summative assessment? Higher Education Research and Development, 16, 537-542.

Magin, D. (2001). Reciprocity as a source of bias in multiple peer assessment of group work. Studies in Higher Education, 26(1), 53–63.

Marcoulides, G. A., & Simkin, M. G. (1995). The consistency of peer review in student writing projects. Journal of Education for Business, 70, 220–223.

McEwen, K. (2013, January 7). Getting to Know Coursera: Peer Assessments. Retrieved from http://cft.vanderbilt.edu/2013/01/getting-to-know-coursera-peer-assessments/

McGarr, O., & Clifford, A. M. (2013). ‘Just enough to make you take it seriously’: exploring students’ attitudes towards peer assessment. Higher education, 65(6), 677-693.

Miller, P. J. (2003). The effect of scoring criteria specificity on peer and self-assessment. Assessment & Evaluation in Higher Education, 28(4), 383-394.

Mok, J. (2011). A case study of students' perceptions of peer assessment in Hong Kong. ELT journal, 65(3), 230-239.

Morrison, D. (2013, March 9). Why and When Peer Grading is Effective for Open and Online Learning. Retrieved from http://onlinelearninginsights.wordpress.com/2013/03/09/why-and-when-peer-grading-is-effective-for-open-and-online-learning/

Mowl, G., & Pain, R. (1995). Using self and peer assessment to improve students’ essay writing—A case study from geography. Innovations in Education and Training International, 32, 324–335.

Neidlinger, J. (2013, May 13). Does peer grading of essays really work in a Coursera online class? Retrieved from http://loneprairie.net/peer-grading-coursera/

Oldfield, K. A., & Macalpine, M. K. (1995). Peer and self-assessment at tertiary level - an experimental report. Assessment and Evaluation in Higher Education, 20(1), 125-131.

Orpen, C. (1982). Student versus lecturer assessment of learning: a research note. Higher Education, 11, 567-572.

Pappano, L. (2012, November 2). The Year of the MOOC. The New York Times. Retrieved from http://www.nytimes.com/2012/11/04/education/edlife/massive-open-oline-courses-are-multiplying-at-a-rapid-pace.html?pagewanted=all&_r=0

Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned Models of Peer Assessment in MOOCs. Retrieved from http://www.stanford.edu/~jhuang11/research/pubs/edm13/edm13.pdf

Race, P. (1998). Practical pointers on peer-assessment. In S. Brown (Ed.) Peer Assessment in Practice (SEDA Paper 102) (pp.113-122). Birmingham, SEDA.

Rees, J. (2013, March 5). Peer Grading Can't Work. Retrieved from http://www.insidehighered.com/views/2013/03/05/essays-flaws-peer-grading-moocs#ixzz2MiKxNP7b

Sadler, P., & Good, E. (2006). The impact of self- and peer-grading on student learning. Educational Assessment, 11 (1), 1-31.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.

Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69–75.

Strijbos, J. W., Narciss, S., & Dünnebier, K. (2010). Peer feedback content and sender's competence level in academic writing revision tasks: are they critical for feedback perceptions and efficiency? Learning and instruction, 20(4), 291-303.

Strijbos, J. W., & Sluijsmans, D. (2010). Unravelling peer assessment: Methodological, functional, and conceptual developments. Learning and Instruction, 20(4), 265-269.

Topping, K. J. (2009). Peer assessment. Theory into Practice, 48(1), 20−27.

Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 149-169.

Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249-276.

Vu, T. T., & Dall’Alba, G. (2007). Students’ experience of peer assessment in a professional course. Assessment & Evaluation in Higher Education, 32(5), 541-556.

Watters, A. (2012, August 27). The Problems with Peer Grading in Coursera. Retrieved from http://www.insidehighered.com/blogs/hack-higher-education/problems-peer-grading-coursera

Wen, M. L., Tsai, C. C., & Chang, C. Y. (2006). Attitudes towards peer assessment: A comparison of the perspectives of pre-service and in-service teachers. Innovations in Education and Teaching International, 43(1), 83–92.

Zhang, B., Johnston, L., & Kilic, G. B. (2008). Assessing the reliability of self‐and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329-340.

Downloads

Published

2014-06-20

Issue

Vol. 18 No. 2 (2014)

Section

Massive Open Online Course (MOOC) Research

License

As a condition of publication, the author agrees to apply the Creative Commons – Attribution International 4.0 (CC-BY) License to OLJ articles. See: https://creativecommons.org/licenses/by/4.0/.

This licence allows anyone to reproduce OLJ articles at no cost and without further permission as long as they attribute the author and the journal. This permission includes printing, sharing and other forms of distribution.

Author(s) hold copyright in their work, and retain publishing rights without restrictions