Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects

Heng Luo, Anthony C. Robinson, Jae-Young Park

Abstract


Peer grading affords a scalable and sustainable way of providing assessment and feedback to a massive student population, and has been used in massive open online courses (MOOCs) on the Coursera platform. However, currently there is little empirical evidence to support the credentials of peer grading as a learning assessment method in the MOOC context. To address this research need, this study examined 1825 peer grading assignments collected from a Coursera MOOC with the purpose of investigating the reliability and validity of peer grading as well as its perceived effects on students’ MOOC learning experience. The empirical findings proved that the aggregate ratings of student graders can provide peer grading scores that were fairly consistent and highly similar to the instructor grading scores. Student responses to a survey also show that the peer grading activity was well received as the majority of MOOC students believed it was fair, useful, beneficial, and would recommend it to be included in future MOOC offerings. Based on the empirical results, this study concludes with a set of principles for designing and implementing peer grading activities in the MOOC context.

Keywords


peer grading, MOOC, reliability, validity

Full Text:

PDF

References


Billington, H. L. (1997). Poster presentations and peer assessment: novel forms of evaluation and assessment. Journal of Biological Education, 31(3), 218-220.

Bloom, B. S. (1956). Taxonomy of educational objectives: Vol. 1. Cognitive domain. New York: McKay.

Bostock, S. (2000). Student peer assessment. Keele, Staffordshire: Centre for Learning Technology, Keele University. Retrieved from http://www.reading.ac.uk/web/FILES/engageinassessment/Student_peer_assessment_-_Stephen_Bostock.pdf

Bouzidi, L., & Jaillet, A. (2009). Can Online Peer Assessment be Trusted? Educational Technology & Society, 12 (4), 257–268.

Brown, S., Race, P., & Rust, C. (1995). Using and experiencing assessment, in P. Knight (Ed.) Assessment for Learning in Higher Education (pp.75-85). London: Kogan Page/SEDA.

Brown, S., Rust, C., & Gibbs, G. (1994). Strategies for diversifying assessment in higher education. Oxford: Oxford Centre for Staff Development.

Butcher, A. C., Stefani, L. A. J., & Tariq, V. N. (1995). Analysis of peer-, self- and staff-assessment in group project work. Assessment in Education, 2(2), 165-185.

Cheng, W., & Warren, M. (1999). Peer and teacher assessment of the oral and written tasks of a group project. Assessment & Evaluation in Higher Education, 24, 301–314.

Cho, K., Schunn, C., & Wilson, R. (2006). Validity and Reliability of Scaffolded Peer Assessment of Writing from Instructor and Student Perspectives. Journal of Educational Psychology, 98 (4), 891-901.

Coursera. (n.d.). Pedagogical Foundations. Retrieved from https://www.coursera.org/about/pedagogy

Coursera. (2014, March 12) How will my grade be determined? Retrieved from. http://help.coursera.org/customer/portal/articles/1163304-how-will-my-grade-be-determined-

Dancey, C. P., & Reidy, J. (2002). Statistics without maths for psychology (2nd ed). London: Prentice Hall.

Falchikov, N. (1994). Learning from peer feedback marking: student and teacher perspectives. In H. C. Foot, C. J. Howe, A. Anderson, A. K. Tolmie, & D. A. Warden (Eds.), Group and interactive learning (pp. 411-416). Southampton and Boston: Computational Mechanics Publications.

Falchikov, N., & Goldfinch, J. (2000). Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research, 70 (3), 287-322.

Freeman, M. (1995). Peer assessment by groups of group work, Assessment and Evaluation in Higher Education, 20(3), 289-300.

Fry, S. A. (1990). Implementation and evaluation of peer marking in higher education. Assessment and Evaluation in Higher Education, 15(3), 177-189.

Gay, L. R., & Airasian, P. (2003). Educational research: Competencies for analysis and application (7th ed.). Columbus, OH: Merrill, Prentice Hall.

Haaga, D. A. F. (1993). Peer review of term papers in graduate psychology courses. Teaching of Psychology, 20(1), 28–32.

Hammond, K. R., & Kern, F. (1959). Teaching comprehensive medical care: a psychological study of a change in medical education. Cambridge, MA: Harvard University Press.

Kaimann, R. A. (1974). The coincidence of student evaluation by professor and peer group using rank correlation. The Journal of Educational Research, 68(4), 152-153.

Korman, M., & Stubblefield, R. L. (1971). Medical school evaluation and internship performance. Journal of Medical Education, 46, 670-673.

Lu, R., & Bol, L. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback. Journal of Interactive Online Learning, 6(2), 100-115.

Lu, J., & Law, N. (2012). Online peer assessment: effects of cognitive and affective feedback. Instructional Science, 40(2), 257-275.

Magin, D. (1993). Should student peer ratings be used as part of summative assessment? Higher Education Research and Development, 16, 537-542.

Magin, D. (2001). Reciprocity as a source of bias in multiple peer assessment of group work. Studies in Higher Education, 26(1), 53–63.

Marcoulides, G. A., & Simkin, M. G. (1995). The consistency of peer review in student writing projects. Journal of Education for Business, 70, 220–223.

McEwen, K. (2013, January 7). Getting to Know Coursera: Peer Assessments. Retrieved from http://cft.vanderbilt.edu/2013/01/getting-to-know-coursera-peer-assessments/

McGarr, O., & Clifford, A. M. (2013). ‘Just enough to make you take it seriously’: exploring students’ attitudes towards peer assessment. Higher education, 65(6), 677-693.

Miller, P. J. (2003). The effect of scoring criteria specificity on peer and self-assessment. Assessment & Evaluation in Higher Education, 28(4), 383-394.

Mok, J. (2011). A case study of students' perceptions of peer assessment in Hong Kong. ELT journal, 65(3), 230-239.

Morrison, D. (2013, March 9). Why and When Peer Grading is Effective for Open and Online Learning. Retrieved from http://onlinelearninginsights.wordpress.com/2013/03/09/why-and-when-peer-grading-is-effective-for-open-and-online-learning/

Mowl, G., & Pain, R. (1995). Using self and peer assessment to improve students’ essay writing—A case study from geography. Innovations in Education and Training International, 32, 324–335.

Neidlinger, J. (2013, May 13). Does peer grading of essays really work in a Coursera online class? Retrieved from http://loneprairie.net/peer-grading-coursera/

Oldfield, K. A., & Macalpine, M. K. (1995). Peer and self-assessment at tertiary level - an experimental report. Assessment and Evaluation in Higher Education, 20(1), 125-131.

Orpen, C. (1982). Student versus lecturer assessment of learning: a research note. Higher Education, 11, 567-572.

Pappano, L. (2012, November 2). The Year of the MOOC. The New York Times. Retrieved from http://www.nytimes.com/2012/11/04/education/edlife/massive-open-oline-courses-are-multiplying-at-a-rapid-pace.html?pagewanted=all&_r=0

Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned Models of Peer Assessment in MOOCs. Retrieved from http://www.stanford.edu/~jhuang11/research/pubs/edm13/edm13.pdf

Race, P. (1998). Practical pointers on peer-assessment. In S. Brown (Ed.) Peer Assessment in Practice (SEDA Paper 102) (pp.113-122). Birmingham, SEDA.

Rees, J. (2013, March 5). Peer Grading Can't Work. Retrieved from http://www.insidehighered.com/views/2013/03/05/essays-flaws-peer-grading-moocs#ixzz2MiKxNP7b

Sadler, P., & Good, E. (2006). The impact of self- and peer-grading on student learning. Educational Assessment, 11 (1), 1-31.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.

Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69–75.

Strijbos, J. W., Narciss, S., & Dünnebier, K. (2010). Peer feedback content and sender's competence level in academic writing revision tasks: are they critical for feedback perceptions and efficiency? Learning and instruction, 20(4), 291-303.

Strijbos, J. W., & Sluijsmans, D. (2010). Unravelling peer assessment: Methodological, functional, and conceptual developments. Learning and Instruction, 20(4), 265-269.

Topping, K. J. (2009). Peer assessment. Theory into Practice, 48(1), 20−27.

Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 149-169.

Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249-276.

Vu, T. T., & Dall’Alba, G. (2007). Students’ experience of peer assessment in a professional course. Assessment & Evaluation in Higher Education, 32(5), 541-556.

Watters, A. (2012, August 27). The Problems with Peer Grading in Coursera. Retrieved from http://www.insidehighered.com/blogs/hack-higher-education/problems-peer-grading-coursera

Wen, M. L., Tsai, C. C., & Chang, C. Y. (2006). Attitudes towards peer assessment: A comparison of the perspectives of pre-service and in-service teachers. Innovations in Education and Teaching International, 43(1), 83–92.

Zhang, B., Johnston, L., & Kilic, G. B. (2008). Assessing the reliability of self‐and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329-340.




DOI: http://dx.doi.org/10.24059/olj.v18i2.429

Bookmark and Share