THE METHOD (AND MADNESS) OF EVALUATING ONLINE DISCUSSIONS

Katrina A. Meyer

Abstract


In addressing how to evaluate online discussions, this paper will describe several concepts, tools, or frameworks that have been used in evaluations and discuss differences in approach based on instructor purpose, be it research, assessment, or learning. Then several common problems are described, including use of content analysis, identification of latent content, inadequate training of coders, lack of reliability, and choosing the correct unit of analysis. Two examples are provided of coding decisions made on portions of student discussions; these examples use two different frameworks to elucidate the process and its difficulties. Conclusions focus on the importance of following standard good research or assessment practice and preparing for a time-consuming and often frustrating coding process.


Keywords


Online Learning,Evaluation of Online Discussions,Methodology

Full Text:

PDF

References


Moore, M. G. Three types of interaction. In K. Harry, M. John, and D. Keegan (Eds.), Distance Education: New Perspectives, 19–24. New York: Routledge, 1993.

Chen, Y. and F. K. Willits. A path analysis of the concepts in Moore’s theory of transactional distance in a videoconferencing learning environment. Journal of Distance Education 3(2): 1998.

Garrison, D. R., T. Anderson and W. Archer. Critical thinking, cognitive presence, and computer conferencing in distance education. The American Journal of Distance Education 15(1): 7–23, 2001.

Garrison, R. Theoretical challenges for distance education in the 21st century: A shift from structural to transactional issues. International Review of Research in Open and Distance Learning 1(1): 2000.

Gunawardena, C. N., C. A. Lowe and T. A. Anderson. Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research 17(4): 397–431, 1997.

Newman, D. R., B. Webb, and C. Cochrane. A content analysis method to measure critical thinking in face-to-face and computer supported group learning. Interpersonal Computing and Technology 3(2): 56–77, 1995.

Treleaven, L. Evaluating a communicative model for web mediated collaborative learning and design. Australian Journal of Educational Technology 19(1): 100–117, 2003.

Aviv, R. Educational performance of ALN via content analysis. Journal of Asynchronous Learning Networks 4(2): 53–72, 2000.

Aviv, R., Z. Erlich, G. Ravid and A. Geva. Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks 7(3): 1–23, 2003.

Brown, R. E. The process of community-building in distance learning classes. Journal of Asynchronous Learning Networks 5(2): 2001.

Edelstein, S. and J. Edwards. If you build it, they will come: Building learning communities through threaded discussions. Online Journal of Distance Learning Administration 5(1): 2002.

Fahy, P. J. Indicators of support in online interaction. International Review of Research in Open and Distance Learning 4(1): 2003.

Campos, M., T. Laferrière and L. Harasim. The post-secondary networked classroom: Renewal of teaching practices and social interaction. Journal of Asynchronous Learning Networks 5(2): 2001.

Jeong, A. C. The sequential analysis of group interaction and critical thinking in online threaded discussions. The American Journal of Distance Education 17(1): 25–43: 2003.

Muirhead, B. Enhancing social interaction in computer-mediated distance education. Ed at a Distance 15(40): 2001.

Hillman, D. C. A. A new method for analyzing patterns of interaction. The American Journal of Distance Education 13(2): 37–47, 1999.

Poscente, K. R. and P. J. Fahy. Investigating triggers in CMC text transcripts. International Review of Research in Open and Distance Learning 4(2): 2003.

Spatariu, A., K. Hartley and L. Bendixen. Defining and measuring quality in online discussions. Journal of Interactive Online Learning 2(4): 2004. http://www.ncolr.org/jiol/issues/PDF/2.4.2.pdf.

Wegerif, R. The social dimension of asynchronous learning networks. Journal of Asynchronous Learning Networks 2(1): 1998.

Luppicini, R. J. Toward a conversation system modeling research methodology for studying computer-mediated learning communities. Journal of Distance Education 17(2): 87–101, 2002.

Meyer, K. A. The ebb and flow of online discussions: What Bloom can tell us about our students’ conversations. Journal of Asynchronous Learning Networks 9(1): 2005.

Brown, G. Guide to Rating Critical Engagement. Pullman, WA: Washington State University. Published online via subscription, September 2004.

Gunawardena, C. N. and F. J. Zittle. Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. The American Journal of Distance Education 11(3): 6–26, 1997.

Oren, A., D. Mioduser and R. Nachmias. The development of social climate in virtual learning discussion groups. International Review of Research in Open and Distance Learning 3(1): 2002.

Richardson, J. C. and K. Swan. Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Journal of Asynchronous Learning Networks 7(1): 68–88, 2003.

Rourke, L. and T. Anderson. Exploring social interaction in computer conferencing. Journal of Interactive Learning Research 13(3): 257–273, 2002.

Rourke, L., T. Anderson, D. R. Garrison and W. Archer. Assessing social presence in asynchronous, text-based computer conferencing. Journal of Distance Education 14(3): 51–70, 1999. http://cade.athabascau.ca/vol14.2/rourke_et_al.html.

Tu, C. and M. McIsaac. The relationship of social presence and interaction in online classes. The American Journal of Distance Education 16(3): 131–150, 2002.

Anderson, T., L. Rourke, D. R. Garrison and W. Archer. Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks 5(2): 2001.

Shea, P. J., A. M. Pickett and W. E. Pelz. A follow-up investigation of “teaching presence” in the SUNY Learning Network. Journal of Asynchronous Learning Networks 7(2): 61–80, 2003.

Hutchins, H. M. Instructional immediacy and the seven principles: Strategies for facilitating online courses. The Online Journal of Distance Learning Administration 6(3): 2003. http://www.westga.edu/~distance/ojdla/fall2003/hutchins63.html.

Gerbic, P. and E. A. Stacey. Purposive approach to content analysis: Designing analytical frameworks. The Internet and Higher Education 8(1): 45–59.

Meyer, K. A. Evaluating online discussions: Four different frames of analysis. Journal of Asynchronous Learning Networks 8(2): 2004.

Perry, W. G. Forms of Ethical and Intellectual Development in the College Years: A Scheme. San Francisco, CA: Jossey-Bass, 1999.

King, P. M. and K. S. Kitchener. Developing Reflective Judgment. San Francisco, CA: Jossey-Bass, 1994.

Bloom, B. S. and D. R. Krathwohl. Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: Longmans, Green, 1956.

Krumme, G. Major Categories in the Taxonomy of Educational Objectives (Bloom 1956). http://faculty.washington.edu/krumme/guides/bloom.html.

Anderson, L. W. and D. R. Krathwohl (Eds.). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. New York: Longman, 2001.

Carnevale, D. Introverts do well in online chats. The Chronicle of Higher Education 50(16): A29, 2003.

Rovai, A. P. and K. T. Barnum. On-line course effectiveness: An analysis of student interactions and perceptions of learning. Journal of Distance Education 18(1): 57–73, 2003.

Fahy, P. J. Use of linguistic qualifiers and intensifiers in a computer conference. The American Journal of Distance Education 16(1): 5–22, 2002.

De Simone, C., Y. Lou and R. F. Schmid. Meaningful and interactive learning supported by the use of metaphor and synthesizing activities. Journal of Distance Education 16(1): 2001.

Roblyer, M.D. and W. R. Wiencke. Design and use of a rubric to assess and encourage interactive qualities in distance courses. The American Journal of Distance Education 17(2): 77–98, 2003.

Roblyer, M. D. and L. Ekhaml. How interactive are YOUR distance courses? A rubric for assessing interaction in distance learning. The Online Journal of Distance Learning Administration 3(2): 2000.

WSU Critical Thinking Project. The Critical Thinking Rubric. Pullman, WA: Washington State University. http://wsuctproject.wsu.edu/ctr.htm.

Moore, W. S. and E. Rousso. Principles for Designing and Evaluating the Quality of Online Assignments/Assessments. Unpublished document.

Goetz, J. P. and M. D. LeCompte. Ethnography and qualitative design in educational research. Orlando, FL: Academic Press, 1984.

Rourke, L., T. Anderson, D. R. Garrison and W. Archer. Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education 12: 8–22, 2001. http://aied.inf.ed.ac.uk/members01/archive/vol_12/rourke/full.html.

Holsti, O. Content analysis for the social sciences and humanities. Reading, MA: Addison-Wesley, 1969.




DOI: http://dx.doi.org/10.24059/olj.v10i4.1746



Copyright (c) 2019 Katrina A. Meyer