ASSESSING STUDENT LEARNING WITH AUTOMATED TEXT PROCESSING TECHNIQUES

Yi-fang Brook Wu, Xin Chen

Abstract


Research on distance learning and computer-aided grading has been developed in parallel. Little work has been done in the past to join the two areas to solve the problem of automated learning assessment in virtual classrooms. This paper presents a model for learning assessment using an automated text processing technique to analyze class messages with an emphasis on course topics produced in an online class. It is suggested that students should be evaluated on many dimensions, including the learning artifacts such as course work submitted and class participation. Taking all these grading criteria into consideration, we design a model which combines three grading factors: the quality of course work, the quantity of efforts, and the activeness of participation, for evaluating the performance of students in the class. These three main items are measured on the basis of keyword contribution, message length, and message count, and a score is derived from the class messages to evaluate students’ performance. An assessment model is then constructed from these three measures to compute a performance indicator score for each student. The experiment shows that there is a high correlation between the performance indicator scores and the actual grades assigned by instructors. The rank orders of students by performance indicator scores and by the actual grades are highly correlated as well. Evidence from the experiment shows that the computer grader can be a great supplementary teaching and grading tool for distance learning instructors.


Keywords


Virtual Classroom,Distance Learning,Learning Assessment,Course Messages,Text Processing,Noun Phrase

Full Text:

PDF

References


Hiltz, S. R. The Virtual Classroom: Learning Without Limits Via Computer Networks. Norwood NJ: Ablex, 1994.

Lazarus, B. D. Teaching Courses Online: How Much Time Does it Take? Journal of Asynchronous Learning Networks 7(3): September 2003.

Winkler, R. L. and R. T. Clemen. Multiple Experts vs. Multiple Methods: Combining Correlation Assessments. Decision Analysis, November 2002.

Astin, A. W., T. W. Banta, K. P. Cross, E. El-Khawas, P. T. Ewell P. Hutchings, T. J. Marchese, K. M. McClenney, M. Mentkowski, M. A. Miller, E. T. Moran, and B. D. Wright. 9 Principles of Good Practice for Assessing Student Learn-ing. American Association for Higher Education, 2003. Online: http://www.aahe.org/assessment/principl.htm.

Race, P. The Art of Assessment. SEDA publication the New Academic 5(3): 1995.

Picciano, A. Beyond Student Perceptions: Issues Of Interaction, Presence, and Performance In An Online Course. Jour-nal of Asynchronous Learning Networks 6(1): July 2002.

Shea, P., E. Fredericksen, A. Pickett, W. Pelz, and K. Swan. Measures of learning effectiveness in the SUNY Learning Network. In Bourne, J. and Moore, J. C. (eds) Online Education, Volume 2, Needham, MA: Sloan-C, 2001.

Kitchen, D. and D. McDougall. Collaborative learning on the Internet. Journal of Educational Technology Systems 27(3): 1998–99.

Hardless, C., J. Lundin, and U. Nulden Mandatory Participation in Asynchronous Learning Networks. Proceedings of HICSS, Maui, USA, January 2001.

Hiltz, S. R., N. Coppola, N. Rotter, M. Turoff, and R. Benbunan-Fich. Measuring the Importance of Collaborative Learn-ing for the Effectiveness of ALN: A Multi-Measure, Multi-Method Approach. Journal of Asynchronous Learning Net-works 4(2): 2000.

Bloom, B. Taxonomy of educational objectives: The classification of educational goals. Handbook I, cognitive domain. New York: Longman, 1956.

Peat, M. Online assessment: The use of web based self-assessment materials to support self-directed learning. Proceed-ings of the 9th Annual Teaching Learning Forum, 2–4 February 2000.

McKenzie, W. and D. Murphy. “I hope this goes somewhere”: Evaluation of an online discussion group. Australian Journal of Educational Technology 16(3): 239–257, 2000.

Moore, M. Three types of interaction. The American Journal of Distance Education 3(2): 1–6, 1992.

Bodomo, A., K. K. Luke, and A. Anttila. Evaluating Interactivity in Web-Based Learning, Global E-Journal of Open, Flexible and Distance Education 3(1): 2003.

Henri, F. Distance learning and computer mediated communication: Interactive quasi-interactive or monologue. In C. O'Malley (ed.) Computer supported collaborative learning, NATO ASI series, Berlin: Springer-Verlag, 1995.

Page, E. B. Grading essays by computer: Progress report. Notes from the 1966 Invitational Conference on Testing Prob-lems, 87–100, 1966.

Jones, E. L. Grading student programs - a software testing approach. The Journal of Computing in Small Colleges 16(2): 2001.

Page, E. B. New Computer Grading of Student Prose Using Modern Concepts and Software. The Journal of Experi-mental Education 62(2): 127–142, 1994.

Bachman, L. F., N. Carr, G. Kamei, M. Kim, M. J. Pan, C. Salvador, and Y. Sawaki. A reliable approach to automatic assessment of short answer free responses. Proceedings of the 19th International Conference on Computational Linguistics, 2002.

Burstein, J., C. Leacock, and M. Chodorow. CriterionSM: Online essay evaluation: An application for automated evaluation of student essays. Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence, Acapulco, Mexico, August 2003.

Foltz, P. W., D. Laham, and T. K. Landauer. The Intelligent Essay Assessor: Applications to Educational Technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning 1(2): 1999.

Landauer, T. K. and J. Psotka. Simulating Text Understanding for Educational Applications with Latent Semantic Analysis: Introduction to LSA. Interactive Learning Environments 8(2): 73–86, 2000.

Larkey, L. S. Automatic essay grading using text categorization techniques. Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, 1998.

Page, E. B. Computer Grading of Essays: A Different Kind of Testing? Address for APA Annual Meeting, Aug. 13, 1995.

Burstein, J., R. Kaplan, S. Wolff, and C. Lu. Using Lexical Semantic Techniques to Classify Free-Responses. In Proceedings of SIGLEX 1996 workshop, Annual Meeting of the Association of Computational Linguistics, University of California, Santa Cruz, 1996.

Snow, C. E. and C. A. Ferguson. Talking to Children: Language Input and Acquisition. Cambridge: Cambridge University Press, 1997.

Kamp, H. A. Theory of Truth and Semantic Representation. In J. Groenendijk, T. Janssen, and M. Stokhof (eds.) Formal Methods in the Study of Language, Vol. 1. Mathema-tische Centrum, 1981.

Brill, E. and Marcus, M. Tagging an unfamiliar text with minimal human supervision. ARPA Technical Report, 1993.

Brill, E. Unsupervised learning of disambiguation rules for part of speech tagging. Proceedings of the ACL Third Workshop on Very Large Corpora, 1–13. Somerset, New Jersey, 1995.

Schütze, H. Part-of-speech induction from scratch. Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics, 251–258, 1993.

Church, K. W. A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text. In Proceedings of the Second Conference on Applied Natural Language Processing, 136–143, 1988.

Cutting, D., J. Kupiec, J. Pedersen, and P. Sibun. A Practical Part-Of-Speech Tagger. Proceedings of the Third Conference on Applied Natural Language Processing, 1992.

Dermatas, E. and G. Kokkinakis. Automatic stochastic tagging of natural language texts. Computational Linguistics 21(2): 137–163, 1995.

DeRose, S. J. Grammatical category disambiguation by statistical optimization. Computational Linguistics 14(1): 31–39, 1998.

Greene, B. B. and G. M. Rubin. Automatic grammatical tagging of English. Technical Report, Brown University. Providence, RI, 1971.

Brill, E. Transformation-Based Error-Driven Learning and Natural Language Processing: A Case Study in Part of Speech Tagging. Computational Linguistics, 1995.

Fellbaum, C. D. WordNet: An Electronic Lexical Database. MIT Press: Cambridge, MA, 1998.

Wu, Y-f. B. and Chen, X. Assessing Distance Learning Students’ Performance: A Natural Language Processing Approach to Analyzing Online Class Discussion Messages. Proceedings of ITCC, Las Vegas, USA 2004.

Chen, X. and Y-f. B. Wu. Automated Evaluation of Students' Performance by Analyzing Online Messages. Proceedings of IRMA, New Orleans, USA, 2004.

Salton, G. and C. Buckley. Term weighting approaches in automatic text retrieval. Information Processing and Management 24(5): 513–523, 1988.

Salton, G. Automatic Text Processing: The Transformation Analysis, and Retrieval of Information by Computer. Addison Wesley, 1989.

Levenburg, N. M. and H. T. Major. Motivating the Online Learner: The Effect of Frequency of Online Postings and Time Spent Online on Achievement of Learning Goals and Objectives. International Online Conference on Teaching Online in Higher Education, 2000.

Williams, R. Automated essay grading: An evaluation of four conceptual models. In A. Herrmann and M. M. Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum, 7–9 February, 2001.

Wasserman, S. and K. Faust. Social Network Analysis. Cambridge University Press, 1994.

Reffay, C. and T. Chanier. How Social Network Analysis Can Help To Measure Cohesion in Collaborative Distance-Learning. Proceeding of Computer Supported Collaborative Learning Conference, June 2003.




DOI: http://dx.doi.org/10.24059/olj.v9i3.1789



Copyright (c) 2019 Yi-fang Brook Wu, Xin Chen