The Efficacy of an Online Cognitive Assessment Tool for Enhancing and Improving Student Academic Outcomes

Lindsay Shaw, Jann MacIsaac, Jill A. Singleton-Jackson

Abstract


With technology at the fingertips of most undergraduate students, it has been difficult for instructors to fully engage students in the classroom, which has resulted in the creation of several innovative assessment platforms, such as mTuner. mTuner integrates several cognitive learning strategies within an assessment, with the goal of actually enhancing learning, as opposed to just measuring it. In the current study, students’ level of engagement and performance with mTuner were compared to their final multiple-choice paper and pencil exam mark to determine the efficacy of mTuner in achieving improved learning outcomes. Results indicated that students had high performance scores on the mTuner assessment despite their limited engagement in the cognitive learning features, putting to question mTuner’s facilitation of long-term learning. Implications and future recommendations of mTuner implementation in educational environments are discussed.


Keywords


Online assessment, student engagement with online tools, online learning

Full Text:

PDF

References


References

Baleni, Z. (2015). Online formative assessment in higher education: its pros and cons. The

Electronic Journal of e-Learning, 13(40, 228-236.

Beck, H.P., Rorrer-Woody, S., & Pierce, L. (1991). The relations of learning and grade

orientations to academic performance. Teaching of Psychology, 18(1), 35-37.

Birenbaum, M., & Feldman, R.A. (1998). Relationships between learning patterns and attitudes

towards two assessment formats. Educational Research, 40(1), 90-97.

Cain, J., Romanelli, F., & Smith, K.M. (2012). Academic entitlement in pharmacy education.

American Journal of Pharmaceutical Education, 76(10), 1-8.

Carpenter, S.K., & DeLosh, E.L. (2006). Impoverished cue support enhances subsequent

retention: Support for the elaborative retrieval explanation of the testing effect. Memory

& Cognition, 34, 268-276.

Cassady, J.C. (2004). The impact of cognitive test anxiety on test comprehension and recall in

the absence of external evaluative pressure. Applied Cognitive Psychology, 18(3), 311

Cassady, J.C., & Johnson, R.E. (2002). Cognitive test anxiety and academic performance.

Contemporary Educational Psychology, 27(2), 270 295.

Chappuis, S., Stiggins, R., Arter, J., & Chappuis, J. 2004. Assessment for learning: An action

guide for school leaders. Portland, OR: Assessment Training Institute.

Coates, H. (2007). A model of online and general campus-based student engagement. Assessment

and Evaluation in Higher Education, 32(2), 121-141.

Cohen, J. Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences, 3rd ed. Mahwah, NJ: Lawrence Erlbaum.

DeCaro, M. S., Thomas, R. D., Albert, N. B., & Beilock, S. L. (2011). Choking under pressure:

Multiple routes to skill failure. Journal of Experimental Psychology: General, 140(3),

Dermo, J. (2009). E-Assessment and the student learning experience. A survey of student

perceptions of e-assessment. British Journal of Educational Technology, 40(2), 203-214.

Eison, J.A., Pollio, H.R., & Milton, O. (1983). Manual for use with LOGO-II. Theory and

Practice in Teacher Education and Other Works. Retrieved from:

http://trace.tennessee.edu/utk_theopubs/2

Eison, J.A., Pollio, H.R., & Milton, O. (1986). Educational and personal characteristics of four

different types of learning- and grade-oriented students. Contemporary Educational

Psychology, 11, 54-67.

Everson, H.T., Smodlake, I., & Tobias, S. (1995). Exploring the relationship of test anxiety and

metacognition on reading test performance: A cognitive analysis. Anxiety, Stress and

Coping, 7, 85-96.

Frymier, A.B., & Weser, B. (2001). The role of student predispositions on student expectations

for instructor communication behaviour. Communication Education, 50, 314-326.

Furlan, L.A., Cassady, J.C., & Pérez, E.R. (2009). Adapting the cognitive test anxiety scale for

use with Argentinean university students. International Journal of Testing, 9(1), 3-19.

Furnham, A., Batey, M., & Martin, N. (2011). How would you like to be evaluated? The

correlates of students’ preferences for assessment methods. Personality and Individual

Differences, 50, 259-263.

Gikandi, J.W., Morrow, D., & Davis, N.E. (2011). Online formative assessment in higher

education: A review of the literature. Computers & Education, 57, 2333-2351.

Greenberger, E., Lessard, J., Chen, C., & Farruggia, S.P. (2008). Self-entitled college students:

Contributions of personality, parenting and motivational factors. Journal of Youth and

Adolescence, 37,1193-1204.

Holbrook, J., Dupont, C., Power, M., & Joordens, S. (2015). Enhancing learning through

testing. Presented at the OND Centre for Teaching Excellence Conference, Waterloo,

Ontario. Retrieved from https://uwaterloo.ca/centre-for-teaching excellence /sites/ca.centre-for-teaching excellence/files/uploads/files/mtuner_ond_2015_final.pdf.

Kalechstein, P., Hocevar, D., Zimmer, J.W., & Kalechstein, M. (1989). Procrastination over test

preparation and test anxiety. In R. Schwarzer, H. M. van der Ploeg, & G. D. Spielberger

(Eds.), Advances in test anxiety research. Vol. 6. (pp. 63–76). Lisse, The Netherlands:

Swets & Zeitlinger.

Karpicke, J.D. (2012). Retrieval-based learning: Active retrieval promotes meaningful learning.

Current Directions in Psychological Science, 21(3), 157-163.

Karpicke, J.D., & Roediger, H.L. (2007). Repeated retrieval during learning is the key to long

term retention. Journal of Memory and Language, 57, 151-162.

Karpicke, J.D., & Roediger, H.L. (2008). The critical importance of retrieval for learning.

Science, 319, 966-968.

Karpicke, J.D., & Zaromb, F.M. (2010). Retrieval mode distinguishes the testing effect from the

generation effect. Journal of Memory and Language, 62, 227-239.

Kember, D., Ho, A., & Hong, C. (2008). The importance of establishing relevance in motivating

student learning. Active Learning in Higher Education, 9, 249-263.

Kopp, J.P., Zinn, T.E., Finney, S.J., & Jurich, D.P. (2011). The development and evaluation of

the academic entitlement questionnaire. Measuring and Evaluation in Counseling and

Development, 44(2), 105-129.

Kornell, N., Hays, M.J., & Bjork, R.A. (2009). Unsuccessful retrieval attempts enhance

subsequent learning. Journal of Experimental Psychology: Learning, Memory, and

Cognition, 35(4), 989-998.

Kuh, G.D., & Hu, S. (2001). The effects of student faculty interaction in the 1990s. Review of

Higher Education, 24(30), 309-332.

Lawrence, K.R., & Frymier, A.B. (2002). The relationship of student-centered teaching with

learning and grade orientation. Paper presented at the annual meeting of the National

Communication Association, New Orleans, LA.

Levine, N. B. (2003). A multidimensional scaling analysis of perceptions of academic dishonesty

among learning- and grade-oriented students. Dissertation Abstracts International,

(4), 1175.

Marsden, H., Carroll, M., & Neill, J.T. (2005). Who cheats at university? A self-report study of

dishonest academic behaviours in a sample of Australian university students. Australian

Journal of Psychology, 57(1), 1-10.

Marsh, E.J., Roediger, H.L, Bjork, R.A., & Bjork, E.L. (2007). Memorial consequences of

multiple-choice testing. Psychonomic Bulletin & Review, 14, 194-199.

McDaniel, M.A., & Masson, M.E.J. (1985). Altering memory representations through retrieval.

Journal of Experimental Psychology: Learning, Memory and Cognition, 11, 371-385.

Merrel, J.D., Cirillo, P.F., Schwartz, P.M., Webb, J.A. (2015). Multiple-choice testing using

immediate feedback-assessment technique forms: Second-chance guessing vs. second

chance learning. Higher Education Studies, 5(5), 50-55.

Miller, T. (2009). Formative computer-based assessment in higher education: The effectiveness

of feedback in supporting student learning. Assessment & Evaluation in Higher

Education, 34(2), 181-192.

Noll, C.L., & Wilkins, M. (2002) Critical skills of IS professionals: A model for curriculum

development. Journal of Information Technology Education, 1(3), 144-154.

Oosterhof, A., Conrad, R.M., & Ely, D.P. (2008). Assessing learners online. New Jersey:

Pearson.

Özden, M.Y., Ertürk, I., & Sanli, R. (2004). Students’ perceptions of online assessment: a case

study. Journal of Distance Education, 19(2), 77-92.

Pare, D.E., Joordens, S. (2009). The invisible researcher: Using educational technologies as

research tools for education. Systemics, Cybernetics and Informatics, 7(5), 37-42.

Plass, J.A., & Hill, K.T. (1986). Children’s achievement strategies and test performance: The

role of time pressure, evaluation anxiety, and sex. Developmental Psychology, 22, 31-36.

Purcell, J.M. (2010). Learning- and grade-orientations of community college students:

Implications for instruction. Community College Journal of Research and Practice, 34,

-511.

Race, P. (2005). Making Learning Happen. London: Sage.

Roedel, T.D., Schraw, G., & Plake, B.S. (1994). Validation of a measure of learning and

performance goal orientations. Educational and Psychological Measurement, 54(4),

-1021.

Roediger, H.L., & Karpicke, J.D. (2006a). The power of testing memory: Basic research and

implications for educational practice. Perspectives on Psychological Science, 1, 181-210.

Saade, R.G., Morin, D., & Thomas, J.D. (2012). Critical thinking in e-learning environments.

Computers in Human Behaviour, 28, 1608-1617.

Scouller, K. (1998). The influence of assessment method on students’ learning approaches:

Multiple choice question examination versus assignment essay. Higher Education, 35,

–472.

Scouller, K., & Prosser, M. (1994). Student’s experiences in studying for multiple choice

question examinations. Studies in Higher Education, 19, 267–279.

Smith, G. (2007). How does student performance on formative assessment relate to

learning assessed by exams? Journal of College Science Teaching, 36(7), 28-34.

Smith, S.D., & Caruso, J.B. (2010). The ECAR study of undergraduate students and information

technology. ECAR Research Study 6, Center for Applied Research, Boulder: CO.

Stöeber, J. (2004). Dimensions of test anxiety: Relations to ways of coping with pre-exam

anxiety and uncertainty. Anxiety, Stress & Coping, 17(3), 213–226.

Struyven, K., Dochy, F., & Janssens, S. (2005). Students’ perceptions about evaluation and

assessment in higher education: A review. Assessment and Evaluation in Higher

Education, 30, 325–341.

Tendhar, C., Culver, S.M., & Burge, P.L. (2013). Validating the national survey of student

engagement (NSSE) at a research-intensive university. Journal of Education and

Training Studies, 1(1), 182-193.

Thomas, C.L., Cassady, J.C., & Heller, M.L. (2017). The influence of emotional intelligence,

cognitive test anxiety, and coping strategies on undergraduate academic performance.

Learning and Individual Differences, 55, 40-48.

Tippin, G.K., Lafreniere, K., & Page, S. (2012). Student perception of academic grading:

personality, academic orientation and effort. Active Learning in Higher Education, 13(1),

-61.

Traub, R.E. & MacRury, K. (1990). Multiple-choice vs. free response in the testing of scholastic achievement, in: K. Ingenkamp & R.S. Jager (Eds) Test und tens 8: jahrbuch der pädagogischen diagnostik (Weinheim und Base, Beltz Verlag), 128–159.

Tulving, E. (1974). Cue-dependent forgetting. American Scientist, 62(1), 74-82.

Walker, D.J., Topping, K., & Rodrigues, S. (2008). Student reflections on formative e

assessment: Expectations and perceptions. Learning, Media and Technology, 33, 221

Williams, K.D., & Frymier, A.B. (2007). The relationship between student educational

orientation and motives for out-of-class communication. Communication Research

Reports, 24, 249-256.

Zeidner, M., & Matthews, G. (2005). Evaluation anxiety. In A. J. Elliot, & C. S. Dweck (Eds.),

Handbook of competence and motivation (pp. 141–163). London: Guildford Press.




DOI: http://dx.doi.org/10.24059/olj.v23i2.1490



Copyright (c) 2019 Lindsay Shaw, Jann MacIsaac, Jill A. Singleton-Jackson