Measuring Learning Effectiveness: A New Look at No-Significant-Difference Findings

Ernest H. Joy, Federico E. Garcia


Researchers, instructional designers and consumers of ALNs must be cautious when interpreting results of media comparison studies. Much of the literature purports to have found no significant difference in learning effectiveness between technology-based and conventional delivery media. This research, though, is largely flawed. In this paper, we first outline the philosophical positions of the opposing sides of an intense debate in the literature as to whether delivery media alone influence learning outcomes. We then select at random several representative media comparison studies to illustrate the inadequacy of their methodologies and conclusions. More important, we derive critical design considerations for those who evaluate or conduct media comparison research. ALN practitioners should not assume that students would learn better from technology delivery systems. Rather, ALN practitioners should adhere to time-tested instructional design strategies, regardless of the medium they choose. Learning effectiveness is a function of effective pedagogical practices. Accordingly, the question for ALN practitioners ought to be: "What combination of instructional strategies and delivery media will best produce the desired learning outcome for the intended audience?"


ALN,Instructional Assessment,Computer-Aided Instruction

Full Text:



Moore, D. M., Personal communications, Spring 1997.

Schramm, V., Big Media, Little Media: Tools and Technologies for Instruction, Beverly Hills, CA, Sage Publications, 1977.

Maddux, C. D., Research in Educational Computing: Problems of Internal and External Validity, Computers in the Schools, 11(3), 7-10, 1995.

Clark, R. E., Reconsidering Research on Learning from Media, Review of Education Research, Vol. 53, No. 4, pp. 445-459, 1983.

Clark, R. E., Evidence for Confounding in Computer Based Instruction Studies: Analyzing the Meta-Analyses, Educational Technology Research and Development, Vol. 33, No. 4, pp. 235-262, 1985.

Clark, R. E., Media Will Never Influence Learning, Educational Technology Research and Development, Vol. 42, No. 2, pp. 21-29, 1994.

Gagne, R. M., Briggs, L. J., and Wager, W. W., Principles of Instructional Design, 4th ed., Orlando, Harcourt Brace Jovanovich, 1992.

Kozma, R. B., Will Media Influence Learning? Reframing the Debate, Educational Technology Research and Development, Vol. 42, No. 2, pp. 7-19, 1994.

Kozma, R. B., A Reply: Media and Methods, Educational Technology Research and Development, Vol. 42, No. 3, pp. 11-13, 1994.

Russell, T. L., Television’s Indelible Impact on Distance Education: What We Should Have Learned From Comparative Research, Research in Distance Education, October 2-4, 1992.

Rachal, J. D., Computer Assisted Instruction in Adult Basic and Secondary Education: A Review of the Experimental Literature 1984-1992, Adult Education Quarterly, Vol. 43, No. 3, pp 165-172, 1993.

Reeves, T. C., Pseudoscience in Computer-Based Instruction: The Case of Learner Control Research, Journal of Computer Based Instruction, Vol. 20, No. 2, pp 39-46, 1993.

Sivin-Kachala, J., and Bialo, E. R., Report on the Effectiveness of Technology in Schools, 1990-1994. New York: Interactive Educational Systems Design, Inc., 1994.

Reglin, G. L., CAI Effects on Mathematics Achievement and Academic Self-Concept Seminar, Journal of Education Technology Systems, Vol. 18, No. 1, pp. 43-48, 1989.

Clements, D. H., Enhancement of Creativity in Computer Environments, American Education Research Journal, Vol. 28, No. 1, pp. 173-187, 1991.

Skinner, M. E., The Effects of Computer-Based Instruction on the Achievement of College Students as a Function of Achievement Status and Mode of Presentation, Computers in Human Behavior, pp. 351-360, 1990.

Gardner, C. M., Simmons, P. E., and Simpson, R. D., The Effects of CAI and Hands-on Activities on Elementary Students’ Attitudes and Weather Knowledge, School Science and Mathematics, Vol. 92, No. 6, pp. 334-336, 1992.

Funkhouser, C. P., The Influence of Problem-Solving Software on Student Attitudes about Mathematics, Journal of Research on Computing in Education, Vol. 25, No. 3, pp. 339-346, 1993.

Orlansky, J., The Cost Effectiveness of Military Training. Paper presented at the Symposium on Military Value and Cost-Effectiveness of Training, Brussels, Belgium, 1985.


Copyright (c) 2019 Ernest H. Joy, Federico E. Garcia