Adapting for Scalability: Automating the Video Assessment of Instructional Learning

Amy M Roberts, Jennifer LoCasale-Crouch, Bridget K Hamre, Jordan M Buckrop


Although scalable programs, such as online courses, have the potential to reach broad audiences, they may pose challenges to evaluating learners’ knowledge and skills. Automated scoring offers a possible solution. In the current paper, we describe the process of creating and testing an automated means of scoring a validated measure of teachers’ observational skills, known as the Video Assessment of Instructional Learning (VAIL). Findings show that automated VAIL scores were consistently correlated with scores assigned by the hand scoring system. In addition, the automated VAIL replicated intervention effects found in the hand scoring system. The automated scoring technique appears to offer an efficient and reliable assessment. This study may offer additional insight into how to utilize similar techniques in other large-scale programs and interventions.


Automated assessment; scalability; teacher education

Full Text:



Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Upper Saddle River, NJ: Prentice-Hall, Inc.

Bierman, K. L., Nix, R. L., Greenberg, M. T., Blair, C., & Domitrovich, C. E. (2008). Executive functions and school readiness intervention: Impact, moderation, and mediation in the Head Start REDI program. Development and Psychopathology, 20, 821-843. doi: 10.1017/S0954579408000394

Biggs, J., & Tang, C. (2011). Teaching for quality learning at university. New York: McGraw-Hill International.

Boston, C. (2002). The concept of formative assessment. ERIC Digest. Retrieved from ERIC database. (ED470206).

Burchinal, M., Howes, C., Pianta, R., Bryant, D., Early, D., Clifford, R., & Barbarin, O. (2008). Predicting child outcomes at the end of kindergarten from the quality of pre-kindergarten teacher-child interactions and instruction. Applied Developmental Science, 12(3), 140-153. doi: 10.1080/10888690802199418

Burnstein, J., Chodorow, M. & Leacock, C. (2003). Criterion online essay evaluation: An application for automated evaluation of student essays. Retrieved from American Association for Artificial Intelligence:

Clauser, B. E., Ross, L. P, Clyman, S. G., Rose, K. M., Margolis, M. J. …Pincetl, P. S. (1997). Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment, Applied Measurement in Education, 10, 345-358. doi: 10.1207/s15324818ame1004_3

Condon, W. (2013). Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings? Assessing Writing, 18, 100-108. doi: 10.1016/j.asw.2012.11.001

Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning, and Assessment, 5(1).

Domitrovich, C. E., Gest, S. D., Gill, S., Jones, D., & DeRousie, R. S. (2009). Individual factors associated with professional development training outcomes of the Head Start REDI Program. Early Education & Development, 20, 402-430. doi: 10.1080/10409280802680854

Downer, J. T., Pianta, R. C., Burchinal, M., Field, S., Hamre. B. K. …Scott-Little, C. (in press). Coaching and coursework focused on teacher-child interactions during language/literacy instruction: Effects on teacher beliefs, knowledge, skills, and practice.

Foddy, W. (1993). Constructing questions for interviews and questionnaires: Theory and practice in social research. Cambridge: Cambridge University Press.

Franks, R. P. & Schroder, J. (2013). Implementation science: What do we know and where do we go from here? In T. Halle, A. Metz, & I. Martinez-Beck (Eds.) Applying implementation science in early childhood programs and systems (pp 5-19). Brookes Publishing: Baltimore.

Gill, W. E. (2011). The Ready to Teach program. A federal initiative in support of online courses for teachers. Retrieved from

Hamre, B. K., Downer, J. T., Jamil, F. M., & Pianta, R. C. (2012). Enhancing teachers’ intentional use of effective interactions with children. In R. C. Pianta (Ed.) Handbook of early childhood education (pp 507-532). New York: The Guilford Press.

Hamre, B. K., Pianta, R. C., Burchinal, M., Field, S., LoCasale-Crouch, J., Downer, J. T., . . . Scott-Little, C. (2012). A Course on Effective Teacher-Child Interactions: Effects on Teacher Beliefs, Knowledge, and Observed Practice. American Educational Research Journal, 88-123. doi: 10.3102/0002831211434596

Jamil, F. M., Sabol, T. J., Hamre, B. K., & Pianta, R. C. (2015). Assessing teachers' skills in detecting and identifying effective interactions in the classroom: Theory and measurement. The Elementary School Journal, 115(3), 407-432. doi: 10.1086/680353

Landauer, T. K., Laham, D. & Foltz, P. (2003). Automatic essay assessment. Assessment in Education, 10(3), 295-308. doi: 10.1080/0969594032000148154

Leacock, C. & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37, 389-405.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. US Department of Education.

Miller, K. (2011). Situation awareness in teaching: What educators can learn from video-based research in other fields. In M. Sherin, V. Jacobs, & R. Phillip (Eds.) Mathematics teacher noticing: Seeing through teachers’ eyes (pp. 51-65). New York: Routledge.

Palloff, R. M. & Pratt, K. (2008). Assessing the online learner: Resources and strategies for faculty. San Francisco: Jossey-Bass.

Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). Linguistic Inquriy and Word Count: LIWC2007 Operator’s Manual.

Perelman, L. (2014). When “the state of the art” is counting words. Assessing Writing, 21, 104-111. doi: 10.1016/j.asw.2014.05.001

Pianta, R. C., Hamre, B. K., & Hadden, D. S. (2012). Scaling up effective professional development. In C. Howes, B. Hamre, & R. Pianta (Eds.) Effective early childhood professional development: Improving teacher practice and child outcomes (pp.191-212). Baltimore: Brookes Publishing Company.

Pianta, R., La Paro, K., & Hamre, B. K. (2008). Classroom Assessment Scoring System. Baltimore: Brookes Publishing Company.

Pianta, R., Mashburn, A., Downer, J. Hamre, B., & Justice, L. (2008). Effects of Web-mediated professional development resources on teacher-child interactions in pre-kindergarten classrooms. Early Childhood Research Quarterly, 23(4), 431-451. doi: 10.1016/j.ecresq.2008.02.001.

Ramineni, C., & Williamson, D. M. (2013). Automated essay scoring: Psychometric guidelines and practice. Assessing Writing, 18, 25-39. doi: 10.1016/j.asw.2012.10.004.

Thomason, A.C. & La Paro, K.M. (2009). Measuring the quality of teacher-child interactions in toddler child care. Early Education & Development, 20(2), 285-304. doi: 10.1080/10409280902773351

U.S. Department of Education, National Center for Education Statistics. (2016). Digest of Education Statistics, 2014 (NCES 2016-006), Table 311.15.

Vale, K & Littlejohn, A. (2014). Massive open online courses: A traditional or transformative approach to learning? In A. Littlejohn & C. Pegler (Eds.) Reusing open resources: Learning in open networks for work, life and education (pp. 138-153). New York: Routledge.

Wiens, P., Hessberg, K., LoCasale-Crouch, J., & DeCoster, J. (2013). Using a standardized video-based assessment in a university teacher education program to examine pre-service teachers knowledge related to effective teaching. Teaching and Teacher Education, 33, 24-33. doi: 10.1016/j.tate.2013.01.010

Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2-13. doi: 10.1111/j.1745-3992.2011.00223.x

Xi, X., Higgins, D., Zechner, K., & Williamson, D. (2012). A comparison of two scoring methods for an automated speech scoring system. Language Testing, 29(3), 371-394. doi: 10.1177/0265532211425673

Yoshikawa, H., Weiland, C., Brooks-Gunn, J., Burchinal, M. R., Espinosa, L. M., Gormley, W. T.,…Zaslow, M. J. (2013). Investing in our future: The evidence base on preschool education. Retrieved from Society for Research in Child Development: