Developing Learning Analytics Design Knowledge in the “Middle Space”: The Student Tuning Model and Align Design Framework for Learning Analytics Use

Alyssa Friend Wise, Jovita Maria Vytasek, Simone Hausknecht, Yuting Zhao

Abstract


This paper addresses a relatively unexplored area in the field of learning analytics: how analytics are taken up and used as part of teaching and learning processes. Initial steps are taken towards developing design knowledge for this “middle space,” with a focus on students as analytics users. First, a core set of challenges for analytics use identified in the literature are compiled. Then, a process model is presented for conceptualizing students’ learning analytics use as part of a self-regulatory cycle of grounding, goal-setting, action and reflection–the Student Tuning Model. Finally, the Align Design Framework is presented with initial validation as a tool for pedagogical design that addresses the identified challenges and supports students’ use of analytics as part of the tuning process. Together, the framework’s four interconnected principles of Integration, Agency, Reference Frame and Dialogue / Audience provide a useful starting point for further inquiry into well-designed learning analytics implementations.

Keywords


Learning analytics, pedagogical design, analytics use, self-directed learning

Full Text:

PDF

References


Aguilar, S. J. (2014, March). Exploring and measuring students' sense-making practices around

representations of their academic information. Paper presented at Learning analytics and Knowledge Conference 2014, Indianapolis, IN, USA.

Andrusyszyn, M. A., & Davie, L. (1997). Facilitating reflection through interactive

journal writing in an online graduate course: A qualitative study. The Journal of

Distance Education, 12(1/2), 103-126.

Arnold, K. E., & Pistilli, M. D. (2012, April). Course signals at Purdue: using learning analytics

to increase student success. In Proceedings of the 2nd International Conference on Learning

Analytics and Knowledge (pp. 267-270). ACM.

Auerbach, C. F., & Silverstein, L. B. (2003). Qualitative data: An introduction to coding and

analysis. New York: New York University Press.

Avramides, K., Hunter, J., Oliver, M. & Luckin, R. (2014). A method for teacher inquiry in

cross-curricular projects: lessons from a case study. British Journal of Educational Technology, 46, 2, 249—264.

Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. In

Learning Analytics (pp. 61-75). Springer New York.

Boekaerts, M., Pintrich, P. & Zeidner, M. (2000). Handbook of self-regulation, 417-450. San

Diego, CA: Academic Press.

Booth, M. (2012, July/August). Learning analytics: The new black. EDUCAUSE Review, 47(4),

-53

boyd, d., & Crawford, K. (2011, September). Six Provocations for Big Data. Paper

presented at A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society, Oxford Internet Institute, UK. Retrieved 12 June, 2012, from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431

Brooks, C., Greer, J., & Gutwin, C. (2014). The Data-Assisted Approach to Building

Intelligent Technology-Enhanced Learning Environments. In Learning Analytics (pp. 123-156). Springer New York.

Buckingham Shum, S, & Deakin Crick, R. (2012). Learning dispositions and

transferable competencies: pedagogy, modelling and learning analytics. In

Proceedings of the 2nd International Conference on Learning Analytics and

Knowledge (pp. 92-101). ACM.

Buckingham Shum, S. (2012). Our learning analytics are our pedagogy. Keynote

address given at Expanding Horizons 2012. Macquire University, Australia.

Buckingham Shum, S. & Ferguson, R. (2012). Social Learning Analytics. Educational

Technology & Society, 15(3), 3-26.

Buder, J. (2011). Group awareness tools for learning: Current and future directions. Computers

in Human Behavior, 27(3), 1114-1117.

Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A

Theoretical Synthesis. Review of Educational Research, 65(3), 245 -281.

Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new

tool for a new era. EDUCAUSE review, 42(4), 40.

Campbell, J.P. & Oblinger, D.G. (2007) Academic Analytics, EDUCAUSE Quarterly. October.

Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2012). A reference model for

learning analytics. International Journal of Technology Enhanced Learning, 4(5), 318-331.

Clow, D. (2012). The learning analytics cycle: closing the loop effectively. doi:10.1145/2330601.2330636

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683—695. doi:10.1080/13562517.2013.827653Cuban, L. (1986). The classroom use of technology since 1920. New York: Teachers College Press, Columbia University.

Colthrope, K., Zimbardi, K., Ainscough, L., & Anderson, S. (2015). Know Thy Student!

Combining Learning Analytics and Critical Reflections to Increase Understanding of Students’ Selfâ€Regulated Learning in an Authentic Setting. Journal of Learning Analytics, 2(1), 134—155

Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered

through learning analytics dashboards. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 629-633)

Cuban, L., (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press.

Cuban, L. 1986. The classroom use of technology since 1920. Teachers College Press, New York, NY.

Cutumisu, M., Blair, K. P., Chin, D. P., & Schwartz, D. L. (2015). Posterlet: A Gameâ€Based Assessment of Children's Choices to Seek Feedback and to Revise. Journal of Learning Analytics, 2(1), 49—71.

Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for

educational inquiry. Educational Researcher, 5-8.

Ferguson, R. (2012). Learning analytics: drivers, developments and challenges.International Journal of Technology Enhanced Learning, 4(5), 304-317.

Ferguson, R., & Buckingham Shum, S. (2011). Learning analytics to identify exploratory dialogue within synchronous text chat. In Proceedings of the 1st International Conference on Learning Analytics & Knowledge, (Banff, Canada)

Ferguson, R., Buckingham Shum, S., & Deakin Crick, R. (2011). EnquiryBlogger: using

widgets to support awareness and reflection in a PLE Setting.

Ghislandi, P. M., & Raffaghelli, J. E. (2015). Forwardâ€oriented designing for learning as a means

to achieve educational quality. British Journal of Educational Technology, 46(2), 280-299.

Gibson, J. J. (1977). The theory of affordances. In R. E. Shaw & J. Bransford (Eds.), Perceiving,

acting and knowing: Towards an ecological psychology. (pp.67-82) Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012). The student activity meter for

awareness and self-reflection. In CHI'12 Extended Abstracts on Human

Factors in Computing Systems (pp. 869-884). ACM. Hlupic, V., Pouloudi, A., and Rzevski, G. (2002). Towards an integrated approach to knowledge management: ‘Hard’, ‘soft’ and ‘abstract’ issues. Knowledge and Process Management, 9(2), 90-102.

Govaerts, S., Verbert, K., Klerkx, J., & Duval, E. (2010). Visualizing Activities for Self-

reflection and Awareness. In Proceedings of the 9th international conference on Web-based Learning. (pp. 91—100). Springer, 2010.

Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for

learning analytics. Journal of Educational Technology & Society, 15(3), 42-57.

Haya, P. A., Daems, O., Malzahn, N., Castellanos, J., & Hoppe, H. U. (2015). Analysing content

and patterns of interaction for improving the learning design of networked learning environments. British Journal of Educational Technology, 46(2), 300-316.

Johnson, L., Becker, S., Estrada, V., & Freeman, A. (2014). Horizon Report: 2014 Higher

Education. Austin, TX: The New Media Consortium.

Janssen, J., & Bodemer, D. (2013). Coordinated computer-supported collaborative learning:

Awareness and awareness tools. Educational Psychologist, 48(1), 40-55.

Kolb, D.A. (1984). Experiential education: Experience as the source of learning and

development. Englewood cliffs, NJ: Prentice Hall.

Kruse, A. N. N. A., & Pongsajapan, R. (2012). Student-centered learning analytics.

CNDLS Thought Papers, 1-9.

Kuhn, T. S. (1962). The Structure of Scientific Revolutions. Chicago : University of Chicago Press.

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning

Learning Analytics with Learning Design. American Behavioral Scientist, 57, 1439—1459. doi:10.1177/0002764213479367

Locke, E. A., & Latham, G. P. (2006). New directions in goal-setting theory. Current Directions

in Psychological Science, 15(5), 265-268.

M'hammed Abdous, He, W., & Yen, C. J. (2012). Using data mining for predicting relationships between online question theme and final grade. Journal of Educational Technology & Society, 15(3), 77-88.Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education,54(2), 588-599.

Macfadyen, L., & Dawson, S. (2012). Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic Plan. Educational Technology & Society, 15(3), 149-163.

Manouselis, N., Drachsler, H., Verbert, K., & Duval, E. (2012). Recommender systems for learning. Springer Science & Business Media.

Mathan, S., & Koedinger, K. R. (2003). Recasting the feedback debate: Benefits of tutoring error detection and correction skills. In U. Hoppe, F. Verdejo & J. Kay (Eds.), Artificial intelligence in education: Shaping the future of learning through intelligent technologies (pp. 13-20). Amsterdam: IOS Press.

McAlpine, L., & Weston, C. (2000). Reflection: Issues related to improving professors' teaching and students' learning. Instructional Science, 28(5), 363-385.McNely, B. J., Gestwicki, P., Hill, J. H., Parli-Horne, P., & Johnson, E. (2012). Learning analytics for collaborative writing: a prototype and case study. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 222-225). ACM.

McKenney, S., & Mor, Y. (2015). Supporting teachers in dataâ€informed educational design. British journal of educational technology, 46(2), 265-279.Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5-11.

McNely, B. J., Gestwicki, P., Hill, J. H., Parli-Horne, P., & Johnson, E. 2012. Learning analytics for collaborative writing: a prototype and case study. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (Vancouver, Canada, 2012). ACM, New York, NY, 222-225.

Melero, J., Hernández-Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of locationâ€based learning design. British Journal of Educational Technology, 46(2), 317-329.

Miller, M. (2015). Leveraging CSCL technology for supporting and researching shared task perceptions in socially shared regulation of learning. (Unpublished doctoral dissertation) Univesity of Vicotria, Canada.

Norman, D. A. (2013). Design of Everyday Things: Revised and Expanded. New York: Basic Books. London: MIT Press (UK edition)

Nussbaumer, A., Hillemann, E. C., Gütl, C., & Albert, D. (2015). A Competence-based Service for Supporting Self-Regulated Learning in Virtual Environments. Journal of Learning Analytics, 2(1), 101-133.

Oblinger, D.G. (2012). Let’s talk analytics. EDUCAUSE Review, July/August, 10-13

Postman, N. (1985). Amusing ourselves to death. New York: Penguin.

Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of educational Psychology, 95(4), 667.

Rodríguezâ€Triana, M. J., Martínezâ€Monés, A., Asensioâ€Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330-343.

Roll, I., & Winne, P. H. (2015). Understanding, evaluating, and supporting selfâ€regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7—12.

Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(1), 12-27.

Segedy, J. R., Kinnebrew, J. S., Biswas, G. (2015). Using Coherence Analysis to Characterize Selfâ€Regulated Learning Behaviours in Openâ€Ended Learning Environments. Journal of Learning Analytics, 2(1), 13—48

Santos, J. L., Govaerts, S., Verbert, K., & Duval, E. (2012, April). Goal-oriented visualizations of activity tracking: a case study with engineering students. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 143-152). ACM.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

Schraw, G. (1998). Promoting general metacognitive awareness. Instructional science, 26(1-2), 113-125.

Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist, 57, 1380—1400. doi:10.1177/0002764213498851

Slade, S., & Prinsloo, P. (2013). Learning Analytics: Ethical Issues and Dilemmas. American Behavioral Scientist, 57(March), 1—20. doi:10.1177/0002764213479366

Stolterman, E., & Wiberg, M. (2010). Concept-driven interaction design research. Human—

Computer Interaction, 25(2), 95-118.

Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data.

American Journal of Evaluation, 27(2), 237—246.

van Harmelen M. & Workman, D. (2012). Analytics for learning and teaching. CETIS Analytics

Series 1(3).

van Leeuwen, A. (2015). Learning analytics to support teachers during synchronous CSCL:

balancing between overview and overload. Journal of Learning Analytics, submitted for publication.

Vatrapu, R., Teplovs, C., Fujita, N., & Bull, S. (2011, February). Towards visual analytics for teachers' dynamic diagnostic pedagogical decision-making. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (pp. 93-98). ACM.

Veenman, M. (2013). Metacognition and learning: conceptual and methodological considerations revisited. What have we learned during the last decade? Keynote given at EARLI 2013.

Verbert, K., Duval, E., Klerkx, J., Govaerts, S. & Santos, J.L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500-1509.

Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Van Assche, F., Parra, G., & Klerkx, J. (2014). Learning dashboards: An overview and future research opportunities. Personal and Ubiquitous Computing, 18, 1499—1514. doi:10.1007/s00779-013-0751-2

Visser, L., Plomp, T., Amirault, R. J., & Kuiper, W. (2002). Motivating students at a distance: The case of an international audience. Educational Technology Research and Development, 50(2), 94-110.

Winne, P. H. (1995). Self-regulation is ubiquitous but its forms vary with knowledge. Educational Psychologist, 30(4), 223-228.

Winne, P. H. (2011). A cognitive and metacognitive analysis of self-regulated learning. Handbook of self-regulation of learning and performance, 15-32.

Winne,P. H. (2000). Information processing models of self-regulated learning. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theory, research, and practice. New York: Longman.

Winne, P. H., & Baker, R. S. J. d. (2013). The Potentials of Educational Data Mining for Researching Metacognition, Motivation and Self-Regulated Learning. JEDM - Journal of Educational Data Mining, 5(1), 1—8. Retrieved from http://www.educationaldatamining.org/JEDM/index.php/JEDM/article/view/28

Winne,P. H.,& Hadwin,A. F. (1998). Studying as self-regulated learning. In D. Hacker,J. Dunlosky,& A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277— 304). Mahwah,NJ: Erlbaum

Winne, P.H., & Perry, N.E. (2000). Measuring self-regulated learning. In M. Boekaerts, P.R. Pintrich & Zeidner (eds), Handbook of Self-regulation. San Diego, CA: Academic Press.

Winne,P. H.,& Hadwin,A. F. (1998). Studying as self-regulated learning. In D. Hacker,J. Dunlosky,& A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277— 304). Mahwah,NJ: Erlbaum.

Authors (2013)

Authors (2014)

Authors (2014)

Zeini, S., Göhnert T., Krempel L., & Hoppe H. U. (2012). The impact of measurement

time on subgroup detection in online communities. In Proceedings of the

IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, (pp. 389-394), Instanbul, Turkey: IEEE.

Zimmerman, B. J., & Schunk, D. H. (Eds.) (2001). Self-regulated learning and academic

achievement: Theoretical perspectives (2nd ed.). Mahwah, NJ: Lawrence

Erlbaum.

Zimmerman, B. J., & Schunk, D. H. (Eds.). (2013). Self-regulated learning and

academic achievement: Theoretical perspectives. Routledge.

Zimmerman, B. J., Bandura, A., & Martinez-Pons, M. (1992). Self-motivation for

academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29(3), 663-676.




DOI: http://dx.doi.org/10.24059/olj.v20i2.783