An Evaluation of Critical Thinking in Competency-Based and Traditional Online Learning Environments


  • Matthew Mayeshiba University of Wisconsin-Extension
  • Kay Jansen University of Wisconsin-Milwaukee
  • Lisa Mihlbauer University of Wisconsin-Milwaukee



AAC&U, Competency-based education, direct assessment, disruptive innovation, critical thinking, learning outcomes, non-term, nonterm, UW Flexible Option, VALUE rubric


Non-term, direct assessment competency-based education (CBE) represents a significant re-imagining of the structure of higher education. By regulating students’ progress through the program based on their mastery of tightly defined competencies rather than based on the time spent learning them, this learning environment affords students far greater flexibility than traditional programs. This focus on defined competencies has led to concerns that students in these types of programs may not demonstrate higher-level skills, such as critical thinking, at levels comparable to those enrolled in more traditional programs. This study evaluated 39 students’ demonstration of critical thinking in two assessments administered in parallel versions of one course: one offered through the non-term, direct assessment CBE University of Wisconsin Flexible Option, and the other offered through a traditional online program. For this study, each of the 78 assessments was scored using the critical thinking rubric from the Valid Assessment of Learning in Undergraduate Education (VALUE) project. We found that students from the CBE version of the course received significantly higher (p=.0013) overall scores than the students in the traditional online version of the course. While further research is required to refine these methods and ensure the generalizability of these results, they do not support concerns about students’ abilities in this learning environment.

Author Biographies

Matthew Mayeshiba, University of Wisconsin-Extension

Matthew Mayeshiba serves as a researcher and data analyst with the University of Wisconsin-Extension. Specializing in competency-based education, since 2014 he has worked extensively with Lumina Foundation and the National Center for Research on Distance Education and Technological Advancements (DETA Center) to document and measure the implementation of the non-term, direct assessment competency-based University of Wisconsin Flexible Option.

Kay Jansen, University of Wisconsin-Milwaukee

Dr. Kay Jansen is a Clinical Professor and Director of the Undergraduate Nursing Program at the University of Wisconsin-Milwaukee College of Nursing. She is a credentialed specialist in adult psychiatric-mental health nursing and a certified nurse educator. She has taught at the graduate and undergraduate levels of nursing education for 35 years. Her research and scholarship have focused on mental health nursing, hospital readmission, and health care informatics.

Lisa Mihlbauer, University of Wisconsin-Milwaukee

Lisa Mihlbauer is a Clinical Associate Professor and the RN-to-BSN Program Director at the University of Wisconsin-Milwaukee College of Nursing. She has over 30 years of classroom and clinical teaching experience in nursing and is a board certified nurse educator.


AAC&U. (2015). Falling Short? College Learning and Career Success. Washington, D.C.: Hart Research Associates. Retrieved from

AAC&U. (2016). Valid Assessment of Learning in Undergraduate Education Project. Retrieved December 21, 2016, from Association of American Colleges and Universities:

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 37-46.

Cohen, J. (1968). Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit. Psychological Bulletin, 70, 213-220.

Eduventures. (2014). Closing the Degree Completion Gap: Challenges and Opportunities. Boston, MA: Eduventures. Retrieved from

Fain, P. (2015). Measuring Competency. Retrieved November 14, 2016, from Inside Higher Ed:

LeBlanc, P., & Selbe, J. (2016). Another Take on Competency. Retrieved January 12, 2017, from Inside Higher Education:

Lowry, R. (2016). Kappa as a Measure of Concordance in Categorical Sorting. Retrieved December 15, 2016, from VassarStats:

Nodine, T. (2016). How did we get here? A brief history of competency-based higher education in the United States. Journal of Competency-based Education, 1(1), 5-11. doi:10.1002/cbe2.1004

SHEEO. (2016). MSC Demonstration Year: September 2015 through August 2016. Retrieved from State Higher Education Executive Officers Association:

Viera, A. J., & Garrett, J. M. (2005). Understanding Interobserver Agreement: The Kappa Statistic. Family Medicine, 37(5), 360-363.

Ward, S. C. (2016). Let Them Eat Cake (Competently). Retrieved November 14, 2016, from Inside Higher Ed:






Special Issue: National Research Center for Distance Education and Technological Advancements (DETA)