Online Credit Recovery School-Level Enrollment: Intended and Unintended Consequences

Prior to the COVID-19 pandemic, online credit recovery (OCR) was the most popular use of distance learning in high schools in the United States. With high course failure rates during the height of the COVID-19 pandemic, high schools have turned to OCR to help students recover lost credit. This study examined the potential consequences of increasing OCR enrollment at the school level using administrative data from North Carolina and found that increasing OCR enrollment is associated with higher rates of passing previously failed courses but with diminishing returns after about three-quarters of students who failed courses enrolling in OCR. Consistent OCR enrollment increases over four years is associated with higher graduation rates. Contrary to prior research, this study finds no evidence that school-level OCR enrollment increases are associated with lower test score proficiency rates. Using pre-pandemic data to help inform post-pandemic decision making, the results suggest that increasing OCR enrollment might address increased pandemic-induced course failure rates by expanding opportunities to re-earn course credit, but this would not necessarily translate to higher graduation rates.

While the rapid shift to online learning during the COVID-19 pandemic was extremely disruptive, it was not the first time most high schools in the United States used distance learning.Over the last 20 years, high schools have increasingly turned to online credit recovery (OCR) for students who fail traditional courses and need credit remediation (Watson & Gemin, 2008).OCR potentially provides cost savings and efficiency by allowing students to rely on software instead of a traditional instructor to remediate credit (Heinrich & Darling-Aduana, 2021).As course failure rates have increased in the wake of pandemic schooling (Borter & O'Brien, 2021;St. George, 2020;Thompson, 2020), educational leaders are likely considering whether to scale OCR to meet growing demand for course remediation.This study provides evidence on the effects of increasing OCR enrollment at the school level.
Decisions about appropriate OCR enrollment levels will be especially complex postpandemic because many students failed courses in which they had no access to the traditional face-to-face format (National Trends in School Openings Since January 2021).However, many OCR tradeoffs are well known to school leaders.Mainly, pervasive criticism from the media, academics, and the NCAA has focused on whether OCR helps students to graduate high school without acquiring the appropriate academic knowledge (Kohli, 2017;Loewenberg, 2020;Sproull, 2018).Using pre-pandemic data to help inform post-pandemic decision making, I investigate both potential benefits and negative side effects of increasing OCR enrollment to assess whether it could narrow opportunity gaps by providing access to credit recovery or whether it exacerbates opportunity gaps already widened by the pandemic if the online courses are of low quality (Goldberg, 2021).

Literature Review Why Schools Use Online Credit Recovery
Beginning in 2010-11, the U.S. Department of Education (ED) required high schools to report their graduation rates and required states to hold high schools accountable for their graduation rates (No Child Left Behind High School Graduation Rate Non-Regulatory Guidance, 2008).At the same time, states competed for significant federal funds through Race to the Top, which offered large financial incentives to states that developed accountability systems that heavily sanctioned low-performing schools as determined by test scores and high school graduation rates (Overview Information; Race to the Top Fund; Notice Inviting Applications for New Awards for Fiscal Year 2010;Notice, 2010).As federal-level school accountability mandates placed greater emphasis on graduation rates, school districts responded by offering credit recovery options along with other reforms, including expanded access to OCR.
Due to evidence indicating that failure to accumulate credits in a timely manner is a major barrier to high school graduation (see Allensworth & Easton, 2005;Bowers, 2010;Mac Iver & Messel, 2013), educational policymakers may view OCR as a means to remove this barrier in response to accountability pressure to increase graduation rates.The hypothesis is that students are less likely to drop out of high school (and, therefore, more likely to graduate) if they can more easily obtain credits for required courses they failed, i.e., without the constraints of the typical face-to-face (F2F) course (Murin et al., 2015;Watson et al., 2008).While F2F was the traditional method for credit recovery, OCR represented a shift to distance education away from the traditional F2F strategy of repeating the failed course after school, during the summer, or during the following school year.In 2013, Connecticut became the first state to mandate that high schools offer OCR to all students who fail a course if the school has a dropout rate of eight percent or higher, representing a shift in the locus of decision-making about credit recovery from local districts to the state (Murin et al., 2015).
OCR is a very popular tool for school districts.In the 2009-10 school year, before the ED required that high schools be held accountable for their graduation rates, nationwide enrollment in OCR was estimated at over 1.1 million (Queen & Lewis, 2011).In surveys from Iowa, Wisconsin, and New York, high school leaders reported that their most common use of distance learning was for OCR (Clements, Stafford, et al., 2015;Clements, Zweig, et al., 2015).During the 2015-16 school year, 72 percent of high schools reported offering credit recovery, and one in 10 high schools enrolled 20 percent or more of the student body in credit recovery (Tyner & Munyan-Penney, 2018).Also in the 2015-16 school year, schools in North Carolina were as likely to enroll students in OCR as F2F credit recovery for the first time (Viano, 2021).The most recent OCR enrollment estimates come from the federal Civil Rights Data Collection from 2017-18.Based on author calculations, about 60% of high schools at that time offered credit recovery to an average of 78 students per school.While press outlets have reported that districts greatly expanded OCR to respond to high course failure rates during the first two years of the COVID-19 pandemic (Gross, 2021;Belsha, 2022), no estimates of actual recent enrollments are available at this time.

The Effects of Online Credit Recovery
Despite large enrollments nationally, a policy mandate in Connecticut, and evidence of high utilization in at least one state, only recently has evidence begun to emerge on the effects of OCR on students (Carr, 2014;Heppen et al., 2016;Stallings et al., 2016).Recently, several studies have investigated, with mixed results, how students individually fare when enrolled in OCR compared to F2F credit recovery courses.This includes two experiments in Chicago and Los Angeles where students who failed Algebra I (both cities) or English 9 (Los Angeles only) were randomly assigned to take OCR or F2F credit recovery over the summer.Immediately after taking the OCR/F2F course, students randomly assigned to OCR sometimes had a lower likelihood of passing the credit recovery course (Chicago experiment and English 9 in the Los Angeles experiment) with lower exam scores in the Chicago experiment than those randomly assigned to F2F (Heppen et al., 2016;Rickles et al., 2023).The Chicago experiment took place in 2011 and 2012, allowing for analysis of longitudinal outcomes but, in this experiment, OCR did not lead to differential outcomes later in high school like high school graduation and grade point average (Rickles et al., 2018).
Quasi-experimental research has found that OCR students were more likely to pass the course, had lower test scores, and were more likely to graduate from high school than F2F credit recovery students (Hart et al., 2019;Heinrich et al., 2019;Stallings et al., 2016;Viano & Henry, 2023).At the same time, these students were less likely to enroll in a four-year university and had lower earnings as young adults (Heinrich & Cheng, 2022;Heinrich & Darling-Aduana, 2021).These studies use a variety of adjustments to regression models to try to account for complex selection bias in student selection into OCR.The main challenge when establishing whether OCR causes differential student outcomes is the likelihood of strategic OCR enrollment approaches by school administrators (Viano, 2021).While it appears that OCR students are more likely to earn course credit but have lower test scores than similar F2F credit recovery students, this might reflect the fact that administrators assign highly motivated but lower achieving students to OCR courses who would have had the same test scores and graduation rates if they had been assigned to a F2F course.
Compared to the quasi-experimental studies, experiments on OCR have not as definitively found OCR to effectively increase high school graduation probability while decreasing test scores, so it remains unclear what results schools are getting from using online learning for credit recovery.Since OCR remains popular (Tyner & Munyan-Penney, 2018), it is possible that school leaders continue to use OCR because they observe positive trends schoolwide that they attribute to OCR.However, prior research has not examined OCR enrollment effects from a school-level perspective.

Conceptual Framework
At its most basic level, OCR is a popular use of distance learning in high schools to allow students the option of retaking failed courses online instead of F2F.As described above, schools are incentivized to use OCR to increase credit accumulation to raise graduation rates.Within the context of this study's framework, two outcomes are the intended consequences of successful OCR enrollment: students will pass more classes that they previously failed, and they will be more likely to graduate high school as OCR enrollment rates increase.However, critics are skeptical that these positive gains come for free (see Carr, 2014).Specifically, several studies indicate that the population of students who fail courses in high school would be particularly illsuited to succeed in an online learning environment (Viano, 2018a).Students who fail courses in high school, often labeled as at-risk students, are more likely to have lower technological and online skills than students who do not fail courses in high school (Judge, 2005;Kuhlemeier & Hemker, 2007;Oliver et al., 2009;Valadez & Duran, 2007).Also, students who fail one class are more likely to have failed other courses as well, perhaps indicating multiple skill deficits that could make it challenging to succeed on a complicated online platform (Bowers & Sprott, 2012;Judge, 2005;Roderick, 1994).As summed up by Huett and colleagues in a review of knowledge about K-12 online learning, "We fear that distance education may become little more than a 'dumping ground' for credit recovery...the exact opposite population the research says tends to thrive in the distance environment" (Huett et al., 2008, p. 64).
Prior research has confirmed some of these concerns about OCR student experiences.Studies of OCR students enrolled in the North Carolina Virtual Public Schools (NCVPS) report challenges with reading comprehension, navigating the online platform, and motivation (Lewis et al., 2014(Lewis et al., , 2015)).Students simultaneously appreciated that courses were self-paced while disliking the challenges related to time management (Lewis et al., 2014).Another study of high school students in an online course offered by a university found that OCR students were more likely to seek help with their course from a parent while non-OCR students in the online course were more likely to go to teachers or peers (Oviatt et al., 2018).This strategy could undermine student performance if OCR students seek assistance with content the parent is not familiar with (Oviatt et al., 2018).While schools across the country turn to OCR as a way for students to earn course credits, there are significant reasons to doubt that an online learning approach would be successful with the population of students who fail courses in high school.
Further, the negative effects of interventions designed to quickly meet accountability targets are well documented (see Balfanz et al., 2007;Dee et al., 2013;Jennings & Bearak, 2014).As an intervention implemented to respond to federal accountability pressure to increase high school graduation rates, OCR has the potential to introduce unintended negative side-effects like lower test scores.OCR could lead to lower end-of-course exam scores if OCR courses are low-quality and/or students learn less in OCR courses than if they had repeated the F2F (Fong et al., 2014;Heinrich et al., 2019;Heppen et al., 2016).While research has examined these connections at the student level, scaling to the school level would help to understand broader consequences of schools' decisions about OCR enrollment levels.

Purpose and Research Questions
This study represents the first known attempt to quantify the impact of offering OCR at the school level on outcomes like graduation rates and exam proficiency rates.Prior OCR research has not investigated whether student-level associations scale.Educational leaders are likely making broader decisions about how many OCR licenses to purchase, computer labs to devote to OCR, and staff to supervise OCR.In other words, educational leaders also benefit from information on the appropriate level of OCR enrollment.While it might be tempting to assume that prior evidence scales to the school level, this would not necessarily be expected based on prior evidence on treatment effect heterogeneity (Olsen, 2017).Specifically, OCR might be associated with positive outcomes, on average, with diminishing returns when scaled to the point where schools are unable to provide the same level of support (e.g., teacher assistance, highquality software), or, conversely, scaled too small for supports to be provided.Correspondingly, I address the following research question using administrative data from the state of North Carolina: to what extent is increasing OCR enrollment at the school level associated with the intended consequences of increased passing rates of previously failed courses and high school graduation rates and the unintended consequence of lowered proficiency rates on end of course exams?

Data and Sample
The data for this project come from an administrative database maintained by the Education Policy Initiative at the University of North Carolina at Chapel Hill (EPIC) including all public schools in North Carolina (NC).I include datasets on course rosters/grades and school demographics/performance for the 2012-13 through 2016-17 school years.I only include OCR enrollment in core courses required to graduate from high school (i.e., English, mathematics, science, and social studies; see High School Graduation Requirements, n.d.).Schools are the unit of analysis.
There are about 600 schools with high school-level grades in NC.Of these, about 400 are traditional high schools with grades 9-12 while the other 200 schools contain other grades in addition to 9-12 (Facts andFigures 2015-16, 2016).The schools' racial makeup ranges from 100 percent white to 99 percent non-white, with the median school having 54 percent white students, 28 percent Black students, and 12 percent Hispanic students.The percentage of economically disadvantaged students ranges from zero to 100 percent, with the median school having half of the student body classified as economically disadvantaged.Overall, NC contains many high schools with a diverse array of racial and socioeconomic demographics.
Schools in NC have two options for OCR: publicly run NCVPS or privately run online course providers.This is a very common configuration of OCR options nationally; at least 40 states have a state-run virtual school and privately provided courses are ubiquitous (Watson et al., 2008).NCVPS courses are available across the state, and schools pay per course enrollment, between $310 and $510 (North Carolina Virtual Public School, n.d.).Private providers have contracts with schools/districts to provide OCR, usually charging per course or based on the number of students logged in at one time.Schools in NC during 2012-13 through 2016-17 school years tended to offer OCR during the school day as part of the students' schedule with an inclass monitor, although this was not a state requirement (Viano, 2018b).While it would be useful to include the OCR provider type (i.e., NCVPS or private) or the actual provider (e.g., an indicator for the company), this information is not available in secondary data, and private providers have been unwilling to share information on their clients (i.e., school districts) with researchers (Stallings et al., 2016).

Measures
I include mean values on the independent variables, outcomes, and covariates in Table 1.These values help to communicate the typical demographics of high schools in NC during this time.In addition, the mean values are meant to aid in interpretation of the findings on the average OCR enrollment and the outcomes.In other words, the models predict the change in outcomes in response to changes in the independent variable which can be interpreted in reference to the mean values in Table 1.

Online Credit Recovery Enrollment
I define OCR enrollment as the percent of students who failed a core, required F2F course and subsequently enrolled in OCR.This is a measure of how often schools assign OCR to the target population, i.e., students who lost course credit.On average, schools assign 18.4% of students who previously failed courses to OCR (see Table 1).This percentage increased throughout this time with the median school assigning 9.9% of students who failed courses to OCR in 2012-13 to 18.9% in 2016-17.

Dependent Variables
The study includes three dependent variables.
(1) The passing rate of previously failed courses represents the number of courses students passed in OCR or F2F each school year divided by the number of courses students failed in the current or previous school year.While I term this the passing rate, this more closely resembles a ratio that can be above 1 if schools assign OCR/F2F for courses failed more than a year prior (data limitations only allow the calculation of the number of failures with a one-year lag).( 2) Graduation rates are the state's official four-year cohort graduation rate, indicating the percent of first-time ninth graders who graduate within four years.(3) The end-of-course exam (EOC) proficiency rate is the percent of students in the school who scored proficient on the Math I, Biology, and English II EOCs (the only subjects high school students are tested on in NC).

Empirical Framework
I assess the effect of changes in OCR enrollment and the outcomes using the following model: (1)   =  0 +  1   +     +   +   + where   represents one of the three outcomes, standardized by year, for each school s in year t.
The variable   represents the student-level OCR enrollment measure (divided by 10 to ease interpretation).I also fit models with lagged versions of   to examine whether the associations between the OCR enrollment and the outcomes are cumulative over time.Based on an F test, I fit the passing rate outcome with quadratic and trinomial terms of   , but I determined that linear models better fit (and did not meaningfully change the findings of) the graduation and EOC proficiency rate outcomes.The model includes a vector of time-varying covariates,   , including school enrollment, demographics (percent Black, Hispanic), percent of limited English proficient students, percent of students with disabilities, percent gifted, and percent economically disadvantaged.I include the course failure rate for initial course enrollments as a covariate to account for the association between the preponderance of course failures and the outcomes, independent of OCR enrollment.See Table 1 for a full list of covariates included in this vector.This model exploits changes over time within schools in their levels of OCR enrollment due to the school fixed effects,   , which subtract the group mean of each variable in the model.The school fixed effect restricts the comparisons to within-school, such that each coefficient represents the effect of a one unit increase within school on the outcome.This eliminates between school variation that might affect the outcome like neighborhood crime levels or availability of afterschool activities.To distinguish these changes from annual trends, like changes in graduation rates over time, I include a year fixed effect,   .Thereby,  1 represents the associated difference in the outcome for each 10-percentage point increase, within school, in OCR enrollment.

Limitations
The ideal method for assessing the impact of school-level OCR enrollment would be to randomly assign different OCR enrollment levels to schools.Given that this is impossible to do using secondary data and the likely reticence of school leaders to forego autonomy on course assignment across the whole school, I attempt to isolate the effect of changes in OCR enrollment as much as possible.This modeling strategy has inherent limitations and does not truly approximate the causal interpretation afforded by random assignment in several ways.First, the school fixed effect is helpful in that it removes between-school variation, but it also means that I can only assess changes in outcomes associated with changes in OCR enrollment over time.If a school was already using OCR strategically to raise graduation rates but did not change OCR enrollment levels during this time, then this strategic use would be undetected in this model.Second, it is possible the estimates include omitted variable bias if schools tend to implement other interventions alongside changes in OCR enrollment that affect the outcomes.In this case, it would appear that OCR enrollment changes are affecting the outcomes while actually, this other, unobserved intervention is causing the outcomes.I minimize this risk through the covariates and the year fixed effect, but it remains present in any quasi-experimental design.

Passing Rate of Previously Failed Courses
The results from the school and year fixed effects model with the outcome of the passing rate of previously failed courses (standardized) are in Table 2.This model, with standardized passing rate as the outcome, includes the quadratic and trinomial terms for OCR enrollment since an F test determined the linear, quadratic, and trinomial terms are jointly significant.These results show that for each 10-percentage point increase in OCR enrollment, the passing rate of failed courses is predicted to increase with a 0.128 coefficient on the linear term, although the negative coefficient (-0.009) on the quadratic percent OCR variable indicates that there are diminishing returns to this positive effect.To ease interpretation given the higher order variables, I predict the passing rate of previously failed courses across the typical range of OCR enrollment in Figure 1 with the unstandardized version of the outcome.The black line shows the predicted value surrounded in grey by the 95% confidence interval.The horizontal line represents the average passing rate of previously failed courses across schools in the sample.Figure 1 shows the positive association between OCR enrollment and the passing rate with higher OCR enrollments associated with higher passing rates of previously failed courses.The slope decreases across values of OCR enrollment due to the negative quadratic term.According to the first derivative of this model, the first turning point in this trinomial is around 72.5%.This turning point is shown visually on Figure 1 where the curve starts to trend downwards (the other turning point would occur above 100%).This indicates that the passing rate is predicted to increase with higher OCR enrollment until about three-quarters of students who fail courses are assigned to OCR, with diminishing returns thereafter.

Test Score Proficiency
To examine potentially unintended consequences on EOC proficiency rates, the results with this outcome (standardized) are in Table 3.These results are mostly null such that I fail to find a significant relationship between OCR enrollment and EOC proficiency rates.OCR enrollment could have a delayed effect on EOC proficiency rates.Students might be taking non-EOC OCR courses, so the negative effects on EOC proficiency rates would occur downstream, as students enter EOC courses less prepared if they had learned little in OCR.To examine this potential lagged effect of OCR enrollment, I estimate models in Table 3, columns 2-4, assessing whether increases in OCR enrollment rates over one (column 2), two (column 3), or three (column 4) years are associated with changes in EOC proficiency.The row labeled Combined PercOCR Coefficients represents the cumulative effect of increasing OCR enrollment 10percentage points (i.e., one unit change on the coefficient) every year for one, two, or three years, respectively.This combined coefficient in column 2 is -0.0003, indicating that a 10-percentage point increase in OCR enrollment in the current year and the previous year is associated with a very small, not statistically significant decrease of -0.0003 standardized units on EOC proficiency rates.I find no evidence in this table that OCR enrollment changes are associated with decreased EOC proficiency rates.PercOCRst is divided by 10 such that each unit increase corresponds to a 10 percentage point increase in the percent of students who failed courses enrolled in OCR.* p < 0.05, ** p < 0.01, *** p < 0.001

High School Graduation Rates
To test whether schools strategically increase their OCR enrollment rates to increase their graduation rates, I present the results from the school and year fixed effect models in Table 4.As shown in column 1, I do not find evidence that increases in OCR enrollment are associated with significant increases in high school graduation in the year the increase occurred.However, I do find evidence that consistent increases over time in OCR enrollment lead to increases downstream in high school graduation.As shown in column 4 of Table 4, a 10-percentage point increase in OCR enrollment over four years is associated with a 0.187 standard deviation increase in the graduation rate.In other words, if schools continually assign more students who failed courses to OCR every year, then after four years they would be predicted to have higher graduation rates.

Discussion and Conclusion
As educational leaders consider how to structure credit recovery post-pandemic, the findings support the potential of increasing OCR enrollment to address large increases in course failure rates during the pandemic with the caveat that the last year of data was the 2016-17 school year, three years before the beginning of the pandemic.The results mirror prior studies that found, at the student-level, OCR enrollment was associated with increased likelihood of recovering course credits compared to F2F, such that the individual-level effects appear to scale to the school level (Hart et al., 2019;Heinrich et al., 2019;Viano, 2021).I did not find evidence to support OCR being the only way students should recover course credits, with diminishing returns to increased OCR enrollments after about three-quarters of students who failed courses enrolled in OCR.
I found little evidence that the increased credit accumulation from higher OCR enrollments translated into higher graduation rates, at least in the short term.This result more closely matches findings from experimental studies which found that OCR did not lead to higher rates of high school graduation, as opposed to prior quasi-experimental work which found significant relationships (Hart et al., 2019;Heinrich & Darling-Aduana, 2021;Rickles et al., 2018;Stallings et al., 2016).This could also reflect that the prior associations between high school graduation probability and OCR were not high enough to lead to significant increases in school-level graduation rates.Similarly, negative evidence on lower test scores of OCR courses compared to F2F might not have impacted proficiency rates if the higher test scores of F2F students were still below proficiency.In other words, if F2F credit recovery is associated with higher test scores than OCR but with averages still below test score proficiency cutoffs, then these higher scores would not translate into changes in proficiency rates.This hypothesis is supported by findings reported elsewhere that OCR and F2F credit recovery students have average standardized assessments scores well below the mean (Viano & Henry, 2023).The null findings on the relationship between OCR and test score proficiency could be reassuring for educational leaders concerned that increasing their use of distance learning through OCR enrollment would harm school-level performance as would have been indicated by prior research on OCR assignment at the student-level (Heinrich et al., 2019;Heppen et al., 2016;Viano & Henry, 2023).
Overall, these results encourage educational leaders to carefully consider resources appropriated to using OCR to recover credits students lost during the pandemic.While OCR might be effective at initially solving an obvious problem caused by the COVID-19 pandemic, higher course failure rates, I did not find evidence that OCR will provide a comprehensive strategy to lessen the impact of high course failure rates on graduation rates.

Declarations
The author declares that there are no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval for this study was granted by the Office of Human Research Ethics Internal Review Board at the University of North Carolina at Chapel Hill.This project was supported by the National Academy of Education and the National Academy of Education/Spencer Dissertation Fellowship Program.

Table 1
Descriptive Statistics Note.Standard errors in parentheses.OCR-online credit recovery, EOC-end of course exam, LEP-limited English proficient, SPED-receives special education services.

Table 2
School and Year Fixed Effects Models with Full Covariates Estimating the Association Between Within-school Changes in the Percent of Students who Failed Courses Enrolled in OCR and the Standardized Passing RateNote.Standard errors in parentheses, are clustered by school.All outcomes are standardized by year.PercOCRst is divided by 10 such that each unit increase corresponds to a 10-percentage point increase in the percent of students who failed courses enrolled in OCR. *

Table 3
School and Year Fixed Effects Models with Full Covariates Estimating the Association Between Within-school Changes in the Percent of Students who Failed Courses Enrolled in OCR and the Standardized EOC Proficiency Rate Standard errors in parentheses, are clustered by school.All outcomes are standardized by year.

Table 4
School and Year Fixed Effects Models with Full Covariates Estimating the Association Between Within-school Changes in the Percent of Students Who Failed Courses Enrolled in OCR and the Standardized Cohort Graduation Rates Note.Standard errors in parentheses, are clustered by school.All outcomes are standardized by year.PercOCRst is divided by 10 such that each unit increase corresponds to a 10-percentage point increase in the percent of students who failed courses enrolled in OCR.* p < 0.05, ** p < 0.01, *** p < 0.001