Gap Analysis : An Innovative Look at Gateway Courses and Student Retention

In this paper we argue that simply identifying gateway courses in which a large number of students fail or withdraw and focusing attention on them may not always be the best use of limited resources. No matter what we do, there will always be courses with high D/F/W rates simply because of the nature of their content and the preparation of the students who must take them. However, some gateway courses defy expectations and produce fewer DFWs than might be expected while others produce more. Moreover, the timing of course taking can make a difference between success or failure for particular types of students, and failing or withdrawing from a course does not always lead to stopping out. In this paper we use examples from our work with the analyses of student records to show how one can use student type and point in their academic life to predict success in particular gateway courses. Relating predictions to observed DFW rates can highlight courses exceeding expectations and those which fall below them, and support a more nuanced understanding of where attention is needed. Further, we look at whether or not actionable information can be found by looking at whether the general connection between DFW’s and attrition holds up for different groups of students in specific courses.


Introduction
Improvement in the numbers of young Americans achieving a postsecondary degree has been a national priority for over a quarter of a century (Arnold, 1999;Shapiro, Dundar, Yuan, Harrell & Wakhungu, 2014), with little improvement seen.Indeed, a myriad of studies in the last decades of the 20th century tested the assumptions of theories concerning the reasons why students drop out of higher education institutions (Bean & Metzner, 1985;Mallette & Canrera, 1991;Munro, 1981;Tinto, 1987) to develop models of student progression.
An emerging strategy for enhancing postsecondary outcomes is to measure the patterns by which students reach and move through intermediate stages of degree completion.One of the issues identified as contributing to attrition is poor performance in gateway courses (Koch & Pistilli, 2015).The Gardner Institute (2015) has identified "gateway courses" as: foundational, credit-bearing, lower division courses, for which large numbers of students are at risk of failure and which accordingly stand as "gatekeepers" to further study and degree completion.Indeed, researchers have found that retention in these courses is strongly correlated with successful degree completion (Cabrera, Burkum & La Nasa, 2005;Herzog, 2005;Lewis & Terry, 2016;Moore & Shulock, 2009;Offenstein & Shulock, 2010).Koch and Pisitilli (2015) add that "courses with high rates of unsuccessful outcomes (DFWI rates) 'kill' a student's grade point average (GPA), motivation, and academic progress" (p.3).
The problem of gateway courses is especially pernicious in online environments, and online educators are attempting to address the issue, primarily through course redesign (Education Advisory Board, 2016).At DePaul University, (n.d.), for example, course redesign focuses on approaches that will help students learn more effectively.Other strategies for improving success in gateway courses include providing extra support for faculty teaching such courses (Nogai & Kans) and/or peer support for students taking them (Arendale, 2004).These approaches, however, often assume that all gateway courses have the same impact on all students.Our investigations indicate that this is not always the case.In this paper we ask whether all gateway courses are equally detrimental to student success and/or whether the detrimental effects of poor performance in gateway courses apply equally to all students.Answers to such questions are important so that institutional resources available for improving courses, student support and advising and placement practices can focus on those areas in which the problems are most critical.

Methods
The challenge is to find the right measures to identify gateway courses, and, once identified, to apply the best "fix."It is simple enough to sort courses in descending order of DFW rates, targeting the highest entries for redesign of the course, for changes in its staffing, or for providing student support as these are all factors that are under institutional control.This approach will often work well enough, but this single, simple measure may overlook some problem cases, or, worse yet, identify problems where none exist, causing misplaced or even damaging "fixes." A more careful approach to finding gateway courses would recognize that some courses are simply harder than others.There is no single standard for DFW rates that is universally applicable.There are also significant differences in students that need to be considered when looking at the effectiveness of courses.Students enter with widely different backgrounds and learning goals.Individual students change significantly over the course of their studies.The effectiveness of a course needs to be judged in the context of the students it serves.Once issues have been identified, the response needs to be tailored to those issues.Simply put, it is not reasonable to expect all courses to serve all students equally well.Efforts to do so are doomed from the start and may actually do harm.More careful matching of students and courses is necessary before other "fixes" are applied.
Finally, if retention and degree completion are the goal, the effectiveness of the course needs to include not only the DFW rates it produces, but the subsequent persistence of the students in the course whether they passed it or not.Enrollment and performance in courses needs to be tied to persistence measures as well.

Subjects and Setting
The data from which out observations are drawn included all undergraduate degree-seeking students enrolled in a small, Midwestern public university over a four-year period.The academic calendar is semester based.Course grades were those awarded at the end of each term, not the final transcript grade which is occasionally different for any number of reasons.This is the grade that appears on the students' report card and is most likely to influence the student's immediate decision to continue their degree studies.
DFW rates were used to measure students' success in courses studied.An end of term grade of D or F or prior Withdrawal indicates that the student failed to complete the course successfully.
Enrollment in the next regularly scheduled term or graduation was used as a measure of persistence.This measure was chosen for two reasons.It is known that students who have a break in continuous enrollment at this institution are much less likely to graduate.Although there are exceptions, of course, most students who leave simply don't come back.Secondly, connecting individual courses to student persistence needs to be done with a short-term measure.It is hard to attribute a student's departure to a particular course if they have taken many other courses in the meantime.
For this study, the degree-seeking undergraduate student population was classified into four Student Types.First-time freshman versus transfer students was the first distinction.Within the first time freshmen group, students who did or did not enter in the Honors program were distinguished.The transfer students were further classified according to whether they were majors in online programs or on campus programs.The four Student Types, then, were Native Freshmen, Honors Freshmen, On-ground Transfers, and Online Transfers.
Students were also further classified according to stages in their academic life cycle.The first term is particularly critical for transfer students and was defined as the first stage.For freshmen, the end of the first year is a critical time, so the second term was also defined as a stage.The second year, and third year were the next two stages considered.Anything beyond the third year was considered to be the last stage.Thus the dozen regular semesters that might make up a students' trek towards a six-year degree completion were broken down into First Term, Second Term, Second Year, Third Year and Later.

Analyses
To provide a better context for the interpretation of course DFW rates, a binary logistic regression was used to predict the probability that a student would post a D, F, or W grade in any specific course using all courses that the institution offered over a particular four-year period of time.The predictor variables used were the Student Type and the point in the Student Life Cycle (as defined above), plus the most significant individual predictors for each student: their prior cumulative GPA, and the fraction of previous courses in which they received D, F, or W grades.The latter factor is known to add predictive power over prior GPA alone.There do appear to be students with high GPAs who also have high W rates, perhaps to protect those GPAs.The regression basically predicts, then, the chance that an individual student will get a DFW grade in an average course.
The regressed model makes it possible to predict the overall DFW rate for any course, or section of a course, based on the characteristics of the students who are enrolled in it.These predicted DFW rates can then be used as a benchmark against which actual course performance can be measured.Thus, these predicted DFW rates as well as the observed DFW rates were recorded for each of the courses studied, and the difference (Gap) between them calculated.Courses were then ranked according to observed DFW rates and according to the Gap between observed and predicted rates.
Context is again critically important for connecting individual courses to student persistence.The fraction of students who persist is quite different for different types of students, at different points in their degree paths.When looking at the impact of any particular event (taking a course or getting a particular grade in the course), another kind of Gap was calculated --the difference between the persistence rate for a particular group of students in the course being examined compared to the persistence rate of students in all courses of the same type at the same point in their degree path.
While all grades posted for the students in this study were used to develop the predictive model for course DFW rates and for persistence rates, the courses that were selected for detailed analysis were chosen from a particular subset of the undergraduate general education curriculum.This particular subset of general education courses is related to the institution's specific mission and values, and is required of all undergraduate students.The structure of the requirement gives students great flexibility in term of the specific courses taken and their timing.This adds degrees of freedom in this part of the curriculum that are often not present in highly structured majors, adding placement and advising to the list as realistic options to improve retention related to these specific courses.

Results
Table 1 shows notable cases that result from the analysis of the gap between predicted and actual DFW rates for the 34 general education courses considered here.DFW rates are given in percentage of students getting a D, F, or W in each course.
As expected, the ranking of courses based on actual and predicted DFW rates often told similar stories, as can be seen in the first three cases presented.There were also, however, some significant differences.Some courses with DFW rates high enough to trigger alarms, also had high predicted DFW rates.When the gaps between observed and predicted DFW rates were considered, the courses receded out of the alarm range.The sixth course in Table 1 moves from being in the top five problematic courses by actual DFW rate to the middle of the pack when judged by the gap.Indeed, Table 1 shows that some courses are just living up to predictions, but that in some cases, such as this one, the courses in question are performing better than expected, given the population they serve.This can be seen most clearly in the seventh course in Table 1.While in the top ten of problem courses by actual DFW rates, the gap between actual and predicted DFW rate is large and negative.It is actually performing better than expected for the student population being served.Efforts to "fix" such a course are at best misdirected and potentially even harmful.Such a course may be worthy of further scrutiny, not to improve it, but rather to replicate its relative success with a specific problematic student population.There are also courses with DFW rates that are not high enough to attract attention, but are much higher than would be expected for their given student population.Exploring the gap between observed and predicted DFW rates brings such courses that might otherwise be overlooked into consideration.The fifth row in Table 1 shows a course that might be overlooked using actual DFW rates.Using that gap as the measure brings it into the top 5 of courses that need to be looked at.
Finally, there are courses with extremely low DFW rates, perhaps even rates of zero, despite the fact that the predicted DFW rates for enrolled students are substantial as seen in the last two rows of Table 1.These might be particularly effective courses whose structure and methods need to be replicated.They might also be courses whose performance is more a cause for concern than celebration.
The connection between specific courses and student persistence was also examined and broken down by student type and point in academic life cycle.Again, this classification of student persistence does not produce useful information for all courses, but there are occasionally findings that are genuinely surprising and useful.Table 2 shows the results of a typical analysis of this sort.The data is presented for students who simply took this course compared with similar students who didn't take it.The same approach can be used, however, for students receiving a D, F, or W versus students passing the course or other criteria of interest.The columns in the table are being used to classify students by type.The rows break out students at various stages of their degree paths.Each cell shows the difference between the persistence of the group of students taking this course compared with those who didn't take it.A positive value indicates the percentage by which the students of interest were more likely to persist.A negative value indicates the percentage by which students were less likely to persist.In the case of the particular course shown in Table 2, it was found that taking the course (regardless of the grade earned) was generally associated with average persistence.When the classifications are removed, students who took this course persisted 1.2% more often than similar students who did not take it.However, Table 2 does show a real issue for transfer students taking the course in their first term, whether or not they were online.First semester transfer students were less likely to persist, by 15-20%.If transfer students took the same course in a later term, there were virtually no problems with persistence.There might be many explanations for this effect, but regardless of the reason, keeping new transfer students out of this particular course would be an obvious, and easily accomplished step toward higher retention.
Another course with average DFW rates produced surprising results when connected to a measure of short term student persistence.Students with a DFW grade in this course did not show the expected drop in persistence of about 18%, but rather persistence fell by only 2%.What might be going on in that course to account for this is not going to be found in the database, but further investigation is surely warranted as it appears that something very good was occurring in it that might be replicated in other courses.That there may be courses which serve as "guardrail" courses associated with increased student persistence is an intriguing possibility.
Along those lines, we found that the very act of enrolling in this particular kind of general education course was associated with increased persistence across student types and levels.While it is tempting to think that the curriculum is resonating well with students, the probable explanation is likely more mundane.The courses investigated fulfill an institution specific general education requirement that reflects particular institutional values.Students who intend to transfer to other institutions to complete their studies need not be concerned about them.If anything, enrollment is this group of courses is a confirmation of the students' intentions to complete their degree here.Finding signals that reveal students' true degree plans can also be helpful in targeting retention efforts.

Discussion
Gateway courses with high DFW rates contribute directly to student attrition and are an obvious place to direct retention efforts.The identification of problem courses benefits from the inclusion, however, of some measure of the expected DFW rates for each course.The approach in the research reported here was to base expectations on student type and point in their degree path, as well as significant predictors of individual student performance.Other approaches might be more appropriate for other institutions based on local factors impacting student success.
In any case, deviations from projected DFW rates can be used to identify specific courses particularly in need of help as well as courses in which students are doing better than expected that could perhaps be mined for strategies that increase student success.Finding courses that are working well can't help but inform responses to courses that could be more effective.
Moreover, connecting specific courses to student persistence for different types of students at different points in their degree paths can be used to improve student advising and placement.When possible, the quickest, most cost-effective solution to high DFW and attrition rates may be to make sure that particular kinds of students are not attempting courses that are known to be particularly difficult for students at their current stage of academic career.
Although the results reported in this paper are clearly limited to the undergraduate population at the university studied, they also plainly suggest that the effects of gateway courses are mediated by student types and academic stages.Thoughtful examination requires, then, adjusting observed course D/F/W rates for the characteristics of the students enrolled.In doing so, it is essential to identify courses whose performance issues are real, not apparent, so that institutional resources available for improving courses, high quality student advising and placement practices can be focused on those areas in which problems actually exist.

Table 1 .
Effect of Gap Analyses to Identify Gateway Courses for Notable Cases: Courses Ranked by Highest to Lowest DFW Rates and by Size of Gap (negative numbers indicate courses performing above expectation).

Table 2 .
Connecting a Course to Near Term Retention by Student Type and Level for Students Enrolled in a Particular Course.