Increasing Undergraduate Success: A Randomized Controlled Trial of U-Pace Instruction

U-Pace instruction, comprised of concept mastery and amplified assistance, has shown promise in increasing undergraduate success. To evaluate the efficacy of U-Pace instruction for students at risk for college noncompletion and students not at risk and to determine whether concept mastery, amplified assistance, or both U-Pace components are responsible for the greater learning associated with U-Pace instruction, an experiment was conducted with four instructional conditions (U-Pace, concept mastery, amplified assistance, and face-to-face). At a public university, 914 undergraduates (576 at risk) participated. U-Pace instruction produced greater learning on a proctored cumulative assessment than the comparisons. Additionally, U-Pace instruction produced greater academic success in terms of final course grades than face-to-face instruction. The percentage of final grades of A or B did not differ for concept mastery, amplified assistance, and U-Pace students. No interaction between instructional condition and risk status was found for final course grades or learning. The efficacy of U-Pace instruction for both at-risk students and students not at risk was supported.


Increasing Undergraduate Success: A Randomized Controlled Trial of U-Pace Instruction
Increasing undergraduate success is vital because as many as 36% of undergraduates enrolled in four-year public colleges in the United States drop out prior to their second year (ACT Inc., 2015). At two-year public institutions, this dropout rate is about 9% higher. There is a need for evidence-based interventions that can increase success in gateway courses, which enroll a large proportion of first-year students. Gateway courses are the prerequisites for many undergraduate programs, first courses in general education requirements, and the introduction to college for most students. Poor performance in gateway courses likely contributes to nonretention and may reflect failure to acquire fundamental skills and knowledge essential for academic success in coursework beyond the introductory level. Not acquiring the intended learning outcomes of gateway courses may create challenges to learning more advanced concepts, lead to poor performance in subsequent coursework, and contribute to non-retention in the sophomore, junior, and senior years. Thus, increasing student success in gateway courses is critical as mastery of skills and knowledge in these courses potentiates students' ability to complete their degree and attain their educational and vocational goals.

Concept Mastery
In concept mastery, also referred to as mastery-based learning, students typically master one content unit before progressing to a new content unit and often demonstrate mastery on small content units, and accordingly are frequently tested, which has been shown to enhance long-term retention (Karpicke & Roediger, 2008;McDaniel, Howard, & Einstein, 2009;Roediger & Karpicke, 2006) and transfer of learning (Carpenter, 2012). As students advance through a mastery-based course, they may develop a greater understanding of the effort required to succeed (achieve mastery and advance to the next unit). Since students typically control the pace with which they master course content (within the bounds of a semester), their anxiety about testing may be reduced. They can decide when they are ready to take an assessment and can retest (with different tests) if they do not reach the mastery criterion on any test attempt. Therefore, the greater control over learning and opportunity for personalized learning afforded by mastery-based instruction can improve mastery of content, but research also indicates that the greater student control can lead to slow progress in completing modules and lack of exposure to full curricular content (Bickerstaff, Fay, & Trimble, 2016).

Amplified Assistance
Amplified assistance is personalized support provided by the instructor to all students without them having to ask for help. Multiple lines of research support the importance of amplified assistance in student success. In a study of nearly 30,000 college graduates in the United States, researchers linked six key undergraduate experiences with long-term success (Gallup-Purdue Index, 2015). Amplified assistance is consistent with three of the key experiences: having a professor who made students excited to learn, who cared about them as individuals, and who encouraged them to reach for their goals. Amplified assistance is also aligned with multiple principles identified by Chickering and Gamson (1987) for good practice in undergraduate education, including frequent student-instructor contact, instructor feedback on student performance, and instructor communication of high expectations. Further, personalized feedback from instructors has been found to improve student-instructor relationships and students' receptiveness to feedback (Gallien & Oomen-Early, 2008). Simply having positive and informative interactions with instructors benefits students (Cox, McIntosh, Terenzini, Reason, & Quaye, 2010;Komarraju, Musulink, & Bhattacharya, 2010;Sax, Bryant, & Harper, 2005). To the extent that students feel a sense of connection with their instructors, college nonretention may be reduced (Tinto, 1987). Further support for amplified assistance is found in a national survey in which students identified instructor support as important for staying motivated and engaged in online courses (Lehman & Conceicao, 2014).
Amplified assistance and concept mastery are the components of U-Pace instruction. Amplified assistance in U-Pace instruction is tailored assistance provided weekly by email to all students during the process of learning without them having to ask for help. It has been shown that students most in need of help are unlikely to seek assistance (Karabenik & Knapp, 1988). All U-Pace students receive early, frequent support tailored to their needs. U-Pace instructors provide amplified assistance based on information about student engagement and performance readily available in learning management systems. The amplified assistance in U-Pace instruction includes both instructional support (e.g., study strategies, help with concepts not yet mastered) and messages intended to shape student behavior toward success through behavior-based praise and modify students' beliefs about themselves as learners. The strategic use of behavior-based praise (e.g., praise for effort, perseverance through obstacles, quality work, etc.) supports students' engagement as they work to master the course material and has been shown to induce positive responses and increased motivation (Willingham, 2005(Willingham, -2006. Likewise, the amplified assistance messages in U-Pace instruction support students' engagement as they address student beliefs that may be undermining motivation and learning. Research has shown that students' beliefs about their capacity to learn play an important role in their learning and can be effectively modified (Mueller & Dweck, 1998;Wilson & Linville, 1982, 1985Yeager & Dweck, 2014).

The Current Study
Past studies on U-Pace instruction were conducted in Introduction to Psychology, one of the largest courses enrolling undergraduates in the United States that attracts a diversity of students, including many at risk for college noncompletion. As such, this course provides an excellent context for interventions for student success. The first published study of U-Pace instruction found that U-Pace students earned a greater percentage of final grades of A or B and performed better both at the end of the course and 6 months later on a proctored cumulative exam compared to conventionally taught face-to-face students (Reddy et al., 2011). Subsequently, Reddy et al. (2013), found significant improvements in the rate of content mastery for at-risk U-Pace students (from the beginning to the end of the course), suggesting gains in self-regulated learning skills. The previous finding of a greater percentage of final grades of A or B in U-Pace instruction was replicated. In both studies, U-Pace and face-to-face students were taught by different instructors, although the course material was held constant. Preexisting differences between U-Pace and face-to-face students were not found in either study; however, students self-selected into U-Pace or face-to-face courses as opposed to being randomly assigned.
In Fleming et al. (2016), random assignment of undergraduates to instructional condition occurred at one of the two institutions in the study, but as in the two previous studies, U-Pace was only compared to face-to-face instruction. It was found that U-Pace instruction was associated with a greater percentage of final grades of A or B at both universities, located in geographically distinct areas, but the improvement in rate of content mastery for U-Pace students across the semester was found at only one of the two institutions. Whether the student outcomes associated with U-Pace instruction in past work would be found using a randomized controlled design (eliminating student self-selection into instructional condition as an explanation for findings) for both students at risk for college noncompletion and students not at risk is unclear. Further, U-Pace instruction integrates two empirically supported components-concept mastery and amplified assistance, and no prior study has examined whether amplified assistance with learning, concept mastery, or the combination of these core U-Pace components, are responsible for the outcomes associated with U-Pace instruction.
The current study utilized a randomized controlled design to address these questions. Objectively determined final course grades of A or B (academic success) and performance on a cumulative proctored exam (a measure of learning independent of final grades) were examined by instructional condition: U-Pace (concept mastery plus amplified assistance), concept mastery only (without amplified assistance), amplified assistance only (without the mastery requirement), or face-to-face (equivalent-sized sections of the same course that combined lecture, discussion, and interactive activities). As in past research on U-Pace instruction, Introduction to Psychology was selected as the course context for this experiment because it enrolls the second-highest percentage of undergraduates in their first year of college (after English Composition) and is taken by 60% of college students who have at least 10 undergraduate credits (Adelman, 2004). Thus, intervening in Introduction to Psychology provides an opportunity to impact many students early in college. It was hypothesized that U-Pace instruction would produce greater learning than the concept mastery, amplified assistance, and face-to-face comparison sections in which the course content, textbook, class size, and instructors were held constant. It was hypothesized that this greater learning with U-Pace instruction would be found for all students, including students at risk for college noncompletion due to various factors-academic unpreparedness, low family income, and/or factors associated with racial/ethnic minority status (Snyder, de Brey, & Dillow, 2016).

Methods
The randomized controlled trial (RCT) was conducted at a public, doctorate-granting university in the Midwest with 24,678 undergraduates. The university enrolls up to 75% of undergraduate applicants. The research was approved by the Institutional Review Board for the Protection of Human Participants. Informed consent was obtained for participation in the RCT and separately for participation in the proctored cumulative assessment of learning, independent of course requirements (not contributing to final course grades).

Recruitment and Participants
All undergraduates in the university interested in taking Introduction to Psychology in the academic year were invited to participate in the study, in which they would be enrolled in either a face-to-face or an online section of Introduction to Psychology. Students were informed that there would be a maximum of 60 students per section, that they would receive $150 for participating (approximately equivalent to the cost of a new textbook), even if they dropped or withdrew from the course, and that normal university policies and dates for dropping or withdrawing from the course and tuition would apply. Potential participants were also told that they had to be at least age 18 and taking Introduction to Psychology for the first time.
Altogether, 1,950 undergraduates took Introduction to Psychology. This number includes all sections of Introduction to Psychology, both in and outside the RCT. Of the 1,128 undergraduates who consented to participate in the RCT, 914 met eligibility requirements (e.g., not repeating Introduction to Psychology, no schedule conflict precluding face-to-face assignment). No student had prior experience with U-Pace instruction. Students were randomly assigned using a stratified procedure to evenly distribute students at risk for college noncompletion across instructional conditions (U-Pace, concept mastery, amplified assistance, and face-to-face). At-risk students were defined as students having low-income backgrounds (Federal Pell Grant eligible), racial/ethnic minority backgrounds showing gaps in graduation at most institutions (African American, Hispanic, Native American, or Southeast Asian), low standardized test scores for college admission (ACT composite scores of less than 19), and/or cumulative college grade point averages of less than 2.0 on a 0.0-4.0 scale. Table 1 presents the characteristics of students by instructional condition and in the overall study sample. Note. Due to missing data, sample sizes for comparisons vary slightly across variables. **Analysis of variance revealed a difference, p < .01. Scheffe post hoc tests indicated that the mean age of the U-Pace students did not differ from any of the other conditions. However, the mean age of the face-to-face students was less than the mean age of both the concept mastery and amplified assistance students. Chi-square analysis did not reveal any differences between condition in the percentages of at-risk, female, freshmen/sophomores, and firstgeneration college students.

Instructional Conditions
Two instructors separately taught all four instructional conditions (U-Pace, concept mastery, amplified assistance, and face-to-face). The course content and textbook were also held constant. The U-Pace, concept mastery, and amplified assistance instructional conditions were delivered online, whereas the face-to-face instructional condition was delivered in a physical classroom.
U-Pace. U-Pace students studied small content units and had to demonstrate mastery (score at least 90% on each associated quiz). U-Pace students who did not demonstrate mastery had to wait at least an hour before they could retake a quiz (with a different set of questions). Each quiz had to be completed within 7 minutes and consisted of 10 multiple-choice items that assessed understanding beyond memorization. Students received an immediate score but were not informed which quiz questions they got wrong because the objective was to master the concepts rather than learn the answer to specific quiz items. U-Pace students could not progress to the next content unit and quiz until they demonstrated mastery. They worked through each of the 24 content units and quizzes in this fashion. U-Pace students received amplified assistance in the process of achieving mastery (i.e., weekly personalized support messages by email). These messages were instructor initiated and contained instructional support (e.g., study strategies, tailored help with concepts not yet mastered), behavior-based praise to shape student behavior toward success, and elements aimed at modifying students' beliefs about themselves as learners. Instructors used field-tested U-Pace templates to tailor support messages in response to student needs inferred from the time elapsed since last quiz attempt and quiz scores, information typically found in learning management systems.
Concept mastery. Concept mastery students studied the same small content units and were required to demonstrate mastery as in the U-Pace instructional condition. They did not receive amplified assistance; that is, they did not receive instructor-initiated supportive email messages each week. However, the instructors supported students as they typically would, responding to students' requests for assistance (e.g., helping students with concepts and answering questions).
Amplified assistance. As in the U-Pace instructional condition, students in the amplified assistance condition received instructor-initiated supportive email messages weekly. Amplified assistance students studied the same content units and took the same associated quizzes as in the U-Pace and concept mastery instructional conditions. However, unlike the students in the U-Pace and concept mastery instructional conditions, amplified assistance students were not held to a mastery requirement; they could progress to new content and the associated quiz without scoring at least 90%.
Face-to-face. Face-to-face students participated in two 75-minute class periods per week that combined lecture, discussion, and interactive activities (e.g., working collaboratively in small groups, taking turns building on each other's contributions). Face-to-face students were not required to demonstrate mastery of the concepts before being exposed to new content and did not receive amplified assistance. However, the instructors supported students as they typically would, responding to students' requests for assistance. They held review sessions for exams and assisted students who requested help with concepts.

Instructors, Training, and Fidelity
The instructors were recognized for their undergraduate teaching excellence and had experience teaching the course. They were trained in person on how to implement the instructional conditions with fidelity, followed an implementation manual throughout the semester, and recorded on an ongoing basis the time spent on instructional activities (e.g., preparing for class, communicating with students by email, and helping students in person) using a log adapted from Worley and Tesdell (2009). Total time spent on instructional activities indicated that the instructors invested an equivalent amount of time in each instructional condition. High fidelity to instructional approach was found when the classes and course sites were checked. The instructors saved every email message (as they were sent to students and received from students) in separate e-folders for each instructional condition, and examination every two weeks also revealed high fidelity to instructional approach. For example, no students in the concept mastery or face-to-face instructional conditions received instructor-initiated weekly support messages, while all students in the amplified assistance and U-Pace instructional conditions did. The amplified assistance messages sent to students in both the amplified assistance and U-Pace conditions were evaluated to be of high quality and aligned with the templates, which specified that the email messages should contain help with concepts not yet mastered, behavior-based praise to shape student performance toward success, and elements to modify students' beliefs that they could be successful even if they were unsuccessful at the moment. Additionally, conversations with the instructors every other week throughout the fall and spring semesters revealed that fidelity to instructional condition was maintained.

Measures
Objectively determined final grades of A or B versus B-through F, and performance on a proctored cumulative assessment of learning independent of final grades were examined. Final grades of A or B were operationally defined as academic success for three reasons. Earning a final grade of at least B has been shown to be the threshold for a high likelihood of graduating (Denley, 2016). Academic honors are not commonly awarded for grades lower than a B, and a grade point average of 3.00 on a 4.00 scale is often the cutoff for selective programs of study.
For all four instructional conditions, grading was completely objective, based on students' performance on multiple-choice assessments. The questions for all instructional conditions were drawn from the same pool of assessment items and were used to form either quizzes for U-Pace, concept mastery, and amplified assistance students or larger exams for the face-to-face students (as is typical in these courses). Face-to-face students' final course grades were based on the mean score on their four exams. The final grades for U-Pace and concept mastery students were based on the number of content units mastered (scoring at least 90% on an associated quiz). Because amplified assistance students could progress without achieving mastery (scoring at least 90%), their final grades were based on the average of their high score on each of the 24 content unit quizzes.
Independent of final grades, learning was assessed using a cumulative exam comprised of questions testing application of knowledge rather than memorization of facts. This cumulative exam was created by subject matter experts, and the highest possible score was 50. The cumulative exam was administered in a proctored classroom with motivating instructions and did not count toward students' course grades. For ethical reasons, all students in the RCT were given the opportunity to take the cumulative exam and earn a $25 incentive. Half of the students from each instructional condition in the RCT were randomly selected to take the cumulative exam so their performance could be compared to the students in the RCT who were not selected a priori and also took the exam. Students were instructed to do their best, were required to put their name on the exam, work for a minimum of 30 minutes, and carefully check their answers before they could be dismissed.

Statistical Analyses
Analysis of variance was used to examine differences in performance by instructional condition and risk status on the proctored cumulative assessment of learning, independent of final grades. Performance on the proctored exam for students in the RCT who were randomly selected a priori to take the exam and those who were not randomly selected and took the exam was also compared using analysis of variance. Students' final grades by instructional condition and risk status were compared using binary logistic regression. The percentage of students earning final grades of A or B was compared to the percentage of students earning final grades below B, including students who withdrew from the course in the group with final grades below B. The percentage of students who withdrew from the course by instructional condition was compared using chi-square analysis. Chi-square analysis was also performed to examine whether the instructional conditions differed in the percentage of at-risk students after students withdrew from the course sections.
Objectively determined final grades of A or B versus B-through F were compared for atrisk students and students not at risk in the U-Pace, concept mastery, amplified assistance and face-to-face instructional conditions using binary logistic regression (see Figure 1). Both instructional condition and risk status predicted final grades of A or B. The odds of earning a final grade of A or B were 8.62 times higher for U-Pace students than face-to-face students (Wald χ 2 = 65.482, p < .001, Exp(B) = 8.62), did not differ for U-Pace students and concept mastery students (Wald χ 2 = 2.669, p = .102, Exp(B) = 1.51), and did not differ for U-Pace students and amplified assistance students (Wald χ 2 = 2.99, p = .083, Exp(B) = 1.53). The odds of earning a final grade of A or B also did not differ between concept mastery students and amplified assistance students (Wald χ 2 = 0.003, p = .959, Exp(B) = .988). The odds of earning a final grade of A or B were also higher for concept mastery (Wald χ 2 = 39.71, p < .001, Exp(B) = 5.68) and amplified assistance students (Wald χ 2 = 41.089, p < .001, Exp(B) = 5.692) than face-to-face students. At-risk students were significantly less likely to earn a final grade of A or B (Wald χ 2 = 8.218, p = .004, Exp(B) = 0.375) than students not at risk. No interaction was found between instructional condition and risk status (Wald χ 2 = 1.961, p = .581). Significant differences in performance were found by instructional condition on the proctored cumulative exam, the measure of learning independent of final course grades, F(3,572) = 21.38, p < .001, η 2 = .101. Scheffe tests revealed that U-Pace students (M = 66.92) scored significantly higher on the proctored cumulative exam than the concept mastery (M = 59.2, p < .001), amplified assistance (M = 55.30, p < .001), and face-to-face (M = 58.38, p < .001) students (see Figure 2). Scheffe tests revealed that the cumulative exam scores of the concept mastery, amplified assistance, and face-to-face students did not differ (all p values > .05). At-risk students (M = 58.39) scored significantly lower on the proctored cumulative exam than students not at risk (M = 62.32; F(1, 572) = 14.65, p < .001, η 2 =.025). No interaction between instructional condition and risk status was found, F(3, 572) = 0.13, p = .942, η 2 =.001. When including only the students in the RCT who were randomly selected a priori to take the cumulative exam in the analysis, the same finding held. That is, the U-Pace students scored significantly higher than the concept mastery, amplified assistance, and face-to-face students.
The mean exam scores were as follows: U-Pace = 67.00, concept mastery = 60.31, amplified assistance = 58.47, and face-to-face = 59.28, F(3, 263) = 6.35, p < .001. To assess whether a different random sample of students from the RCT would have yielded the same results, a resampling (bootstrap) procedure using 1,200 samples was performed, and the results revealed that the original random sample drawn to take the exam was a highly representative sample. None of the cumulative probabilities (all ps > .15) of the a priori mean values fell in the tails of the corresponding bootstrapped distributions, indicating the a priori sample could be considered representative of the distribution of all possible randomly drawn samples of exam scores.

Discussion
College completion affords the opportunity for economic advancement. Yet, a sizeable minority of students do not complete college. Most who drop out do so before the second year. Gateway courses dominate the first year of college. Mastery of skills and knowledge in these courses potentiates students' ability to complete their degree. Accordingly, this RCT was undertaken to evaluate the efficacy of U-Pace, an instructional intervention that has shown promise for increasing learning and academic success in a gateway course. Additionally, this RCT was conducted to determine for whom U-Pace instruction is effective (i.e., whether students not at risk and students at risk both benefit). This RCT also addressed why U-Pace instruction works (i.e., whether concept mastery, amplified assistance, or both components of U-Pace instruction are responsible for the greater learning reported in the literature). In this tightly controlled experiment, where students were randomly assigned to course sections in which the course content, textbook, class size, and instructors were held constant, U-Pace instruction produced greater learning on a proctored cumulative assessment than concept mastery, amplified assistance, and face-to-face instruction that combined lecture, discussion, and interactive activities. Thus, as hypothesized, the findings indicate that the greater learning associated with U-Pace instruction in past studies is due to the combination of concept mastery and amplified assistance, the two empirically supported components that comprise U-Pace instruction. Importantly, this greater learning (assessed independently of final grades with a proctored cumulative exam) was found for both at-risk U-Pace students and U-Pace students not at risk. Thus, the findings suggest that regardless of student risk status, U-Pace instruction produces greater learning; both the concept mastery and amplified assistance components of U-Pace instruction are necessary to achieve these effects. Confirming past findings (Fleming et al., 2016;Reddy et al., 2011Reddy et al., , 2013, U-Pace instruction not only produced greater learning on the proctored cumulative exam but also greater academic success in terms of final course grades than face-to-face instruction. The percentage of final course grades of A or B produced by U-Pace instruction did not differ from the percentages produced by concept mastery or amplified assistance, although as discussed, U-Pace instruction produced greater learning on the proctored cumulative exam than concept mastery, amplified assistance, and faceto-face instruction. If this RCT did not include two independent objective student outcomes, one might conclude from the final course grade findings in isolation that either the concept mastery component or the amplified assistance component of U-Pace instruction is sufficient. However, in tandem, the final course grade and cumulative exam findings indicate that both components of U-Pace instruction are necessary. The findings on the proctored cumulative assessment of learning indicate the clear superiority of U-Pace. Neither concept mastery nor amplified assistance produced greater learning than face-to-face instruction.
The outcomes found for U-Pace instruction may be understood in the context of several factors. First, the mastery requirement and amplified assistance messages in U-Pace instruction communicate high performance expectations to students. Second, U-Pace students are proactively supported by their instructor throughout the learning process to enable concept mastery. Finally, U-Pace instructors intentionally intervene to increase student motivation. Because motivational processes are key to student learning and achievement, especially when academic content is challenging or difficult, U-Pace instructors use information common to learning management systems to make inferences about the psychology of students.
Specifically, change in students' patterns of engagement and performance recorded in the learning management system signals opportunities to modify student beliefs that may be undermining motivation and learning. To effectively intervene, personalized proactive email messages are crafted from freely available, field-tested U-Pace templates grounded in psychological research and theory. In this way, the concept mastery and amplified assistance support components of U-Pace instruction work together and, as the findings of this experiment indicate, these components together produce student learning. This finding highlights the importance of a number of burgeoning research areas: instructor presence in online learning (Reupert, Maybery, Patrick, & Chittleborough, 2009), instructor-student rapport (Hagenauer & Volet, 2014), communicating caring in teaching (Wilson, 2008), motivational processes in learning and achievement (Zumbrunn, McKim, Buhs, & Hawley, 2014), the capacity of psychologically grounded interventions to shape student beliefs and behavior (Brady et al., 2016), the potential of personalized learning (Johnson et al., 2016), self-regulated learning effectiveness (Tsai, Shen, & Fan, 2013;Zimmerman, 2002), use of learner analytics to proactively support students (Clow, 2013), and the importance of requiring students to demonstrate mastery or competency (Klein-Collins, 2013). These promising areas of scientific inquiry are all aligned with U-Pace instruction.
Although this research has notable strengths, including randomization of nearly 1,000 undergraduates to instructional condition, incorporation of objective student outcomes, and examination of outcomes for both students at risk for college noncompletion and students not at risk, there are several limitations. Grades are determined differently in mastery approaches (U-Pace and concept mastery) and nonmastery approaches (amplified assistance and face-to-face). On the one hand, the opportunity to take multiple attempts to reach mastery could enhance grades, while on the other hand, failure to reach mastery may preclude students from being exposed to content, negatively affecting grades. Likewise, students in nonmastery courses are typically exposed to all material, potentially affecting grades positively. However, they are typically afforded only one opportunity to take an assessment, which may negatively affect grades. Despite this difference between mastery and nonmastery approaches in how grades are determined, in this study, the students randomly assigned to U-Pace instruction scored significantly higher on the proctored cumulative exam, independent of final course grades than the face-to-face, amplified assistance, and concept mastery students. Another limitation of this study is that U-Pace instruction was evaluated in a single context (Introduction to Psychology), albeit the context of previous research and the gateway course with the second highest enrollment in the first year of college nationally. While evaluating the effects of U-Pace instruction in this particular course context made sense, a shortcoming is that there is no single next course in which to examine students' subsequent performance. Students subsequently take many different courses and course sequences in line with a large number of intended majors that vary in difficulty and grading criteria, which complicates investigation of possible carryover effects. Future research aggregating data across numerous course sequences would allow statistical control of differences due to sequences of courses. Although learning management systems create efficiencies for monitoring student progress and proactively supporting all students with tailored assistance, future studies could examine whether amplified assistance integrated with concept mastery in a face-to-face classroom would produce comparable outcomes to U-Pace online instruction. Future RCTs in course contexts other than Introduction to Psychology are warranted to determine whether the effects of U-Pace instruction hold across other courses and/or disciplines. While not addressing generalizability across course and/or discipline, Fleming et al.'s (2016) findings suggest that the academic success produced by U-Pace instruction generalizes across universities with different student populations and learning management systems. Although students can memorize material to earn reasonable grades on quizzes and exams, whether they retain the material is an important question. The findings of another study (Reddy et al., 2011) suggest the greater learning produced by U-Pace instruction persists for at least a semester. Six months after the Introduction to Psychology course ended, the U-Pace students scored significantly higher on a second cumulative exam taken in a proctored setting compared to the conventionally taught face-to-face students. It has been demonstrated that mastery-based instruction leads to longer retention than non-mastery-based instruction (Kulik, Kulik, & Bangert-Drowns, 1990).
Future RCTs with sufficiently large sample sizes to provide statistical power could uncover the mediator(s) underlying the effects of U-Pace instruction for at-risk students and students not at risk. Identifying the mechanism of action would contribute to theory and possibly lead to new applications to benefit students. Whether U-Pace instruction changes students' understanding that they are capable of college-level work, perception of belonging in college, sense of control over learning, or other productive beliefs linked to learning and academic success could be examined. In this experiment, exposure to U-Pace instruction in a single gateway course was found to significantly impact student performance on a proctored cumulative exam and final course grades. Future studies could examine whether exposure to U-Pace instruction in multiple courses amplifies benefits. Both at-risk U-Pace students and U-Pace students not at risk for college noncompletion benefitted. The at-risk U-Pace students performed better on the proctored cumulative assessment of learning than the at-risk students randomly assigned to all of the other conditions. Further, the at-risk U-Pace students performed better on the proctored cumulative exam than the not at-risk students in the face-to-face, concept mastery, and amplified assistance instructional conditions. The findings of this randomized controlled trial significantly increase understanding of U-Pace instruction and provide the strongest level of evidence to date supporting its efficacy.