Impact of a Web-Based Adaptive Supplemental Digital Resource on Student Mathematics Performance

Much literature has presented evidence that supplemental digital resources enhance student performance with mathematics. The purpose of this study was to explore the impact of a webadaptive digital resource, Think Through Math©, on student performance with state-mandated annual standardized mathematics assessments. This study utilized a quantitative research design and conducted multiple regression analyses among 723 students from a north Texas school district that showed high-levels of use with Think Through Math© during the 2015-2016 school year. Data from Think Through Math© reports and annual standardized mathematics assessments were collected and analyzed from elementary, intermediate, junior high, and high school campus levels. Results of the multiple regression analyses were reported for each campus level, as well as several statistically significant and positive associations.


Impact of a Web-Based Adaptive Supplemental Digital Resource on Student Mathematics Performance
Annual standardized assessments have become the predominant means through which student learning is measured in the United States.Numerous studies have ascertained that traditional supplemental digital resources, such as apps, websites, videos, and software, are effective tools to enhance student performance with mathematics (e.g., Boster et al., 2007;Foster, Anthony, Clements, Sarama, & Williams, 2016;Kiriakidis & Geer, 2014;Securro, Jones, Cantrell, & Blackwell, 2006).While these studies have presented research-based findings to substantiate the positive effects associated with the use of supplemental digital resources during math instruction, the literature has also pointed to possible disadvantages.For example, supplemental digital resources may be cost-prohibitive for schools, as well as time-prohibitive for teachers (Clark & Whetstone, 2014).Also, supplemental digital resources may have limited effects on student subpopulations, such as underperforming students and English language learners (Rutherford et al., 2014).Moreover, perceptions related to the benefits and value of supplemental digital resources among teachers may influence how they implement and use these resources with students (Martindale, Pearson, Curda, & Pilcher, 2005).
In Texas, student performance in reading, mathematics, writing, science, social studies, English I, English II, Algebra I, biology, and U.S. History is measured annually through the State of Texas Assessments of Academic Readiness (STAAR) program (Texas Education Agency, 2016a).Within the content area of mathematics, students are assessed in Grades 3-8 and also complete end-of-course (EOC) assessments for Algebra I. Beginning in 1999, Texas enacted state legislation, currently known as the Student Success Initiative (SSI), which ties grade advancement for students in Grades 5 and 8 to successful completion of the required STAAR mathematics assessments (Texas Education Agency, 2016b).As schools implement these requirements for students to move to the next grade level, Texas sanctioned the development of various supplemental digital resources to support performance with mathematics among all students with the Texas Students Using Curriculum Content to Ensure Sustained Success (SUCCESS) program (Texas Education Agency, 2016c).
Through the SUCCESS program, Think Through Math © was designated as one of the statefunded vendors to support mathematics instruction for students in Grades 3-8 and Algebra (Texas Education Agency, 2016c).Consequently, a large number of school districts in Texas have provided students with access to Think Through Math © (Think Through Learning, Inc., 2016b).Designed as a web-based adaptive digital resource, Think Through Math © provides supplemental math instruction that aligns with the state-mandated curriculum accessible at any given time.Unlike traditional supplemental digital resources that have been used to support mathematics instruction (i.e., apps, websites, videos, and software), Think Through Math © uses a guided instructional framework that includes many levels of support.This guided instructional framework begins by administering an initial placement test in order to determine each student's level of understanding and gauge their readiness level for instruction.Once this level has been determined, students receive individualized instruction as they advance through interactive, adaptive lessons.During each lesson, students receive instant feedback and also have access to live, qualified math specialists.Throughout the school year, there are also benchmark assessments that the teacher may assign to students in order to obtain a measure of their growth and performance.In order to evaluate the effectiveness of Think Through Math © , this study sought to explore the following question: What impact does high usage of Think Through Math © have on student performance with STAAR mathematics assessments?

Methods Participants
The population of this study included male and female students who were enrolled in 3 rd grade -8 th grade and Algebra I classes throughout a school district located in northern Texas.Participants were selected based upon their level of usage with Think Through Math © during the 2015-2016 school year.Think Through Math © reports were reviewed and revealed that high-level users with the supplemental digital resource were associated with specific teachers at four different school campus levels (see • number of lessons passed by scoring an 80% of higher on the pre-quiz or a 70% or higher on the post-quiz; • percentage of successful completion for on grade level lessons; • percentage of successful completion for below grade level lessons; • number of lessons passed by pre-quiz (i.e., scored 80% or higher); • pre-quiz average; • post-quiz average; • average number of problems attempted; • average number of earned points; • average number of times that the "Help" feature was accessed during lessons; • average number of times that the "Live Help" feature was accessed during lessons; • total amount of time spent in lessons; • total amount of time spent in lessons during school hours; and • total amount of time spent in lessons weekdays outside of school hours and all day Saturday and Sunday.
Data collected were entered into IBM SPSS Statistics 22 software, and data sets for each campus level were created for multiple linear regression analyses.STAAR scale scores were assigned as the dependent variable, and data obtained from Think Through Math © reports were operationalized and assigned as independent predictor variables (see Figure 1).Missing values in each data set were treated using pairwise deletion, which included all data except for specific missing values during analyses (Field, 2013).The following null hypotheses were established: H01: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the elementary campus level.
H02: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the intermediate campus level.H03: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the junior high campus level.H04: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the high school campus level.Earned Points Figure 1.Variables for multiple regression analyses.Correlations were revealed among seven independent predictor variables.The independent predictor variable with one asterisk was retained for analyses, and independent predictor variables with a double asterisk were removed from analyses.
Each data set was examined separately to ensure that all assumptions for a multiple regression analysis were satisfied (Osborne & Waters, 2002).Initial analyses of each data set revealed the presence of multicollinearity, which detected correlations among seven independent predictor variables.Upon closer inspection of these variables, it was determined that each provided information related to user performance with lessons.Based upon this redundancy with information, Lessons Attempted was retained as an independent predictor variable and the other six were removed from additional analyses with each data set (see Figure 1).After these modifications were made, follow-up analyses confirmed that each data set satisfied all assumptions for a multiple regression analysis.With each separate analysis (i.e., elementary campus level, intermediate campus level, junior high campus level, and high school campus level), all 11 independent predictor variables were entered into the regression model at the same time (Field, 2013).

Elementary Campus Level
A multiple regression analysis was conducted to test the following null hypothesis: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the elementary campus level.Descriptive statistics for this analysis were reported in Table 2 Results showed that Think Through Math © usage explained a significant amount of variance in student performance (n = 90) with the Grade 4 STAAR Mathematics assessment (F(10, 72) = 10.91,p = 0.00), with an adjusted R 2 of .547.These findings suggested that the model accounts for approximately 55% of variance in students' STAAR mathematics assessment scores than would be explained by chance.Based on this finding, the null hypothesis was rejected.As shown in Table 2, this analysis further showed two statistically significant and positive relationships with Think Through Math © independent predictor variables: • Lessons Attempted (β = .27,t(82) = 2.90, p = .01);and • On Grade Level Pass Rate (β = .35,t(82) = 2.73, p = .01).
These findings indicated that these two aspects of Think Through Math © usage have predictive ability on students' STAAR mathematics assessments at the elementary campus level.

Intermediate Campus Level
A multiple regression analysis was conducted to test the following null hypothesis: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the intermediate campus level.Descriptive statistics for this analysis were reported in Table 3 Results showed that Think Through Math © usage explained a significant amount of variance in student performance (n = 315) on the Grade 6 STAAR Mathematics assessment (F(10, 286) = 48.19,p = 0.00), with an R 2 of .628.These findings suggested that the model accounts for approximately 63% of variance in students' STAAR mathematics assessment scores than would be explained by chance.Based upon this finding, the null hypothesis was rejected.As shown in Table 3, this analysis further showed five statistically significant and positive relationships with Think Through Math © independent predictor variables: • Placement Performance Level (β = .14,t(296) = 2.85, p = .01); • Most Recent Benchmark Performance Level (β = .21,t(296) = 4.78, p = .00); • Lessons Attempted (β = .24,t(296) = 6.01, p = .00); • On Grade Level Pass Rate (β = .51,t(296) = 9.03, p = .00);and • School Time (β = .18,t(296) = 4.34, p = .00).
These findings indicated that these five aspects of Think Through Math © usage have predictive ability on students' STAAR mathematics assessments at the intermediate campus level.

Junior High Campus Level
A multiple regression analysis was conducted to test the following null hypothesis: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the junior high campus level.Descriptive statistics for this analysis were reported in Table 4 Results showed that Think Through Math © usage explained a significant amount of variance in student performance (n = 259) on the Grade 7 STAAR Mathematics and Grade 8 STAAR Mathematics assessments (F(10, 209) = 48.52,p = 0.00), with an R 2 of .699.These findings suggested that the model accounts for approximately 70% of variance in students' STAAR mathematics assessment scores than would be explained by chance.Based upon this finding, the null hypothesis was rejected.As shown in Table 4, the analysis further showed five statistically significant and positive relationships with Think Through Math © independent predictor variables: • Placement Performance Level (β = .19,t(219) = 2.98, p = .00);• Most Recent Benchmark Performance Level (β = .30,t(219) = 4.52, p = .00); • Lessons Attempted (β = .25,t(219) = 5.47, p = .00);• On Grade Level Pass Rate (β = .36,t(219) = 4.84, p = .00);and • Below Grade Level Pass Rate (β = .13,t(219) = 2.02, p = .00).
These findings indicated that these five aspects of Think Through Math © usage have predictive ability on students' STAAR mathematics assessments at the junior high campus level.

High School Campus Level
A multiple regression analysis was conducted to test the following null hypothesis: Think Through Math © usage does not have an impact on student performance with STAAR mathematics assessments at the high school campus level.Descriptive statistics for this analysis were reported in Table 5 Results showed that Think Through Math© usage did not explain a statistically significant amount of variance in student performance on the STAAR mathematics assessments (F(10, 44) = 1.84, p = 0.08), with an adjusted R 2 of .135.These findings suggested that the model was not statistically significant; therefore, the null hypothesis was not rejected.

Campus Level Overview, Performance Levels, and Progress Measures
An overview for each campus level was created that included specific information related to the scale score state performance standards for STAAR mathematics assessments administered during in March/Spring 2016 (see Table 6).As described previously, Lessons Attempted was an independent predictor variable that had a statistically significant and positive relationship on students' STAAR mathematics assessments at the elementary, intermediate, and junior high campus levels.As shown in Table 6, students at the intermediate campus level attempted a higher number of Think Through Math © lessons (M = 85.53) and achieved higher STAAR scale scores (M = 1735.07)related to state performance standards articulated for these grade levels.Similarly, students at the high school campus level attempted the lowest number of Think Through Math © lessons (M = 56.18)and also achieved lower STAAR scale scores (M = 3660.55)related to state performance standards articulated for Algebra I. Examination of available performance level ratings among students included in data analyses presented interesting findings.As shown in Table 7, almost all of the high level Think Through Math © users at the intermediate campus level (n = 288, 96%) earned a Level II: Satisfactory Academic Performance rating.This same rating was also earned by over two-thirds of the high level users at the junior high campus level (n = 187, 80%) and more than half of the high level users at the elementary campus level (n = 56, 65%).Over 40% (n = 122) of the high level Think Through Math © users at the intermediate campus level earned the Level III: Advanced Academic Performance rating, as well as 37% (n = 32) of the high level users at the elementary campus level and 25% (n = 59) of the high level users at the junior high campus level.Conversely, Think Through Math © users at the high school campus level earned lower academic performance ratings.Finally, STAAR progress measures also revealed the greatest performance among high level Think Through Math © users at the intermediate campus level.Over 60% (n = 174) of these students demonstrated the expected amount of annual academic growth.Similarly, almost half of the high level users at the elementary campus level (n = 38, 47%) and the junior high campus level (n = 98, 43%) demonstrated the same level of expected academic growth.Further analysis with progress measure data showed that almost 30% (n = 80%) of high level Think Through Math © users at the intermediate campus level exceeded the expected amount of annual academic growth.Not surprisingly, almost all of the Think Through Math © users at the high school campus level did not meet the expected amount of annual academic growth.

Discussion
The findings from this study aligned with previous studies that demonstrated positive effects associated with the use of supplemental digital resources and students' mathematics performance among students at the elementary, intermediate, and junior high levels (e.g., Chappell, Arnold, Nunnery, & Grant, 2015;Clark & Whetstone, 2014;Kiriakidis & Geer, 2014;Martindale et al., 2005;Nunnery & Ross, 2007;Ysseldyke & Bolt, 2007).Among each campus level, Think Through Math © usage accounted for more than half of the variance in students' STAAR mathematics assessment scores than would be explained by chance.Moreover, findings reported for each campus level identified aspects of Think Through Math © that had statistically significant and positive predictive ability on students' mathematics performance with the STAAR assessments, particularly at the intermediate and junior high campus levels.Notably, two specific aspects of Think Through Math © (i.e., the number of lessons attempted and the number of on grade level lessons passed) were identified as significant and positive predictors for students' mathematics performance for all campus levels.
Findings from this study did not demonstrate positive effects resulting from Think Through Math © usage among high level users at the high school campus level.Previously published studies have revealed mixed findings related to the use of supplemental digital resources among high school students (e.g., Campuzanno, Dynarski, Agodini, & Rall, 2009;Cavalluzzo, Lowther, Mokher, & Fan, 2012;Martindale et al., 2005;Smith & Suzuki, 2015).However, this campus level included the smallest number of high level Think Through Math © users.With this in mind, it is recommended that additional research be conducted that explores the impact of Think Through Math © usage with high school students' mathematics performance.
Findings from this study have suggested that higher levels of Think Through Math © usage correspond to greater levels of performance on STAAR mathematics assessments.As shown in the campus level overview, the highest level of activity and STAAR scale scores were reported among Think Through Math © users at the intermediate campus level.Additionally, nearly all of these users achieved a satisfactory academic performance rating (i.e., Level II) and almost half achieved an advanced academic performance rating (i.e., Level III) on their STAAR mathematics assessments.Furthermore, when compared to high level Think Through Math © users at the elementary, junior high, and high school levels, users at the intermediate campus level exhibited the highest level of expected annual academic growth, as well as annual academic growth that exceeded the expected amount.These findings have presented empirical evidence that both corroborate and extend findings from previously published case studies related to Think Through Math © usage among students in Texas (Think Through Learning, Inc. 2016a).
Finally, findings from this study have pointed to the importance of sustained fidelity with implementation of supplemental digital resources.It is vital to acknowledge that "teacher adherence to and students' engagement" with supplemental digital resources have a larger impact on student performance than time (Crawford, Carpenter II, Wilson, Schmeister, & McDonald, 2012, p. 233).It is equally important that teachers utilize supplemental digital resources with the same level of attention, diligence, and rigor as a teacher-led instructional supplement.Also, teachers must continually monitor students' progress with supplemental digital resources that are selected as extensions for mathematics instruction (Bolt, Ysseldyke, & Patterson, 2010).
As school leaders strive to use supplemental digital resources with fidelity, it is imperative that they provide teachers with initial training and subsequent professional development (Clark & Whetstone, 2014).Teachers must be familiar with the components and structure of selected supplemental digital resources as well as the corresponding guidelines for use (Crawford et al., 2012).School leaders must also recognize that the implementation of selected supplemental digital resources is heavily influenced by beliefs and perceptions among teachers (Clements, Sarama, Wolfe, & Spitler, 2015).Teachers must be familiar with the advantages associated with the use of selected supplemental digital resources as well their benefits to students (Clark & Whetstone, 2014).Therefore, school leaders should make strategic efforts to collect empirical evidence regarding the impact that selected supplemental digital resources has on student performance and share this data with teachers.Teachers are more likely to support "sustained fidelity of implementation to a program that has demonstrated improved child achievement" (Clements et al., 2015, p. 445).

Limitations
Although this study presented promising findings related to the impact that Think Through Math © usage has on student performance, there were a few limitations.First, this analysis was limited to high level Think Through Math © users.As a preliminary analysis, exploring the impact that Think Through Math © usage had on overall student performance was appropriate.However, it is recommended that follow-up studies compare the academic performance of students who have varying levels of Think Through Math © usage, including students who have none.Further, we recommend that analyses explore the impact of usage among gender, race/ethnicity, socioeconomic status, and special student populations (e.g., bilingual students, English language learners, gifted and talented learners).With this in mind, additional studies may identify optimal concentrations of Think Through Math © usage, such as the identification of an optimal number of lessons attempted or an ideal number of on grade level lessons that a student must pass, among specific groups of students.
Another limitation of this study was its exclusion of perceptions among stakeholders.Since many findings reported presented positive results related to Think Through Math © usage, it is recommended that follow-up studies utilize qualitative methods to explore how implementation of this supplemental digital resource was perceived by students, teachers, school administrators, and stakeholders beyond the school campus.Since previously published studies have highlighted the potential influence of teachers' perceptions of the use of supplemental digital resources (Clark & Whetstone, 2014;Clements et al., 2015), it is recommended that additional studies be conducted that include all stakeholders.

Author Note
This study was conducted in order to evaluate the effectiveness of Think Through Math © as a supplemental digital resource for mathematics instruction.We are not employees of the company, and we were not compensated in any way for our work.Our primary goal was to explore the value of Think Through Math © as an online teaching and learning tool

Table 1 )
. Based upon this trend, identified high-level users were categorized by the district's campus level: elementary, intermediate, junior high, and high school.

Table 1 .
High Usage Think Through Math © Users by Campus and Grade Level

Table 4 .
. Results-Junior High Campus Level

Table 5 .
. Results-High School Grade Level

Table 6 .
Campus Level Overview with Scale Score State Performance Standards

Table 7 .
STAAR Academic Ratings and Progress Measures