A total of 39,697 students who were enrolled at Columbus City Schools in the school year 2022-23 were considered in this study in which I investigated the effects of utilizing the Aleks and Iready educational platforms on students’ math outcomes across different grades. Five different models were employed to assess various aspects of platform usage and dosage effects. The results indicate that using both Aleks and Iready did not lead to significant improvements in 8th-grade OST Math scores or 9th-grade Algebra I scores when compared to not using either platform. However, students who engaged with either platform for a combined total of 30 hours or more demonstrated positive effects on 6th, 7th, and 8th-grade OST Math scores compared to those who engaged for less than 30 hours combined. Further, using Iready exclusively for more than 15 hours demonstrated small but statistically significant positive effects on 4th and 5th-grade OST Math outcomes, compared to using the platform for less than 15 hours. Conversely, the intensity of Aleks usage, whether exclusively or for more than 15 hours, did not significantly influence math performance.
The usage data also reveals important demographic patterns in platform usage. Notably, while elementary school students predominantly engage with Iready, high school students show a preference for Aleks, while middle school studenrs were predominately engaged in both platforms. Black students were the most represented across all platform usage categories, followed by White, Hispanic/Latino, Multiracial, and Asian students. Similarly, 17.3% of students who utilized both Aleks and Iready were identified as having an Individualized Education Plan (IEP), while 13.8% were classified as Limited English Proficiency (LEP) students. In contrast, among students who used neither platform, a higher proportion—22.3%—were identified as IEP students, while 18.2% were classified as LEP.
In this study, I apply propensity score matching using the nearest neighbor method, to estimate the Average Treatment Effect on the Treated (ATT) for OST scores following the use of the Aleks and Iready educational platforms. I ensure precise comparisons by exactly matching participants on demographic variables such as gender, race/ethnicity, and baseline test scores.
The analysis is structured around five key models:
For models incorporating dosage effects, I control for the percentage of overall lessons passed. This control is vital as successfully completing lessons reflects more meaningful engagement than merely attempting them. In contrast, for models without dosage effects, I do not control for lesson completion rates to avoid multicollinearity, given that no participants in the comparison group would have completed any lessons.
The propensity score matching method I use to estimate the Average Treatment Effect on the Treated (ATT) for using educational platforms Aleks and Iready can be represented by the following equation:
\[ \text{Match}_{i} = \alpha + \beta_1 \text{Grade}_i + \beta_2 \text{BaselinePerformanceLevel}_i + \beta_3 \text{PhysicalSchool}_i + \beta_4 \text{Gender}_i + \beta_5 \text{RaceEthnicity}_i + \epsilon_i \]
where:
After propensity score matching, I estimate the causal effect using a weighted linear regression model. The model equation is as follows:
\[ Y_{i} = \gamma + \theta_1 X_{1i} + \theta_2 X_{2i} + \theta_3 X_{3i} + \theta_4 X_{4i} + \theta_5 X_{5i} + \theta_6 X_{6i} + \epsilon_{i} \]
where:
weights, which are
derived from the propensity score matching to adjust for the covariate
balance and improve the precision of the estimates.All the data required for this project was sourced internally. Usage information for Aleks and Iready was obtained directly from their respective platforms. Test scores and enrollment details were gathered from internal data repositories at Columbus City Schools. This analysis is confined to the 2022-23 school year.
The data summary table below provides a detailed look at how students across various demographic backgrounds are using different educational tools. Among the total of 39,697 students, 90% of those in elementary schools are engaged with Iready Only. In contrast, high school students show a clear preference for the Aleks program, with 92.5% using it exclusively, suggesting that the choice of educational tools varies markedly with educational level. Middle school students are the most versatile, with 90.1% participating in both the Aleks and Iready programs.
Demographically, Black students are the most represented across all platforms, especially in Aleks Only where they make up 57.4% of the participants. White students are also well represented, especially in the Iready Only group. Students of Hispanic/Latino, Multiracial, and Asian backgrounds are present in considerable numbers, though they form smaller proportions of the groups. The distribution between male and female students is almost even, with a slight majority being male.
The table also highlights the participation of students with special educational needs. Notably, about 17% of all students have an Individualized Education Plan (IEP), with the highest concentration among those using neither platforms. 21.4% of all those who use Iready only are LEP students, while 14.5% of those who use Aleks only are LEP students. More highlights of demographic distribution is provided below.
| Aleks and Iready (N=7186) |
Aleks Only (N=8322) |
Iready Only (N=18866) |
Neither (N=5323) |
Overall (N=39697) |
|
|---|---|---|---|---|---|
| Grade | |||||
| Elementary | 188 (2.6%) | 2 (0.0%) | 16971 (90.0%) | 470 (8.8%) | 17631 (44.4%) |
| High School | 525 (7.3%) | 7701 (92.5%) | 194 (1.0%) | 4198 (78.9%) | 12618 (31.8%) |
| Middle School | 6473 (90.1%) | 619 (7.4%) | 1701 (9.0%) | 655 (12.3%) | 9448 (23.8%) |
| Race/Ethnicity | |||||
| Black | 3783 (52.6%) | 4775 (57.4%) | 9293 (49.3%) | 2878 (54.1%) | 20729 (52.2%) |
| White | 1362 (19.0%) | 1460 (17.5%) | 4190 (22.2%) | 1130 (21.2%) | 8142 (20.5%) |
| Multiracial | 553 (7.7%) | 497 (6.0%) | 1668 (8.8%) | 343 (6.4%) | 3061 (7.7%) |
| Hispanic/Latino | 1252 (17.4%) | 1353 (16.3%) | 3050 (16.2%) | 777 (14.6%) | 6432 (16.2%) |
| Asian | 212 (3.0%) | 210 (2.5%) | 594 (3.1%) | 175 (3.3%) | 1191 (3.0%) |
| Native American/Alaskan Native | 19 (0.3%) | 22 (0.3%) | 54 (0.3%) | 16 (0.3%) | 111 (0.3%) |
| Native Hawaiian/Pacific Islander | 5 (0.1%) | 5 (0.1%) | 17 (0.1%) | 4 (0.1%) | 31 (0.1%) |
| Gender | |||||
| Male | 3615 (50.3%) | 4172 (50.1%) | 9762 (51.7%) | 2836 (53.3%) | 20385 (51.4%) |
| Female | 3571 (49.7%) | 4150 (49.9%) | 9104 (48.3%) | 2487 (46.7%) | 19312 (48.6%) |
| Gifted | |||||
| Yes | 737 (10.3%) | 832 (10.0%) | 1390 (7.4%) | 565 (10.6%) | 3524 (8.9%) |
| No | 6449 (89.7%) | 7490 (90.0%) | 17476 (92.6%) | 4758 (89.4%) | 36173 (91.1%) |
| Individualized Education Plan | |||||
| Yes | 1243 (17.3%) | 1244 (14.9%) | 3039 (16.1%) | 1189 (22.3%) | 6715 (16.9%) |
| No | 5943 (82.7%) | 7078 (85.1%) | 15827 (83.9%) | 4134 (77.7%) | 32982 (83.1%) |
| Limited English Proficiency | |||||
| Yes | 993 (13.8%) | 1205 (14.5%) | 4046 (21.4%) | 971 (18.2%) | 7215 (18.2%) |
| No | 6193 (86.2%) | 7117 (85.5%) | 14820 (78.6%) | 4352 (81.8%) | 32482 (81.8%) |
The results from the study indicate that the combined usage of Aleks and Iready did not lead to significant improvements in 8th-grade OST Math scores or 9th-grade Algebra I scores compared to not using either platform. This suggests that while these platforms may offer valuable educational resources individually, their combined usage did not result in notable enhancements in math performance. However, when you factor in the dosage effect, we do observe some positive outcomes. Students who spent more than 30 combined hours on Aleks and Iready demonstrated significant positive effects on 6th, 7th, and 8th-grade OST Math scores, compared to those who used the platforms for less than 30 hours. This effect, however, did not extend to Algebra I scores, suggesting that prolonged engagement with these platforms may primarily benefit foundation math skills rather than advanced algebraic concepts.
When examining the impact of using Aleks exclusively, results showed no statistically significant difference in 9th-grade Algebra I outcomes between students who exclusively used Aleks and those who used neither of the two platforms. Similarly, there was no substantial impact on outcomes when comparing students who used Aleks exclusively for more than 15 hours to those who used it for less than 15 hours, indicating that the intensity of Aleks usage did not significantly influence math performance.
In contrast, when analyzing the impact of using Iready exclusively, students who used Iready exclusively for more than 15 hours demonstrated small but statistically significant positive effects on 4th and 5th-grade OST Math outcomes compared to those who used it for less than 15 hours. This suggests that increased usage of Iready may lead to slight improvements in math performance at the elementary level.
The regression output above in Table 1 compares the effect of having 30 or more hours of combined lessons on Aleks and Iready with having less than 30 hours on OST Math scores across different grades. The results show significant positive effects for the treatment group on 6th (coefficient = 0.058, p < 0.05), 7th (coefficient = 0.103, p < 0.001), and 8th-grade OST Math scores (coefficient = 0.112, p < 0.001). However, the effect does not extend to Algebra I scores (coefficient = 0.118, p > 0.05). Moreover, earlier grade OST Math scores significantly influence later grades’ scores, suggesting cumulative academic progress. The model is based on Propensity Score Matching with the nearest neighbor method, incorporating exact matching on key covariates such as school, gender, race/ethnicity, and baseline test placement levels. In this model, I also control for the percentage of lessons passed.
| 6th Grade MATH | 7th Grade MATH | 8th Grade MATH | Algebra I | |
|---|---|---|---|---|
| Treatment Effect (30+Aleks+Iready) | 0.058* | 0.103*** | 0.112*** | 0.014 |
| (0.025) | (0.022) | (0.023) | (0.133) | |
| Percent Lessons Passed | 0.002 | 0.002* | 0.001 | 0.008 |
| (0.001) | (0.001) | (0.001) | (0.006) | |
| 5th Grade OST MATH | 0.812*** | |||
| (0.025) | ||||
| 6th Grade OST MATH | 0.657*** | |||
| (0.026) | ||||
| 7th Grade OST MATH | 0.498*** | |||
| (0.029) | ||||
| 8th Grade OST MATH | 0.459 | |||
| (0.965) | ||||
| Female | 0.001 | 0.012 | 0.042 | 0.281 |
| (0.026) | (0.024) | (0.024) | (0.200) | |
| White | 0.117** | 0.066 | 0.037 | 0.113 |
| (0.036) | (0.034) | (0.045) | (0.329) | |
| Multiracial | 0.164** | −0.035 | −0.012 | |
| (0.058) | (0.059) | (0.053) | ||
| Hispanic/Latino | −0.005 | −0.024 | 0.032 | 0.195 |
| (0.034) | (0.034) | (0.032) | (0.320) | |
| Asian | 0.053 | 0.116 | 0.161 | |
| (0.094) | (0.109) | (0.165) | ||
| Num.Obs. | 968 | 838 | 782 | 130 |
| R2 Adj. | 0.689 | 0.563 | 0.439 | 0.282 |
| * p < 0.05, ** p < 0.01, *** p < 0.001 | ||||
| Other Controls: School |
The regression estimates for the treatment effect of using both Aleks and Iready compared to using neither on 8th-grade OST Math scores are 0.027 with a standard error of 0.080. For Algebra I scores, the estimate is -0.057 with a standard error of 0.155. However, both of these estimates are not statistically significant. The model is based on Propensity Score Matching with the nearest neighbor method, incorporating exact matching on key covariates such as school, gender, race/ethnicity, and baseline test placement levels. Results indicate that using both Iready and Aleks as opposed to using neither does not necessarily yield a positive score outcomes on 8th Grade and 9th Grade Algebra I OST tests.
| 8th Grade OST MATH | Algebra I | |
|---|---|---|
| Treatment Effect (Aleks and Iready) | 0.027 | −0.057 |
| (0.080) | (0.155) | |
| 7th Grade OST MATH | 0.266* | |
| (0.124) | ||
| 8th Grade OST MATH | 0.424 | |
| (0.533) | ||
| Female | 0.202 | 0.188 |
| (0.121) | (0.176) | |
| White | −0.177 | |
| (0.375) | ||
| Multiracial | −0.291 | −0.059 |
| (0.263) | (0.317) | |
| Hispanic/Latino | 0.385 | 0.301 |
| (0.301) | (0.217) | |
| Num.Obs. | 42 | 96 |
| R2 Adj. | 0.288 | 0.224 |
| * p < 0.05, ** p < 0.01, *** p < 0.001 | ||
| Other Controls: School |
This model considers the treatment effect of using Aleks only for more than 15 hours on 9th-grade Algebra I outcomes, contrasting it with usage of less than 15 hours. The treatment effect estimate for Aleks usage exceeding 15 hours is insignificant (- 0.011), indicating no substantial impact on outcomes compared to lower usage. Additionally, the model considers factors such as the percentage of lessons passed and performance in 8th-grade math standardized tests. The model is based on Propensity Score Matching with the nearest neighbor method, incorporating exact matching on key covariates such as school, gender, race/ethnicity, and baseline test placement levels.
| Algebra I | |
|---|---|
| Treatment Effect (Aleks Only 15+ Hours) | −0.011 |
| (0.062) | |
| Percent Lessons Passed | 0.002 |
| (0.001) | |
| 8th Grade OST MATH | 1.177*** |
| (0.098) | |
| Female | 0.047 |
| (0.056) | |
| White | 0.182* |
| (0.079) | |
| Multiracial | 0.222 |
| (0.138) | |
| Hispanic/Latino | 0.065 |
| (0.077) | |
| Asian | 0.100 |
| (0.315) | |
| Num.Obs. | 538 |
| R2 Adj. | 0.450 |
| * p < 0.05, ** p < 0.01, *** p < 0.001 | |
| Other Controls: School |
This summary analyzes the impact of using Aleks exclusively compared to not using it at all on 9th-grade Algebra I outcomes. The treatment effect of using Aleks exclusively is statistically insignificant (- 0.004), indicating no significant difference in outcomes compared to not using Aleks. The model is based on Propensity Score Matching with the nearest neighbor method, incorporating exact matching on key covariates such as school, gender, race/ethnicity, and baseline test placement levels.
| Algebra I | |
|---|---|
| Treatment Effect (Aleks Only) | −0.004 |
| (0.068) | |
| 8th Grade OST MATH | 0.749*** |
| (0.173) | |
| Female | −0.106 |
| (0.068) | |
| White | 0.209 |
| (0.132) | |
| Multiracial | 0.193 |
| (0.171) | |
| Hispanic/Latino | 0.044 |
| (0.118) | |
| Asian | 0.242 |
| (1.677) | |
| Num.Obs. | 518 |
| R2 Adj. | 0.410 |
| * p < 0.05, ** p < 0.01, *** p < 0.001 | |
| Other Controls: School |
This model considers the treatment effect of using Iready only for more than 15 hours on 4th and 5th-grade OST Math outcomes, contrasting it with usage of less than 15 hours. The treatment effect coefficients for Iready usage exceeding 15 hours are statistically significant for both 4th and 5th-grade math outcomes (0.063* and 0.060* respectively), indicating a small positive effect on performance compared to lower usage. Additionally, the model considers factors such as the percentage of lessons passed and performance in 8th-grade math standardized tests. The model is based on Propensity Score Matching with the nearest neighbor method, incorporating exact matching on key covariates such as school, gender, race/ethnicity, and baseline test placement levels.
| 4th Grade OST Math | 5th Grade OST Math | |
|---|---|---|
| Treatment Effect (Iready Only 15+ Hours) | 0.063* | 0.060* |
| (0.029) | (0.026) | |
| Percent Lessons Passed | 0.002 | 0.002 |
| (0.001) | (0.001) | |
| 3rd Grade OST MATH | 0.642*** | |
| (0.036) | ||
| 4th Grade OST MATH | 0.656*** | |
| (0.029) | ||
| Female | −0.017 | 0.006 |
| (0.032) | (0.028) | |
| White | 0.068 | 0.102* |
| (0.050) | (0.043) | |
| Multiracial | 0.074 | 0.065 |
| (0.078) | (0.059) | |
| Hispanic/Latino | 0.069 | −0.002 |
| (0.054) | (0.043) | |
| Asian | −0.053 | |
| (0.109) | ||
| Num.Obs. | 712 | 722 |
| R2 Adj. | 0.677 | 0.724 |
| * p < 0.05, ** p < 0.01, *** p < 0.001 | ||
| Other Controls: School |
The findings of this analysis are subject to several limitations. Firstly, the data used for this study is restricted to the 2022-2023 academic year, potentially limiting the generalizability of the results to other time periods. Secondly, the matching method employed in this analysis only controls for observed differences between groups, and thus, it’s possible that there are unobserved differences between students who use one or more of the platforms and those who use neither that could significantly alter our estimates. Additionally, data shows that middle schoolers predominantly use the Aleks platform for grade-level content, while high-level Aleks usage varies, with some students using it for enrichment purposes and others for makeup work. This discrepancy might also be leading to differences in effect sizes across grade levels.
Furthermore, while a dosage effect was observed when comparing students who used Aleks and Iready for more than 30 hours compared to those who used them for less than 30 hours, no difference was found between using both platforms and using neither. This discrepancy might be attributed to several factors, such as students who use neither already performing well and not needing additional support, or other reasons not captured in the data. Additionally, results are limited to data availability, and outcomes with insufficiently large comparable sizes were not analyzed. These limitations highlight the need for caution in interpreting the results.