School-Level Analyses of NWRI

Author

NWRI D Team

Takeaways

For grades 3-5, there is a positive relationship between a school’s NWRI enrollment % and a school’s average FAST ELA score (PM2, Grades 3-5, 2024-25). This relationship is strongest for schools with higher proportions of economically disadvantaged students which are also schools that have higher than average eligibility and enrollment numbers (across K-5).

MAIN POINTS: Eligibility, Enrollment, and Economic Disadvantage across schools for Grades K through 5

  1. Data shows that schools with the highest percentage of eligible students for grades K through 5 have higher percentages of economically disadvantaged students. This pattern is NOT an artefact of Title I vs. Non-Title I schools (Figure 1 below). Additionally, schools above the average eligibility percentage (dashed lines Figure 1), are schools with higher percentages of English Language Learners (ELLs).

  2. There are 700 schools that have a lower than average enrollment and higher than average eligibility (Figure 2) and they make up 31 % of all schools in the analysis. These schools on average have 92.6% of their student enrollment classified as economically disadvantaged while the other schools average at 69.6%.

Figure 1. Scatterplots showing the relationship between NWRI eligibility percentage (by school) against the percent of economically disadvantaged students (also by school) across Title I and Non-Title I schools (left and right). Colors of the points represent the percentages of students classified as English Language Learners (ELL) in these schools. Orange horizontal lines represent the average eligibility percentage for Title I and Non-Title I schools.

MAIN POINTS: Impacts of NWRI enrollment on FAST Reading Scores for Grades 3-5 PM2 2024-25

  1. Increasing NWRI Enrollment is significantly associated with higher average FAST ELA scores BUT this effect is moderated by the percent of economically disadvantaged students in a school (Figure 2).

  2. Specifically, our model indicates that schools with ~ 80% of their students being classified as economically disadvantaged show a strong positive relationship between NWRI enrollment percentage and average FAST ELA score. This relationship begins to break down under this ~ 80% value.

Figure 2. Scatterplots showing the relationship between school average FAST ELA scores (Grades 3-5, PM2) and the percent of NWRI enrollment per school across Title I and Non-Title I (colored points). Different facets show the different percentages of economically disadvantaged students by school.

Descriptive Analyses

We were able to calculate school specific variables for 2254 schools across 71 districts. Certain variables were calculated using student data from grades K through 5 while others only used data from grades 3 through 5 (namely test scores). Variables to note are included in Table 1.

Table 1.

Variable Description Notes
NWRI enrollment percent Percent of students enrolled in NWRI across grades K through 5. Not all schools may include grades K or other grades.
NWRI eligible percent Percent of students not enrolled in NWRI but are eligible across grades K through 5.
Average FAST Reading Score The average FAST score calculated using scores from PM2 (2024-25) for grades 3 to 5. K-2 take a different exam and the ability to standardize scores between the two types is questionable
Percent Economically Disadvantaged Students The percent of students in the school with a lunch code status of “C,” “R,” “3,” “D,” “E,” “F,” or “4” on Survey 3 of the 2023-24 school year. For breakdown:
https://www.fldoe.org/core/fileparse.php/18617/urlt/1819-146025.pdf
Title 1 School Whether a school is a Title 1 school or not

In general we observe that schools with higher percentages of economically disadvantaged students are also schools with higher eligibility percentages (Figure 1). This general trend was observed across both Title I and Non-Title I schools. In general eligibility percentages also seemed to be higher in Title I schools (67.1%) versus Non-Title I schools (52.3%).

Schools that have a lower than average enrollment but higher than average eligibility become of significant interest (Figure 3 orange square). These schools are mostly comprised of Title I schools (Figure 3).

Figure 3. Scatterplots showing the relationship between NWRI eligibility percentage for grades K-5 (by school) against the NWRI enrollment percentage for grades K-5 (also by school) across Title I and Non-Title I schools (left and right). Vertical and horizontal black lines pinpoint the average eligibility and enrollment percentages across all schools. The orange square pinpoint schools that fall under the average enrollment percentage and over the average eligibility percentage.

Figure 4. Density plots showing where most of the schools are in the relationship between NWRI eligibility percentage for grades K-5 (by school) against the NWRI enrollment percentage for grades K-5 (also by school) across Title I and Non-Title I schools (left and right). This plot essentially shows the highest concentration of points (schools) within the plot, reflecting the points from Figure 2. Vertical and horizontal black lines pinpoint the average eligibility and enrollment percentages across all schools. The orange square pinpoint schools that fall under the average enrollment percentage and over the average eligibility percentage.

There are 700 schools that can be found in the orange square (Figure 3 & 4). Together they have 160762 students through grades K-5 who are eligible for NWRI. This represents 34% of all eligible students in grades K-5 across the schools in this analysis. Most of these schools are found in Miami-Dade County with 21450 eligible students across 108 schools. But the county with most students per school on average is St. Lucie County at 341 students. For more extensive data for each of these schools please use the interactive table below. Note: It is often easier to search for the county and then click it to view the schools and their statistics. You can sort the table by different columns as well.

Statistical Analyses

We’ve been mostly analyzing NWRI impacts on learning gains primarily at the student level where analyses are conducted on hundreds of thousands or rows of data with every row representing a student. While these analyses are useful, detecting impacts (e.g. improved scores etc.) can be difficult and hard to convey to non-technical stakeholders, policymakers, and internal team members. Additionally, data that we know are informative, such as socioeconomic data (e.g. household income), are not available at the student-level.

To work around this we aggregated student-level data (test scores, enrollment in NWRI) to the school-level and were able to combine other school-specific data (e.g. percent of economically disadvantaged students etc.) from the Florida Dept. of Education and the National Center of Education Statistics. More specifically, we calculated the average achievement level by students in grades 3 through 5 for the FAST Reading PM2 Test for the year 2024-25 for each school. We then calculated enrollment percentage of NWRI students per school.

Note: we removed schools with less than 100 students in grades 3 through 5 to minimize schools with smaller sample sizes. This left a sample size of 1837 schools.

We then fitted a multi-level model where our response variable was average FAST score as a function of: whether the schools was a Title I or non-Title I school, the percentage of students classified as economically disadvantaged, the total enrollment numbers of grades 3-5 (including all students), and the NWRI enrollment percentage. We included three competing models based on three lines of reasoning:

  1. The impact of enrollment percentage on the average FAST score is dependent on the number of students in grades 3-5.

  2. The impact of enrollment percentage on the average FAST score is dependent on the percentage of students classified as economically disadvantaged.

  3. There is no effect of any variables resulting in an intercept only model.

Additionally we specify that we also expect the impact of NWRI enrollment percentage to vary (random slope) based on locality type (suburban, urban etc.) of the school which are hierarchically nested within school district (nested random intercepts). Note: we did not add more complexity to the random effects of the model due to model convergence issues.

Fitting and evaluating models

Below we show the model fittings:

#here we begin model fitting with TMB with gaussian multilevel models
set.seed(123) #for reproduciblity
require(glmmTMB)
m1_size_enroll<-glmmTMB(data = analysis_data, 
            mean_normscorepm2 ~  Title.I  +
              scale(Percent.of.Economically.Disadvantaged.Students) +
              scale(enrollment_3_5) *
              scale(perc_enroll_3_5) +
              (1+scale(perc_enroll_3_5)|Dis/LOCALE))

m2_econ_enroll<-glmmTMB(data = analysis_data, 
            mean_normscorepm2 ~  Title.I +scale(enrollment_3_5) +
              scale(Percent.of.Economically.Disadvantaged.Students) *
              scale(perc_enroll_3_5) +
              (1+scale(perc_enroll_3_5)|Dis/LOCALE))

m3null<-glmmTMB(data = analysis_data, 
            mean_normscorepm2 ~  1 +
              (1+scale(perc_enroll_3_5)|Dis/LOCALE))

Based of AIC rankings we observed the most plausible model being one where the percentage of economically disadvantage students interacted with the NWRI enrollment percentage:

#AIC table
bbmle::AICtab(m1_size_enroll, m2_econ_enroll,m3null, weights = TRUE, sort =TRUE, delta =TRUE)
               dAIC   df weight
m2_econ_enroll    0.0 13 1     
m1_size_enroll   38.4 13 <0.001
m3null         1539.1 8  <0.001

Our top model also showed strong correlation coefficicents (below) where our fixed effects explained ~ 55% of the variation in the data and the inclusion of random effects provided an additional ~16% of explained variation (see below). Additionally, the model was free from any serious collinearity.

#R2
MuMIn::r.squaredGLMM(m2_econ_enroll)
           R2m       R2c
[1,] 0.5549452 0.6580455
#VIF calculation
performance::check_collinearity(
       m2_econ_enroll,
       component = c('all') # 'all' shows both conditional and zi components
)
# Check for Multicollinearity

Low Correlation

                                                                         Term
                                                                      Title.I
                                                        scale(enrollment_3_5)
                        scale(Percent.of.Economically.Disadvantaged.Students)
                                                       scale(perc_enroll_3_5)
 scale(Percent.of.Economically.Disadvantaged.Students):scale(perc_enroll_3_5)
  VIF   VIF 95% CI Increased SE Tolerance Tolerance 95% CI
 2.96 [2.75, 3.20]         1.72      0.34     [0.31, 0.36]
 1.13 [1.09, 1.21]         1.06      0.88     [0.83, 0.92]
 3.12 [2.89, 3.37]         1.77      0.32     [0.30, 0.35]
 1.02 [1.00, 1.19]         1.01      0.98     [0.84, 1.00]
 1.03 [1.01, 1.15]         1.02      0.97     [0.87, 0.99]

Based off the initial model summary we see a signficant interaction effect where on its own, the percent of economically disadvantaged students leads to lower FAST ELA scores for grades 3-5 BUT this effect is reversed when being moderated by the percent of NWRI enrollment. Additionally Title I schools have on average lower FAST ELA scores than non-Title I schools.

#model summary
sjPlot::tab_model(m2_econ_enroll)
  mean normscorepm 2
Predictors Estimates CI p
(Intercept) 3.16 3.11 – 3.20 <0.001
Title I [YES] -0.15 -0.19 – -0.11 <0.001
enrollment 3 5 0.03 0.01 – 0.04 <0.001
Percent of Economically
Disadvantaged Students		 -0.21 -0.24 – -0.19 <0.001
perc enroll 3 5 0.02 -0.00 – 0.04 0.050
Percent of Economically
Disadvantaged Students ×
perc enroll 3 5		 0.04 0.03 – 0.05 <0.001
Random Effects
σ2 0.05
τ00 LOCALE:Dis 0.00
τ00 Dis 0.01
τ11 LOCALE:Dis.scale(perc_enroll_3_5) 0.00
τ11 Dis.scale(perc_enroll_3_5) 0.00
ρ01 LOCALE:Dis -0.77
ρ01 Dis -0.14
ICC 0.23
N LOCALE 12
N Dis 68
Observations 1837
Marginal R2 / Conditional R2 0.555 / 0.658

Examining the interaction

A deeper dive into the interaction effect shows school FAST ELA scores increased with NWRI enrollment percentage when the schools also held a high proportion of economically disadvantaged students. This positive relationship breaks down in schools where ~80% or less of the student population is classified as economically disadvantaged.

# Generate effect of pct_enrolled_z at different values of economic disadvantage
pred <- ggpredict(m2_econ_enroll, 
                  terms = c("perc_enroll_3_5",
                            "Percent.of.Economically.Disadvantaged.Students [0,20,40,60,80,100]"))

pred_df <- data.frame(pred) %>%
  mutate(label_group = factor(paste0(group, " %"),
                              levels = paste0(c(0, 20, 40, 60, 80, 100), " %")))


ggplot(data = NULL) +
  geom_point(data = analysis_data,
             aes(x = perc_enroll_3_5, y = mean_normscorepm2),
             alpha = 0.3) +
  geom_line(data = pred_df,
            aes(x = x, y = predicted, color = label_group)) +
  geom_ribbon(data = pred_df,
              aes(x = x, ymin = conf.low, ymax = conf.high, fill = label_group),
              alpha = 0.3) +
  theme_bw() +
  labs(y = "FAST ELA Reading Score",
       x = "Percent enrolled in NWRI (%)",
       fill = "Percent\nDisadvantaged\nStudents",
       color = "Percent\nDisadvantaged\nStudents") +
  scale_color_viridis_d() +
  scale_fill_viridis_d() +
  theme( axis.title = element_text(size = 14),
    axis.text = element_text(size = 12))

Interestingly our model also showed that at ~40% of economically disadvantaged students, the relationship between NWRI enrollment and FAST ELA scores is negative. But it’s important to note that most of the data fed into this model comes from

analysis_data  %>%
  count(econ_bin) %>%
  ggplot(.) +
  geom_col(aes(x = econ_bin, y=n )) +
  theme_bw() +
  labs(x = "Percent Economically Disadvantaged Students",
       y = "No. of Schools",
       title = "Data used for the model") +
  theme(
    axis.title = element_text(size = 14),
    axis.text = element_text(size = 12),
     title = element_text(face = "bold", size =16)
  )