library(tidyverse)
library(tibble)
library(survival)
library(survminer)
library(rpart)
library(rpart.plot)
library(partykit)
library(mice)
library(ranger)
library(caret)Dealing with Missing Values: Imputation and Random Forest Modeling on Vermont School Data
The following work is an excerpt from an open-note exam for my Statistical Learning course in Spring 2025. This excerpt is meant to show the process of how I create and apply Statistical Learning models, prior to writing a formal report.
Model Creation
vt_schools <- read_csv("Vermont_Schools.csv")Rows: 177 Columns: 36
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): SupervisoryUnion
dbl (35): LEA, Year, EducationSpendingGrant, SchoolConstructionAid, SmallSch...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.md.pattern(vt_schools)
SupervisoryUnion LEA Year ELA05 ELA06 M6 EducationSpendingGrant
96 1 1 1 1 1 1 1
27 1 1 1 1 1 1 1
14 1 1 1 1 1 1 1
11 1 1 1 1 1 1 1
4 1 1 1 1 1 1 1
2 1 1 1 1 1 1 1
7 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
4 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
2 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
2 1 1 1 1 1 1 0
1 1 1 1 0 0 0 1
0 0 0 1 1 1 2
StateTransportationAid SpecialEducationGrants AllOtherStateGrants
96 1 1 1
27 1 1 1
14 1 1 1
11 1 1 1
4 1 1 1
2 1 1 1
7 1 1 1
1 1 1 1
4 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
2 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
2 0 0 0
1 1 1 1
2 2 2
TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue
96 1 1 1
27 1 1 1
14 1 1 1
11 1 1 1
4 1 1 1
2 1 1 1
7 1 1 1
1 1 1 1
4 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
2 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
2 0 0 0
1 1 1 1
2 2 2
PupilSupportServices InstructionalStaffSupportServ GeneralAdministration
96 1 1 1
27 1 1 1
14 1 1 1
11 1 1 1
4 1 1 1
2 1 1 1
7 1 1 1
1 1 1 1
4 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
2 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
2 0 0 0
1 1 1 1
2 2 2
SchoolAdministration StudentTransportation M5 OtherCategoricalGrants
96 1 1 1 1
27 1 1 1 1
14 1 1 1 1
11 1 1 1 1
4 1 1 1 1
2 1 1 1 1
7 1 1 1 1
1 1 1 1 1
4 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
2 1 1 1 1
1 1 1 1 1
1 1 1 1 0
1 1 1 0 1
2 0 0 1 0
1 1 1 0 1
2 2 2 3
FoodServiceOperations ELA03 ELA04 ELA07 ELA08 M3 M4 M7 M8
96 1 1 1 1 1 1 1 1 1
27 1 1 1 1 1 1 1 1 1
14 1 1 1 1 1 1 1 1 1
11 1 1 1 1 1 1 1 1 1
4 1 1 1 1 1 1 1 1 1
2 1 1 1 1 1 1 1 1 1
7 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1
4 1 1 1 1 1 1 1 1 1
1 1 1 1 0 0 1 1 0 0
1 1 1 1 0 0 1 1 0 0
1 1 1 0 1 1 0 0 1 1
2 1 0 0 1 1 0 0 1 1
1 0 1 1 1 1 1 1 1 1
1 1 1 1 0 0 1 1 0 0
1 1 1 1 1 1 1 1 1 0
2 0 1 1 1 1 1 1 1 1
1 1 0 0 0 0 0 0 0 0
3 3 4 4 4 4 4 4 5
StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA011 M11
96 1 1 1 1 1
27 1 1 1 1 1
14 1 1 1 1 1
11 1 1 1 0 0
4 1 1 1 0 0
2 1 1 1 0 0
7 0 0 0 1 1
1 0 0 0 1 1
4 0 0 0 1 1
1 1 1 1 0 0
1 0 0 0 0 0
1 1 1 1 1 1
2 1 1 1 1 1
1 1 1 1 1 1
1 0 0 0 0 0
1 1 1 1 1 1
2 0 0 0 1 1
1 0 0 0 0 0
17 17 17 21 21
SmallSchoolsGrant SchoolConstructionAid
96 1 1 0
27 1 0 1
14 0 0 2
11 1 1 2
4 1 0 3
2 0 0 4
7 1 1 3
1 1 0 4
4 0 0 5
1 1 1 6
1 1 1 9
1 1 0 4
2 1 1 4
1 1 0 2
1 0 0 12
1 0 0 4
2 0 0 19
1 0 0 19
25 59 244sum(is.na(vt_schools$M8))[1] 5sum(vt_schools$M8 == 0, na.rm = TRUE)[1] 1vt_schools$M8[vt_schools$M8 == 0] <- NAWow! That’s a lot of missing data. Unlike the last one, removing all rows with missing data would be (1) losing a lot of important information and (2) kind of a hassle. First, I tried using MICE to help me impute missing values.
imputed_vt_schools <- mice(vt_schools,
method = "rf",
seed = 1)
iter imp variable
1 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11Warning: Number of logged events: 1stripplot(imputed_vt_schools)
imputed_vt_schools$loggedEvents it im dep meth out
1 0 0 constant SupervisoryUnionBut, when I tried to run the code above, I found that the output had no points on the graphs and no values were actually imputed. I determined that there could be a few things going on. First, there could be too many variables for too few observations. There could also be some really highly correlated variables or variables that have very low variation or that are almost constant across all rows. First, I’ll figure out if there are variables that are almost constant. Then, I will figure out which values are most correlated to see if I can take out some variables (which would solve the remaining two possible problems)
num_data <- vt_schools[sapply(vt_schools, is.numeric)]
cor_matrix <- cor(num_data, use = "pairwise.complete.obs")findCorrelation(cor_matrix, cutoff = 0.90)[1] 11 3Columns 11 and 3 seem to be highly correlated with other columns in the data set, which are the year and total local revenue columns. I should probably take the year column out anyway because its range is relatively small, and it wouldn’t be super helpful to use it as a predictor variable. I don’t really want to lose Total Local Revenue, so I’ll leave it in for now. If I still have problems, I’ll try taking it out.
vt_schools_clean <- vt_schools |>
select(-c(Year, LEA))
imputed_vt_schools <- mice(vt_schools_clean,
method = "rf",
seed = 1)
iter imp variable
1 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11Warning: Number of logged events: 1When removing year and LEA, the strip plot still doesn’t show anything. My next thought was that I wanted to try to select just a few variables to use. I’m thinking it might be useful to split up the data set so that we can impute values based on test scores separately from school funding and enrollment data imputations. That way, I could see where the issue is happening, and reduce columns until I determine what’s causing the problem. Then, I can find a way to put the separated imputed values back together.
vt_schools_clean <- vt_schools_clean |> mutate(row_id = row_number())
vt_schools_clean_by_scores <- vt_schools_clean |>
select(-c(1:20) & -35)
vt_schools_clean_by_else <- vt_schools_clean |>
select(-1 & -c(21:32) & -c(34:35))
imputed_vt_schools_scores <- mice(vt_schools_clean_by_scores,
method = "rf",
seed = 1)
iter imp variable
1 1 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 2 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 3 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 4 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
1 5 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 1 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 2 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 3 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 4 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
2 5 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 1 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 2 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 3 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 4 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
3 5 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 1 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 2 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 3 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 4 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
4 5 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 1 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 2 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 3 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 4 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11
5 5 ELA03 ELA04 ELA05 ELA06 ELA07 ELA08 ELA011 M3 M4 M5 M6 M7 M8 M11stripplot(imputed_vt_schools_scores)
imputed_vt_schools_else <- mice(vt_schools_clean_by_else,
method = "rf",
seed = 1)
iter imp variable
1 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
1 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
1 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
1 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
1 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
2 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
2 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
2 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
2 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
2 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
3 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
3 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
3 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
3 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
3 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
4 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
4 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
4 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
4 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
4 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
5 1 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
5 2 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
5 3 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
5 4 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8
5 5 EducationSpendingGrant SchoolConstructionAid SmallSchoolsGrant StateTransportationAid SpecialEducationGrants OtherCategoricalGrants AllOtherStateGrants TotalLocalRevenue StateSourcesRevenue FederalLocalRevenue PupilSupportServices InstructionalStaffSupportServ GeneralAdministration SchoolAdministration StudentTransportation FoodServiceOperations StudentsFreeReducedLunch TotalEnrollment PercentLowIncome M8stripplot(imputed_vt_schools_else)
imputed_else <- complete(imputed_vt_schools_else) |>
mutate(row_id = vt_schools_clean$row_id)
imputed_scores <- complete(imputed_vt_schools_scores)|>
mutate(row_id = vt_schools_clean$row_id)
imputed_vt_schools <- full_join(imputed_else, imputed_scores, by = "row_id") |>
mutate(SupervisoryUnion = vt_schools_clean$SupervisoryUnion) |>
mutate(M8 = (M8.x + M8.y)/2) |>
select(-c("M8.x", "M8.y"))Since both of these worked, and lots of my other attempts and ideas (not included here) didn’t, I am going to stick with this technique. I don’t love it because I know that the scores didn’t use school spending data as predictors and that school spending didn’t use scores as predictors (though I’m more concerned with the former than the latter). This means that I might not be using the data to the fullest extent possible (i.e. having it give me the most amount of information.) But, for the sake of time on this exam, I am going to move forward with this method. And since I have a full data set now, I can create a random forest with all the variables and assess each variables’ importance.
rf_schools_data <- imputed_vt_schools |>
select(-SupervisoryUnion, -row_id)
rf_schools <- ranger(M8 ~ ., data = rf_schools_data, importance = "impurity")
rf_schoolsRanger result
Call:
ranger(M8 ~ ., data = rf_schools_data, importance = "impurity")
Type: Regression
Number of trees: 500
Sample size: 177
Number of independent variables: 32
Mtry: 5
Target node size: 5
Variable importance mode: impurity
Splitrule: variance
OOB prediction error (MSE): 459.7752
R squared (OOB): 0.6977815 sort(rf_schools$variable.importance) SchoolConstructionAid AllOtherStateGrants
146.8614 2180.9755
FederalLocalRevenue StudentsFreeReducedLunch
2304.1898 2332.6419
FoodServiceOperations SmallSchoolsGrant
2402.5371 2627.4604
TotalEnrollment OtherCategoricalGrants
2629.1374 2749.4538
InstructionalStaffSupportServ StateSourcesRevenue
2922.8177 2996.7684
GeneralAdministration StateTransportationAid
3213.9776 3259.6616
SchoolAdministration StudentTransportation
3275.9223 3300.2777
EducationSpendingGrant SpecialEducationGrants
3356.8327 3435.7111
TotalLocalRevenue PupilSupportServices
3689.9085 4409.5335
PercentLowIncome ELA011
5711.3693 6333.6172
ELA06 M3
7240.1920 8324.8894
ELA05 ELA03
9592.0715 9835.0847
ELA04 M4
10274.2076 10444.6694
M11 M6
12188.7584 12265.0188
M5 ELA07
16206.6857 21464.4877
M7 ELA08
32304.7972 46323.5729 It looks like among the top predictors are various test scores, as well as Percent Low Income, Pupil Support Services, Total Local Revenue, Special Education Grant, and Education Spending Grant.
target_col <- imputed_vt_schools[[35]]
compare_cols <- imputed_vt_schools[, 1:33]
correlations <- cor(target_col, compare_cols)
view(correlations)In order to best predict how a school’s students will do on eighth grade math scores, we can look at the correlation matrix of how math scores are correlated with each variable. Obviously, this shows us that there is a positive correlation between M8 scores and all other test scores; if other scores are high, M8 is likely to be high too. But funding has interesting correlations. Some things, like Education Spending, Special Education Grants, Pupil Support Services, Instructional Staff Support Services, or Student Transportation, have a positive correlation, meaning more money allocated to those departments is correlated with a better test score. However, other things, like Percent Low Income, General Administration, Other Categorical Grants, or School Construction Aid, show negative correlations– meaning the more money allocated to those causes is correlated with worse test scores.
Actionable Steps
schools_decision_tree <- rpart(M8 ~ GeneralAdministration, data = rf_schools_data)
rpart.plot(schools_decision_tree)
The first piece of advice I would offer is that the funding for the General Admissions should be less than at least $1,100,000.
schools_decision_tree2 <- rpart(M8 ~ EducationSpendingGrant, data = rf_schools_data)
rpart.plot(schools_decision_tree2)
However, this tree shows that schools with a higher education spending grant tend to get higher scores. I would recommend to policymakers that the education spending grant be higher than $14,000,000 with less than $1,100,000 of that grant being allocated to General Administration. This way, more money can be used to fund Student Transportation, Student Support Services, Instructional Staff Support Services, Special Education, and more, which all show up as significant alternatives for predicting M8 scores on my second random forest model; and shown by my trees below, when using these variables to predict M8 scores, higher allocations to these services indicate higher scores.
schools_decision_tree3 <- rpart(M8 ~ StudentTransportation, data = rf_schools_data)
rpart.plot(schools_decision_tree3)
schools_decision_tree4 <- rpart(M8 ~ PupilSupportServices, data = rf_schools_data)
rpart.plot(schools_decision_tree4)
schools_decision_tree5 <- rpart(M8 ~ InstructionalStaffSupportServ, data = rf_schools_data)
rpart.plot(schools_decision_tree5)
schools_decision_tree6 <- rpart(M8 ~ SpecialEducationGrants, data = rf_schools_data)
rpart.plot(schools_decision_tree6)
Overall, the best piece of advice I can give is to increase funding to schools that have budgets below $14,000,000, while making sure that these schools only allocate around $1,100,000 to their General Administration needs and increasing funding in significant areas discussed above and other areas of need.