── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(sandwich)library(lmtest)
Loading required package: zoo
Attaching package: 'zoo'
The following objects are masked from 'package:base':
as.Date, as.Date.numeric
library(psych)
Attaching package: 'psych'
The following objects are masked from 'package:ggplot2':
%+%, alpha
library(purrr)library(tibble)library(stargazer)
Please cite as:
Hlavac, Marek (2022). stargazer: Well-Formatted Regression and Summary Statistics Tables.
R package version 5.2.3. https://CRAN.R-project.org/package=stargazer
Import Kent data.
kent =read_csv("vt_kent_all_students.csv")
Rows: 5634 Columns: 45
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (4): school, school_grade, race, race_recoded
dbl (41): student_id, school_num, grade, grade6, grade7, grade8, male, sped,...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 1279 Columns: 46
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (4): school, school_grade, race, race_recoded
dbl (42): student_id, school_num, grade, grade3, grade4, grade5, grade6, mal...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Inspect similar categories for cleaning pooled variables later (after appending).
table(penn$race)
AMERICAN INDIAN/ALASKAN ASIAN
22 31
BLACK/AFRICAN AMERICAN HISPANIC
1101 43
MULTI RACIAL - NON HISPANIC NATIVE HAWAIIAN
1 8
WHITE/CAUCASIAN
73
table(kent$race)
American Indian or Alaska Native
21
Asian
1365
Black or African American
760
Hispanic/Latino
1312
Native Hawaiian or Other Pacific Islander
189
Two or More Races
512
White
1475
table(cumber$race)
A B H I Multi P W
181 4069 1465 100 816 46 1962
Clean IEP
Inspect Kent’s Special Ed (sped) variable.
table(kent$sped)
0 1
4803 831
Decision: Treat Kent’s SPED variable as an IEP to align with Cumberland and Penn.
kent$iep = kent$spedtable(kent$iep)
0 1
4803 831
Clean ELL
Inspect Kent’s MLL variable.
table(kent$mll)
0 1
4023 1611
Decision: Treat Kent’s MLL variable as ELL to align with Cumberland and Penn.
kent$ell= kent$mlltable(kent$ell)
0 1
4023 1611
Clean FRPL
Free/ reduced price lunch as proxy for low income status. FRPL for Penn, Low-income for Kent, but FRPL missing for Cumberland.
Inspect frpl for Penn.
table(penn$frpl)
0 1
353 926
Decision: Treat Penn’s FRPL as low income variable to align with Kent.
penn$low_inc = penn$frpltable(penn$low_inc)
0 1
353 926
Decision: Treat Cumberland’s FRPL as low income variable to align with Kent. Note, since FRPL data was all missing, Cumberland will not have low-income status available and thus will be excluded from this subgroup analysis.
table(cumber$frpl)
< table of extent 0 >
cumber$low_inc = cumber$frpltable(cumber$low_inc)
< table of extent 0 >
Clean At-risk
Inspect Kent’s variable for low academic performance
table(kent$low_ap)
0 1
4157 1477
Decision: Treat Kent’s variable for low academic performance as at-risk variable to align with Cumberland. Since nothing similar is present fr Penn, create at-risk variable for Penn where all entries at NA. Note, since at-risk data was not present, Penn will not have at-risk status available and thus will be excluded from this subgroup analysis.
Use rbind since columns matched by name and by order.
pooled =rbind( penn_to_pool, kent_to_pool, cumber_to_pool)
Finish pooled race
table(pooled$race)
A
181
American Indian or Alaska Native
21
AMERICAN INDIAN/ALASKAN
22
Asian
1365
ASIAN
31
B
4069
Black or African American
760
BLACK/AFRICAN AMERICAN
1101
H
1465
HISPANIC
43
Hispanic/Latino
1312
I
100
Multi
816
MULTI RACIAL - NON HISPANIC
1
NATIVE HAWAIIAN
8
Native Hawaiian or Other Pacific Islander
189
P
46
Two or More Races
512
W
1962
White
1475
WHITE/CAUCASIAN
73
Decision: Combine categories where they appears across all districts: White, Black, Hispanic, Asian, American Indian/Native, Multi-racial, Pacific Islander.
pooled = pooled %>%mutate(race_pooled =case_when(race %in%c("WHITE/CAUCASIAN", "White", "W") ~"White",race %in%c("BLACK/AFRICAN AMERICAN","Black or African American", "B") ~"Black",race %in%c("ASIAN", "Asian", "A") ~"Asian",race %in%c("HISPANIC", "Hispanic/Latino","H") ~"Hispanic",race %in%c("MULTI RACIAL - NON HISPANIC","Two or More Races", "Multi") ~"Multi-racial",race %in%c("AMERICAN INDIAN/ALASKAN", "American Indian or Alaska Native", "I") ~"Native American",#for American Indianrace %in%c("NATIVE HAWAIIAN", "Native Hawaiian or Other Pacific Islander", "P") ~"Pacific Islander",TRUE~NA_character_))table(pooled$race_pooled)
Asian Black Hispanic Multi-racial
1577 5930 2820 1329
Native American Pacific Islander White
143 243 3510
###Create race dummy
Collapse race_pooled so that Multi-racial, Native American, Pacific Islander, and Asian fall under Other Races.
pooled = pooled %>%mutate(race_pooled =recode(race_pooled,"Multi-racial"="Other Races","Native American"="Other Races","Pacific Islander"="Other Races","Asian"="Other Races" ) )table(pooled$race_pooled)
Black Hispanic Other Races White
5930 2820 3292 3510
Create new categorical variable for grade by school
pooled = pooled %>%mutate(district_grade =paste(sub(" .*", "", district), grade))table(pooled$district_grade)
Cumberland 6 Cumberland 7 Cumberland 8 Kent 6 Kent 7 Kent 8
2985 2892 2762 1876 1883 1875
Penn 3 Penn 4 Penn 5 Penn 6
304 315 314 346
Create school_treat
“school_treat” will equal 1 if the school is a Live+AI school, and 0 will represent a comparison school.
treatment_schools <-c("Spring Lake Middle","Luther Nick Jeralds Middle","Aldan Elementary School","Bell Avenue Elementary School","Colwyn Elementary School","Walnut Street Elementary School","Meridian Middle")pooled$school_treat =ifelse(pooled$school %in% treatment_schools, 1, 0)table(pooled$school_treat) #provides enrollment between treat and comp schools
0 1
13508 2044
Construct treat_itt (based on Live+AI offering)
Variable vt_present indicates whether a student has a VT Account ID (i.e., was offered Live+AI tutoring)
table(pooled$vt_present)
0 1
14358 1194
Double check coding on vt_present. Ensure all students with VT Account IDs were only at treatment school sites.
pooled %>%filter(vt_present ==1) %>%count(school_treat, name ="n") %>%mutate(pct =round(100* n /sum(n), 1) )
# A tibble: 1 × 3
school_treat n pct
<dbl> <int> <dbl>
1 1 1194 100
Filter out “treatment” students at treatment schools who were not offered Live+AI. They are not part of the final sample group and should not be used as comparison student under RQ1. Among students where vt_present = 1, exclude those where num_sessions_week is less than once per week.
pooled_treat_comp = pooled %>%filter(#keep students at comp schools school_treat ==0|#keep students at treat schools IF (school_treat ==1&# VT ID is present aka offered Live+AI vt_present ==1) )
Create treat: Equals 1 if students had more than 2 sessions. All else equals 0.
Based treat_itt grouping based on vt_present. Equal 1 if student was offered Live+AI, i.e., intended to be treated (regardless of usage).
Variable “treat” will equal 1 if a student offered Live+AI meets the threshold of at least 1 session per week.
Filter out “treatment” students who do not meet usage threshold as well as students at treatment schools who were not offered Live+AI. They are not part of the final dosage sample group and should not be used as comparison student under RQ4. Among students where vt_present = 1, exclude those where num_sessions_week is less than once per week.
pooled_treat_comp_usage = pooled_treat_comp %>%filter(#keep students at comp schools school_treat ==0|#keep students at treat schools IF (school_treat ==1&# meet usage threshold & VT ID is present num_sessions_week >=1& vt_present ==1) )
Create treat: Equals 1 if students had at least one session logged per week. All else equals 0. Kent = 87, Cumberland = 350, Kent = 99, Total = 536.
Save e2i file with only containing matched students from the merge with complete test info for ELA
table(pooled_treat_comp_usage$both_ela)
0 1
1867 12177
#subset: keep only students with complete test infopooled_treat_comp_usage_ela = pooled_treat_comp_usage %>%filter(both_ela ==1)print(pooled_treat_comp_usage_ela)
“vt_present” will equal 1 if a student was offered Live+AI, as shown by the presence of a VT account ID.
Create pooled data frame containing only students with VT Account IDs. This will also include students from treatment schools with no usage or usage falling below the threshold (for RQ2 later).
pooled_vt = pooled %>%filter(vt_present ==1)print(pooled_vt)
Rows: 2180 Columns: 54
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (6): district_grade, school, district, school_grade, race, race_pooled
dbl (48): treat_itt, iep, moy_math_z, distance, weights, subclass, student_i...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
# A tibble: 16 × 10
Characteristic Treatment Comparison SMD Abs_SMD Median_Treat Median_Comp
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 MOY Math z-score -0.199 -0.204 0.005 0.005 -0.23 -0.23
2 Black 0.685 0.619 0.139 0.139 NA NA
3 Hispanic 0.141 0.15 -0.023 0.023 NA NA
4 White 0.076 0.128 -0.17 0.17 NA NA
5 Other Race 0.097 0.104 -0.021 0.021 NA NA
6 FRPL 0.705 0.659 0.1 0.1 NA NA
7 IEP 0.123 0.122 0.003 0.003 NA NA
8 ELL 0.061 0.11 -0.174 0.174 NA NA
9 Male 0.484 0.507 -0.046 0.046 NA NA
10 Female 0.516 0.493 0.046 0.046 NA NA
11 Grade 3 0.077 0.077 0 0 NA NA
12 Grade 4 0.09 0.09 0 0 NA NA
13 Grade 5 0.061 0.061 0 0 NA NA
14 Grade 6 0.35 0.35 0 0 NA NA
15 Grade 7 0.212 0.212 0 0 NA NA
16 Grade 8 0.209 0.209 0 0 NA NA
# ℹ 3 more variables: SD_Treat <dbl>, SD_Comp <dbl>, WWC_Equivalence <chr>
# A tibble: 7 × 2
school n_students
<chr> <int>
1 Luther Nick Jeralds Middle 343
2 Spring Lake Middle 316
3 Bell Avenue Elementary School 115
4 Aldan Elementary School 83
5 Meridian Middle 83
6 Walnut Street Elementary School 80
7 Colwyn Elementary School 70
Rows: 2060 Columns: 55
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (6): district_grade, school, district, school_grade, race, race_pooled
dbl (49): treat_itt, iep, moy_ela_z, distance, weights, subclass, student_id...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
# A tibble: 16 × 10
Characteristic Treatment Comparison SMD Abs_SMD Median_Treat Median_Comp
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 MOY ela z-score -0.162 -0.157 -0.006 0.006 -0.1 -0.11
2 Black 0.688 0.618 0.147 0.147 NA NA
3 Hispanic 0.134 0.15 -0.045 0.045 NA NA
4 White 0.078 0.126 -0.161 0.161 NA NA
5 Other Race 0.1 0.106 -0.019 0.019 NA NA
6 FRPL 0.705 0.696 0.02 0.02 NA NA
7 IEP 0.127 0.122 0.015 0.015 NA NA
8 ELL 0.061 0.103 -0.152 0.152 NA NA
9 Male 0.475 0.497 -0.045 0.045 NA NA
10 Female 0.525 0.503 0.045 0.045 NA NA
11 Grade 3 0.082 0.082 0 0 NA NA
12 Grade 4 0.094 0.094 0 0 NA NA
13 Grade 5 0.065 0.065 0 0 NA NA
14 Grade 6 0.368 0.368 0 0 NA NA
15 Grade 7 0.22 0.22 0 0 NA NA
16 Grade 8 0.171 0.171 0 0 NA NA
# ℹ 3 more variables: SD_Treat <dbl>, SD_Comp <dbl>, WWC_Equivalence <chr>
# A tibble: 7 × 2
school n_students
<chr> <int>
1 Luther Nick Jeralds Middle 341
2 Spring Lake Middle 255
3 Bell Avenue Elementary School 114
4 Meridian Middle 87
5 Aldan Elementary School 82
6 Walnut Street Elementary School 81
7 Colwyn Elementary School 70
Research Question 1
What are the demographic and academic characteristics of students and schools in LIVE+ AI schools and non-LIVE+ AI schools? We will summarize the demographic and academic characteristics of students and schools in the treatment and comparison groups by reporting descriptive statistics (mean, median, SDs) across both groups. In accordance with What Works Clearinghouse v5.0 standards, we will also report standardized mean differences between the treatment and comparison schools.
Function for data summary
#create function to streamline calculationscalc_pct =function(x) {prop.table(table(x)) *100}
School level characteristics
Characteristics at the school level include MOY math and ELA scores, enrollment, and the percentage of students who are Black, Hispanic, White, FRPL-eligible, have IEPs, and are female. Note, each school is either a treatment or comparison school; this data frame does not contain students in comparison schools with unexpected usage (clean groupings by school).
Using pooled data dataframe, which includes students at treatment schools who were not offered Live+AI, students from treatment school who were offered Live+AI, and all students from comparison schools.
# A tibble: 11 × 16
characteristic trt_n trt_mean trt_median trt_sd cmp_n cmp_mean cmp_median
<chr> <int> <dbl> <dbl> <dbl> <int> <dbl> <dbl>
1 moy_math_z 1905 -0.205 -0.17 1.11 12322 -0.108 -0.06
2 moy_ela_z 1859 -0.198 -0.06 1.04 12360 -0.139 0.02
3 Black NA NA NA NA NA NA NA
4 Hispanic NA NA NA NA NA NA NA
5 White NA NA NA NA NA NA NA
6 Other race NA NA NA NA NA NA NA
7 FRPL eligible NA NA NA NA NA NA NA
8 IEP NA NA NA NA NA NA NA
9 ELL NA NA NA NA NA NA NA
10 Male NA NA NA NA NA NA NA
11 Female NA NA NA NA NA NA NA
# ℹ 8 more variables: cmp_sd <dbl>, smd <dbl>, trt_count <int>, trt_prop <dbl>,
# cmp_count <int>, cmp_prop <dbl>, abs_smd <dbl>, wwc_equivalence <chr>
Characteristics at the student level include MOY math and ELA scores and the percentage of students who are Black, Hispanic, White, FRPL-eligible, have IEPs and are female.
# A tibble: 1 × 1
avg_sessions_per_week
<dbl>
1 1.60
Data frame for reference For regressions, only work with those students who had VT Account IDs (those who were offered Live+AI). Pull from data frame where all students were included, regardless of meeting usage threshold.
# Full modelq2_model_pooled_by_school =lm( #before clustering by school num_sessions_week ~factor(school), data = pooled_vt)summary(q2_model_pooled_by_school)
Call:
lm(formula = num_sessions_week ~ factor(school), data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-3.6435 -0.8018 -0.3831 0.2982 17.3861
Coefficients:
Estimate Std. Error t value
(Intercept) 0.7288 0.2016 3.614
factor(school)Bell Avenue Elementary School 0.2607 0.2659 0.980
factor(school)Colwyn Elementary School 0.4091 0.2989 1.369
factor(school)Luther Nick Jeralds Middle 0.1730 0.2224 0.778
factor(school)Meridian Middle 3.4147 0.2827 12.078
factor(school)Spring Lake Middle 1.5774 0.2241 7.038
factor(school)Walnut Street Elementary School -0.3167 0.2878 -1.101
Pr(>|t|)
(Intercept) 0.000314 ***
factor(school)Bell Avenue Elementary School 0.327118
factor(school)Colwyn Elementary School 0.171344
factor(school)Luther Nick Jeralds Middle 0.436865
factor(school)Meridian Middle < 2e-16 ***
factor(school)Spring Lake Middle 3.29e-12 ***
factor(school)Walnut Street Elementary School 0.271254
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.859 on 1187 degrees of freedom
Multiple R-squared: 0.2226, Adjusted R-squared: 0.2187
F-statistic: 56.65 on 6 and 1187 DF, p-value: < 2.2e-16
Reference group is Aldan Elementary School. Results: Aldan had an average a 0.73 sessions/week (p<0.001). Bell, Colwyn, and Luther observed slightly higher usage than Aldan but it was not significant. Meridian observed 3.41 sessions/week more than Aldan (p<0.001). Spring Lake observed 1.60 sessions/week more than Aldan (p<0.001). Walnut Street students logged 0.31 sessions/week less than Aldan but it was not signifcantly different.
# A tibble: 7 × 2
school mean_num_sessions_week
<chr> <dbl>
1 Aldan Elementary School 0.729
2 Bell Avenue Elementary School 0.989
3 Colwyn Elementary School 1.14
4 Luther Nick Jeralds Middle 0.902
5 Meridian Middle 4.14
6 Spring Lake Middle 2.31
7 Walnut Street Elementary School 0.412
Pooled - Regress sessions per week ~ District
# Full modelq2_model_pooled_by_district =lm( #before clustering by school num_sessions_week ~factor(district), data = pooled_vt)summary(q2_model_pooled_by_district)
Call:
lm(formula = num_sessions_week ~ factor(district), data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-3.6435 -1.0366 -0.5593 0.3355 18.1172
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.57508 0.07082 22.242 < 2e-16 ***
factor(district)Kent 2.56838 0.21892 11.732 < 2e-16 ***
factor(district)Penn -0.75266 0.12535 -6.004 2.55e-09 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.943 on 1191 degrees of freedom
Multiple R-squared: 0.1477, Adjusted R-squared: 0.1462
F-statistic: 103.2 on 2 and 1191 DF, p-value: < 2.2e-16
#####################################Cluster-robust standard errors by schoolq2_pooled_district_clustered =vcovCL( q2_model_pooled_by_district,cluster =~ school_num)coeftest( q2_model_pooled_by_district,vcov = q2_pooled_district_clustered #after clustering by school)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.57508 0.53586 2.9394 0.003352 **
factor(district)Kent 2.56838 0.53586 4.7930 1.85e-06 ***
factor(district)Penn -0.75266 0.55365 -1.3595 0.174259
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Cumberland as reference group. Results, after school clustering: Average student at Cumberland has 1.57 sessions/week logged (p<0.01), Penn had 0.75 sessions/week less than Cumberland (not significant) and Kent had 2.56 session/week more than Cumberland (p<0.001).
# A tibble: 3 × 2
district mean_num_sessions_week
<chr> <dbl>
1 Cumberland 1.58
2 Kent 4.14
3 Penn 0.822
Pooled - Regress sessions per week ~ Male
# Full modelq2_model_pooled_by_male =lm( num_sessions_week ~ male, data = pooled_vt)summary(q2_model_pooled_by_male) #before clustering by school
Call:
lm(formula = num_sessions_week ~ male, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-1.5675 -1.2180 -0.7812 0.4233 18.1743
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.51804 0.08456 17.952 <2e-16 ***
male 0.04945 0.12185 0.406 0.685
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.104 on 1192 degrees of freedom
Multiple R-squared: 0.0001381, Adjusted R-squared: -0.0007007
F-statistic: 0.1647 on 1 and 1192 DF, p-value: 0.6849
#####################################Cluster-robust standard errors by schoolq2_pooled_male_clustered =vcovCL( q2_model_pooled_by_male,cluster =~ school_num)coeftest( q2_model_pooled_by_male,vcov = q2_pooled_male_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.518044 0.404230 3.7554 0.0001814 ***
male 0.049451 0.041079 1.2038 0.2289055
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Female as reference group. Results, after school clustering: Average female has 1.51 sessions/week logged (p<0.001); males had 0.05 sessions/week (not significant).
# A tibble: 2 × 2
male mean_num_sessions_week
<dbl> <dbl>
1 0 1.52
2 1 1.57
Pooled - Regress sessions per week ~ Race
# Set White as the reference grouppooled_vt$race_pooled =relevel(factor(pooled_vt$race_pooled),ref ="White")# Full modelq2_model_pooled_by_race =lm( num_sessions_week ~ race_pooled, data = pooled_vt)summary(q2_model_pooled_by_race) #before clustering by school
Call:
lm(formula = num_sessions_week ~ race_pooled, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-2.2383 -1.1371 -0.6108 0.4419 18.3973
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.6933 0.2170 7.803 1.32e-14 ***
race_pooledBlack -0.3983 0.2287 -1.741 0.0819 .
race_pooledHispanic 0.4889 0.2697 1.813 0.0701 .
race_pooledOther Races 0.5450 0.2893 1.883 0.0599 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.07 on 1190 degrees of freedom
Multiple R-squared: 0.03359, Adjusted R-squared: 0.03115
F-statistic: 13.79 on 3 and 1190 DF, p-value: 7.683e-09
#####################################Cluster-robust standard errors by schoolq2_pooled_race_clustered =vcovCL( q2_model_pooled_by_race,cluster =~ school_num)coeftest( q2_model_pooled_by_race,vcov = q2_pooled_race_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.69329 0.45620 3.7117 0.0002154 ***
race_pooledBlack -0.39831 0.19004 -2.0960 0.0362954 *
race_pooledHispanic 0.48890 0.29531 1.6556 0.0980750 .
race_pooledOther Races 0.54496 0.50049 1.0888 0.2764441
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Reference group is White. Results after school clustering: Average White student had 1.69 sessions/week (p<0.001). Asian students used 2.38 sessions more per week (p<0.001) compared to White students. Black students used 0.39 sessions less per week (p<0.05) compared to White students. Hispanic students used 0.49 sessions more per week than White students (p<0.1). All other groups’ differences in usage was not significantly difference than Whites.
# A tibble: 4 × 2
race_pooled mean_num_sessions_week
<fct> <dbl>
1 White 1.69
2 Black 1.29
3 Hispanic 2.18
4 Other Races 2.24
Pooled - Regress sessions per week ~ IEP
# Full modelq2_model_pooled_by_iep =lm( num_sessions_week ~ iep, data = pooled_vt)summary(q2_model_pooled_by_iep) #before clustering by school
Call:
lm(formula = num_sessions_week ~ iep, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-1.5812 -1.1966 -0.7790 0.4188 18.4418
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.58119 0.06478 24.41 <2e-16 ***
iep -0.33069 0.18786 -1.76 0.0786 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.101 on 1192 degrees of freedom
Multiple R-squared: 0.002593, Adjusted R-squared: 0.001756
F-statistic: 3.099 on 1 and 1192 DF, p-value: 0.07861
#####################################Cluster-robust standard errors by schoolq2_pooled_iep_clustered =vcovCL( q2_model_pooled_by_iep,cluster =~ school_num)coeftest( q2_model_pooled_by_iep,vcov = q2_pooled_iep_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.58119 0.42502 3.7203 0.0002083 ***
iep -0.33069 0.22998 -1.4379 0.1507233
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Reference group is students without IEP. Results after school clustering, Students without IEP averaged 1.58 sessions/week (p<0.001). Compared to students without IEPs, students with IEPs had 0.33 sessions less per week (not significant).
# Full modelq2_model_pooled_by_ell =lm( num_sessions_week ~ ell, data = pooled_vt)summary(q2_model_pooled_by_ell) #before clustering by school
Call:
lm(formula = num_sessions_week ~ ell, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-3.0966 -1.1372 -0.7372 0.4235 18.2551
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.43724 0.06154 23.354 < 2e-16 ***
ell 1.75939 0.25238 6.971 5.19e-12 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.062 on 1192 degrees of freedom
Multiple R-squared: 0.03917, Adjusted R-squared: 0.03837
F-statistic: 48.6 on 1 and 1192 DF, p-value: 5.188e-12
#####################################Cluster-robust standard errors by schoolq2_pooled_ell_clustered =vcovCL( q2_model_pooled_by_ell,cluster =~ school_num)coeftest( q2_model_pooled_by_ell,vcov = q2_pooled_ell_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.43724 0.37065 3.8776 0.0001112 ***
ell 1.75939 0.47191 3.7283 0.0002018 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Reference group are student who are not ELLs. Results after clustering: Student who are not ELL have 1.44 session per week (p<0.001). ELLs logged 1.76 sessions more per week than non ELLs (p<0.001).
# Full modelq2_model_pooled_by_low_inc =lm( num_sessions_week ~ low_inc, data = pooled_vt)summary(q2_model_pooled_by_low_inc) #before clustering by school
Call:
lm(formula = num_sessions_week ~ low_inc, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-2.0000 -1.1551 -0.7341 0.6344 14.2500
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.0000 0.1716 11.656 < 2e-16 ***
low_inc -0.7397 0.2057 -3.597 0.000359 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.986 on 439 degrees of freedom
(753 observations deleted due to missingness)
Multiple R-squared: 0.02862, Adjusted R-squared: 0.02641
F-statistic: 12.94 on 1 and 439 DF, p-value: 0.0003591
#####################################Cluster-robust standard errors by schoolq2_pooled_low_inc_clustered =vcovCL( q2_model_pooled_by_low_inc,cluster =~ school_num)coeftest( q2_model_pooled_by_low_inc,vcov = q2_pooled_low_inc_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.00005 0.88805 2.2522 0.02480 *
low_inc -0.73968 0.38053 -1.9438 0.05255 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Reference group is students who are not low income. Results after clustering: Students who are not low income logged 2 sessions per week (p>0.05). Students who are low income logged 0.74 sessions less per week compared to students who are not low income (p<0.1).
# A tibble: 3 × 2
low_inc mean_num_sessions_week
<dbl> <dbl>
1 0 2.00
2 1 1.26
3 NA 1.58
Pooled - Regress sessions per week ~ At-Risk
# Full modelq2_model_pooled_by_at_risk =lm( num_sessions_week ~ at_risk, data = pooled_vt)summary(q2_model_pooled_by_at_risk) #before clustering by school
Call:
lm(formula = num_sessions_week ~ at_risk, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-1.9539 -1.4539 -0.7728 0.5272 17.9195
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.7728 0.1030 17.217 <2e-16 ***
at_risk 0.1811 0.1644 1.102 0.271
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.328 on 839 degrees of freedom
(353 observations deleted due to missingness)
Multiple R-squared: 0.001444, Adjusted R-squared: 0.000254
F-statistic: 1.213 on 1 and 839 DF, p-value: 0.271
#####################################Cluster-robust standard errors by schoolq2_pooled_at_risk_clustered =vcovCL( q2_model_pooled_by_at_risk,cluster =~ school_num)coeftest( q2_model_pooled_by_at_risk,vcov = q2_pooled_at_risk_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.77278 0.71607 2.4757 0.01349 *
at_risk 0.18107 0.97133 0.1864 0.85217
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Reference group are student who are not academically at-risk. Results after clustering: students who are not at-risk logged 1.77 sessions/week (p<0.05); compared to at-risk students who logged 0.18 sessions more per week (not significant).
Self QA - Passed Note: Penn had no at-risk status data and was excluded from analysis.
Tukey multiple comparisons of means
95% family-wise confidence level
Fit: aov(formula = num_sessions_week ~ school, data = pooled_vt)
$school
diff
Bell Avenue Elementary School-Aldan Elementary School 0.26068111
Colwyn Elementary School-Aldan Elementary School 0.40908734
Luther Nick Jeralds Middle-Aldan Elementary School 0.17299314
Meridian Middle-Aldan Elementary School 3.41467334
Spring Lake Middle-Aldan Elementary School 1.57740815
Walnut Street Elementary School-Aldan Elementary School -0.31672582
Colwyn Elementary School-Bell Avenue Elementary School 0.14840623
Luther Nick Jeralds Middle-Bell Avenue Elementary School -0.08768797
Meridian Middle-Bell Avenue Elementary School 3.15399222
Spring Lake Middle-Bell Avenue Elementary School 1.31672704
Walnut Street Elementary School-Bell Avenue Elementary School -0.57740693
Luther Nick Jeralds Middle-Colwyn Elementary School -0.23609420
Meridian Middle-Colwyn Elementary School 3.00558600
Spring Lake Middle-Colwyn Elementary School 1.16832081
Walnut Street Elementary School-Colwyn Elementary School -0.72581316
Meridian Middle-Luther Nick Jeralds Middle 3.24168019
Spring Lake Middle-Luther Nick Jeralds Middle 1.40441501
Walnut Street Elementary School-Luther Nick Jeralds Middle -0.48971896
Spring Lake Middle-Meridian Middle -1.83726518
Walnut Street Elementary School-Meridian Middle -3.73139916
Walnut Street Elementary School-Spring Lake Middle -1.89413397
lwr
Bell Avenue Elementary School-Aldan Elementary School -0.5246530
Colwyn Elementary School-Aldan Elementary School -0.4736297
Luther Nick Jeralds Middle-Aldan Elementary School -0.4839148
Meridian Middle-Aldan Elementary School 2.5797048
Spring Lake Middle-Aldan Elementary School 0.9154932
Walnut Street Elementary School-Aldan Elementary School -1.1665708
Colwyn Elementary School-Bell Avenue Elementary School -0.6802535
Luther Nick Jeralds Middle-Bell Avenue Elementary School -0.6699385
Meridian Middle-Bell Avenue Elementary School 2.3763934
Spring Lake Middle-Bell Avenue Elementary School 0.7288333
Walnut Street Elementary School-Bell Avenue Elementary School -1.3709584
Luther Nick Jeralds Middle-Colwyn Elementary School -0.9442293
Meridian Middle-Colwyn Elementary School 2.1297437
Spring Lake Middle-Colwyn Elementary School 0.4555384
Walnut Street Elementary School-Colwyn Elementary School -1.6158489
Meridian Middle-Luther Nick Jeralds Middle 2.5940395
Spring Lake Middle-Luther Nick Jeralds Middle 1.0039185
Walnut Street Elementary School-Luther Nick Jeralds Middle -1.1564291
Spring Lake Middle-Meridian Middle -2.4899839
Walnut Street Elementary School-Meridian Middle -4.5741012
Walnut Street Elementary School-Spring Lake Middle -2.5657780
upr
Bell Avenue Elementary School-Aldan Elementary School 1.0460152
Colwyn Elementary School-Aldan Elementary School 1.2918044
Luther Nick Jeralds Middle-Aldan Elementary School 0.8299011
Meridian Middle-Aldan Elementary School 4.2496419
Spring Lake Middle-Aldan Elementary School 2.2393231
Walnut Street Elementary School-Aldan Elementary School 0.5331191
Colwyn Elementary School-Bell Avenue Elementary School 0.9770659
Luther Nick Jeralds Middle-Bell Avenue Elementary School 0.4945625
Meridian Middle-Bell Avenue Elementary School 3.9315911
Spring Lake Middle-Bell Avenue Elementary School 1.9046207
Walnut Street Elementary School-Bell Avenue Elementary School 0.2161446
Luther Nick Jeralds Middle-Colwyn Elementary School 0.4720409
Meridian Middle-Colwyn Elementary School 3.8814283
Spring Lake Middle-Colwyn Elementary School 1.8811032
Walnut Street Elementary School-Colwyn Elementary School 0.1642226
Meridian Middle-Luther Nick Jeralds Middle 3.8893208
Spring Lake Middle-Luther Nick Jeralds Middle 1.8049115
Walnut Street Elementary School-Luther Nick Jeralds Middle 0.1769912
Spring Lake Middle-Meridian Middle -1.1845464
Walnut Street Elementary School-Meridian Middle -2.8886971
Walnut Street Elementary School-Spring Lake Middle -1.2224899
p adj
Bell Avenue Elementary School-Aldan Elementary School 0.9582422
Colwyn Elementary School-Aldan Elementary School 0.8184563
Luther Nick Jeralds Middle-Aldan Elementary School 0.9870536
Meridian Middle-Aldan Elementary School 0.0000000
Spring Lake Middle-Aldan Elementary School 0.0000000
Walnut Street Elementary School-Aldan Elementary School 0.9279905
Colwyn Elementary School-Bell Avenue Elementary School 0.9984299
Luther Nick Jeralds Middle-Bell Avenue Elementary School 0.9994151
Meridian Middle-Bell Avenue Elementary School 0.0000000
Spring Lake Middle-Bell Avenue Elementary School 0.0000000
Walnut Street Elementary School-Bell Avenue Elementary School 0.3246248
Luther Nick Jeralds Middle-Colwyn Elementary School 0.9573429
Meridian Middle-Colwyn Elementary School 0.0000000
Spring Lake Middle-Colwyn Elementary School 0.0000301
Walnut Street Elementary School-Colwyn Elementary School 0.1957775
Meridian Middle-Luther Nick Jeralds Middle 0.0000000
Spring Lake Middle-Luther Nick Jeralds Middle 0.0000000
Walnut Street Elementary School-Luther Nick Jeralds Middle 0.3130775
Spring Lake Middle-Meridian Middle 0.0000000
Walnut Street Elementary School-Meridian Middle 0.0000000
Walnut Street Elementary School-Spring Lake Middle 0.0000000
Aligns with results shown in q2_model_pooled_by_school.
Pooled - F Test Usage ~ District
One-way ANOVA F-test.
q2_anova_district =aov(num_sessions_week ~ district, data = pooled_vt)summary(q2_anova_district)
Df Sum Sq Mean Sq F value Pr(>F)
district 2 779 389.6 103.2 <2e-16 ***
Residuals 1191 4498 3.8
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Difference between districts’ usage is significant.
Tukey multiple comparisons of means
95% family-wise confidence level
Fit: aov(formula = num_sessions_week ~ district, data = pooled_vt)
$district
diff lwr upr p adj
Kent-Cumberland 2.5683816 2.054648 3.0821157 0
Penn-Cumberland -0.7526599 -1.046812 -0.4585079 0
Penn-Kent -3.3210416 -3.864379 -2.7777039 0
Aligns with results shown in q2_model_pooled_by_district.
Pooled - T Test Usage ~ Male
Welch’s T-test.
t.test(num_sessions_week ~ male, data = pooled_vt)
Welch Two Sample t-test
data: num_sessions_week by male
t = -0.4061, df = 1188.4, p-value = 0.6847
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.2883592 0.1894579
sample estimates:
mean in group 0 mean in group 1
1.518044 1.567495
Difference between the genders’ usage is not significant.
Pooled - F Test Usage ~ Race
One-way ANOVA F-test.
q2_anova_race =aov(num_sessions_week ~ race_pooled, data = pooled_vt)summary(q2_anova_race)
Df Sum Sq Mean Sq F value Pr(>F)
race_pooled 3 177 59.07 13.79 7.68e-09 ***
Residuals 1190 5099 4.29
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Difference between race subgroups’ usage is significant.
Tukey multiple comparisons of means
95% family-wise confidence level
Fit: aov(formula = num_sessions_week ~ race_pooled, data = pooled_vt)
$race_pooled
diff lwr upr p adj
Black-White -0.39831482 -0.9867946 0.1901650 0.3027168
Hispanic-White 0.48889622 -0.2050158 1.1828083 0.2678254
Other Races-White 0.54495793 -0.1994167 1.2893325 0.2356754
Hispanic-Black 0.88721104 0.4350306 1.3393915 0.0000031
Other Races-Black 0.94327275 0.4169204 1.4696251 0.0000264
Other Races-Hispanic 0.05606171 -0.5860070 0.6981304 0.9960027
Align with results shown in q2_model_pooled_by_race.
Pooled - T Test Usage ~ IEP
Welch’s T-test.
t.test(num_sessions_week ~ iep, data = pooled_vt)
Welch Two Sample t-test
data: num_sessions_week by iep
t = 1.695, df = 177.22, p-value = 0.09183
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.05432281 0.71569669
sample estimates:
mean in group 0 mean in group 1
1.581187 1.250500
Difference between the groups is not significant.
Pooled - T Test Usage ~ ELL
Welch’s T-test.
t.test(num_sessions_week ~ ell, data = pooled_vt)
Welch Two Sample t-test
data: num_sessions_week by ell
t = -4.9749, df = 74.139, p-value = 4.103e-06
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-2.464042 -1.054747
sample estimates:
mean in group 0 mean in group 1
1.437238 3.196633
Difference between the groups is significant.
Pooled - T Test Usage ~ Low-income
Welch’s T-test.
t.test(num_sessions_week ~ low_inc, data = pooled_vt)
Welch Two Sample t-test
data: num_sessions_week by low_inc
t = 3.0915, df = 186.92, p-value = 0.002296
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
0.2676821 1.2116721
sample estimates:
mean in group 0 mean in group 1
2.000049 1.260372
Difference between the groups is significant.
Pooled - T Test Usage ~ At-risk
Welch’s T-test.
t.test(num_sessions_week ~ at_risk, data = pooled_vt)
Welch Two Sample t-test
data: num_sessions_week by at_risk
t = -1.1113, df = 722.66, p-value = 0.2668
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.5009584 0.1388212
sample estimates:
mean in group 0 mean in group 1
1.772783 1.953852
Difference between the groups is not significant.
Research Question 3
How is the usability of Live + AI perceived by teachers? How do teachers perceive its impact on student confidence and learning?
Analyze teacher survey outside of R.
Research Question 4
What is the effect of offering LIVE+ AI on student learning compared to students who are not offered the tool? Do improvements in student learning vary meaningfully by student characteristics?
Conduct matched comparisons in e2i Coach.
Research Question 5
Among students who are offered LIVE+ AI, do students who use the tool more show greater gains in learning? PAP: Examine the relationships between student usage and learning outcomes using correlational analyses…Regression models adjusting for school clustering will explore associations, with subgroup analyses conducted as appropriate.
*Repeat for each subject.
POOLED - Regress delta MATH ~ Number of sessions
Change in math achievement
Calculate the change in Math scores between MOY and EOY.
Results after clustering are not significant. Sessions logged between MOY and EOY did not seem to change math z scores significantly.
Regression model - District
# Full modelq5_model_pooled_by_district =lm( delta_math_z ~ total_sessions * district, data = pooled_vt)summary(q5_model_pooled_by_district) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * district, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-5.0639 -0.3508 0.0080 0.3732 3.6653
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.495e-02 3.408e-02 2.199 0.0281 *
total_sessions -6.109e-05 9.766e-04 -0.063 0.9501
districtKent 2.743e-01 1.605e-01 1.709 0.0877 .
districtPenn 4.919e-02 5.870e-02 0.838 0.4022
total_sessions:districtKent -2.628e-03 4.090e-03 -0.642 0.5207
total_sessions:districtPenn -1.903e-03 2.020e-03 -0.942 0.3463
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.723 on 1084 degrees of freedom
(104 observations deleted due to missingness)
Multiple R-squared: 0.005932, Adjusted R-squared: 0.001347
F-statistic: 1.294 on 5 and 1084 DF, p-value: 0.2641
#####################################Cluster-robust standard errors by schoolq5_pooled_by_district_clustered =vcovCL( q5_model_pooled_by_district,cluster =~ school_num)coeftest( q5_model_pooled_by_district,vcov = q5_pooled_by_district_clustered #after clustering)
Reference group is Cumberland. Results after clustering: Cumberland did not see significant changes in math score.
For Kent students, for each additional session, their change in math z scores will yield 0.003 points lower compared to Cumberland student logging sessions at the same rate, p<0.001 (2.7434e-01 represents the delta math score for Kent students with no sessions logged, p<0.001)
For Penn students, each additional session, their change in math z scores will yield 0.002 points lower compared to Cumberland students logging sessions at the same rate, p<0.01 (4.9191e-02 represent the delta math score for Penn students with no sessions logged, p>0.4).
—- SUBGROUPS —-
Regression model - Male
# Full modelq5_model_pooled_by_male =lm( delta_math_z ~ total_sessions * male, data = pooled_vt)summary(q5_model_pooled_by_male) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * male, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-5.1163 -0.3384 0.0119 0.3650 3.6621
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.082046 0.037285 2.201 0.028 *
total_sessions -0.001026 0.001111 -0.924 0.356
male 0.035753 0.054516 0.656 0.512
total_sessions:male 0.001523 0.001656 0.920 0.358
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.7234 on 1086 degrees of freedom
(104 observations deleted due to missingness)
Multiple R-squared: 0.002974, Adjusted R-squared: 0.0002201
F-statistic: 1.08 on 3 and 1086 DF, p-value: 0.3566
#####################################Cluster-robust standard errors by schoolq5_pooled_by_male_clustered =vcovCL( q5_model_pooled_by_male,cluster =~ school_num)coeftest( q5_model_pooled_by_male,vcov = q5_pooled_by_male_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.08204650 0.02078827 3.9468 8.431e-05 ***
total_sessions -0.00102622 0.00045808 -2.2403 0.02527 *
male 0.03575348 0.03644306 0.9811 0.32677
total_sessions:male 0.00152331 0.00074228 2.0522 0.04039 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_male, type ="text", out ="q5_math_model_male.txt")
Reference group is female. Results after clustering:
Females with zero sessions logged observed 0.082 point change in math z score, p<0.001. For female students, each additional session yielded -0.001 point change in math z scores, p<0.05. Males with zero sessions logged observed 0.04 points more in delta math z score when compared to females that no sessions logged (not sig).For male students, each additional session yielded an additional 0.001 point change in math z score, p<0.05.
Regression model - Race
# Full modelq5_model_pooled_by_race =lm( delta_math_z ~ total_sessions * race_pooled, data = pooled_vt)summary(q5_model_pooled_by_race) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * race_pooled, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-5.0716 -0.3323 0.0087 0.3659 3.6458
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.1871186 0.1127552 1.660 0.0973 .
total_sessions -0.0016315 0.0039479 -0.413 0.6795
race_pooledBlack -0.0890280 0.1171837 -0.760 0.4476
race_pooledHispanic -0.0767686 0.1379828 -0.556 0.5781
race_pooledOther Races -0.1386245 0.1446181 -0.959 0.3380
total_sessions:race_pooledBlack 0.0006625 0.0040736 0.163 0.8708
total_sessions:race_pooledHispanic 0.0016806 0.0045170 0.372 0.7099
total_sessions:race_pooledOther Races 0.0042348 0.0045768 0.925 0.3550
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.7247 on 1082 degrees of freedom
(104 observations deleted due to missingness)
Multiple R-squared: 0.003076, Adjusted R-squared: -0.003374
F-statistic: 0.4769 on 7 and 1082 DF, p-value: 0.8518
#####################################Cluster-robust standard errors by schoolq5_pooled_by_race_clustered =vcovCL( q5_model_pooled_by_race,cluster =~ school_num)coeftest( q5_model_pooled_by_race,vcov = q5_pooled_by_race_clustered #after clustering)
White is the reference group. Results after clustering:
Main race groups did not observe significant changes to math z scores from MOY to EOY.
Regarding Pacific Islanders (PIs): The average white students with no sessions logged saw 0.19 point increase in math z score, p<0.05. For white students, each additional session yielded -0.002 point decrease in math z score (not sig).
For PI studnets with no sessions logged, change in math z scores were 0.2 points less than whites with no sessions logged. decrease in math z scores (not sig). For PI students, each additional session yielded an additional 0.004 point change in math z score when compared to white students logging the same number of sessions, p<0.05.
Regression model - IEP
# Full modelq5_model_pooled_by_iep =lm( delta_math_z ~ total_sessions * iep, data = pooled_vt)summary(q5_model_pooled_by_iep) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * iep, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-5.0872 -0.3434 0.0064 0.3615 3.6370
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.1048386 0.0294227 3.563 0.000382 ***
total_sessions -0.0004479 0.0009024 -0.496 0.619730
iep -0.0302648 0.0783942 -0.386 0.699528
total_sessions:iep 0.0006983 0.0022312 0.313 0.754369
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.7244 on 1086 degrees of freedom
(104 observations deleted due to missingness)
Multiple R-squared: 0.0002995, Adjusted R-squared: -0.002462
F-statistic: 0.1085 on 3 and 1086 DF, p-value: 0.9552
#####################################Cluster-robust standard errors by schoolq5_pooled_by_iep_clustered =vcovCL( q5_model_pooled_by_iep,cluster =~ school_num)coeftest( q5_model_pooled_by_iep,vcov = q5_pooled_by_iep_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.10483863 0.02629755 3.9866 7.15e-05 ***
total_sessions -0.00044793 0.00034447 -1.3004 0.1938
iep -0.03026483 0.07185257 -0.4212 0.6737
total_sessions:iep 0.00069829 0.00129949 0.5374 0.5911
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_iep, type ="text", out ="q5_math_model_iep.txt")
Reference group is no-IEP. Results after clustering:
No observed significant changes to math z scores from MOY to EOY for IEP or nonIEP students using Live+AI.
Regression model - ELL
# Full modelq5_model_pooled_by_ell =lm( delta_math_z ~ total_sessions * ell, data = pooled_vt)summary(q5_model_pooled_by_ell) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * ell, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-5.0864 -0.3421 0.0111 0.3664 3.6357
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.1067335 0.0278783 3.829 0.000136 ***
total_sessions -0.0006051 0.0008745 -0.692 0.489104
ell -0.1231560 0.1298468 -0.948 0.343100
total_sessions:ell 0.0031921 0.0028485 1.121 0.262697
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.724 on 1086 degrees of freedom
(104 observations deleted due to missingness)
Multiple R-squared: 0.001348, Adjusted R-squared: -0.001411
F-statistic: 0.4885 on 3 and 1086 DF, p-value: 0.6903
#####################################Cluster-robust standard errors by schoolq5_pooled_by_ell_clustered =vcovCL( q5_model_pooled_by_ell,cluster =~ school_num)coeftest( q5_model_pooled_by_ell,vcov = q5_pooled_by_ell_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.10673348 0.02538405 4.2047 2.829e-05 ***
total_sessions -0.00060515 0.00041772 -1.4487 0.1477
ell -0.12315598 0.21497960 -0.5729 0.5668
total_sessions:ell 0.00319210 0.00274888 1.1612 0.2458
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_ell, type ="text", out ="q5_math_model_ell.txt")
Reference group is no-ELL. Results after clustering:
No observed significant changes to math z scores from MOY to EOY for ELL or nonELL students using Live+AI.
Regression model - Low-Income
# Full modelq5_model_pooled_by_low_inc =lm( delta_math_z ~ total_sessions * low_inc, data = pooled_vt)summary(q5_model_pooled_by_low_inc) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * low_inc, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-1.4652 -0.3397 -0.0321 0.2896 3.1671
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.1963326 0.0580308 3.383 0.000782 ***
total_sessions -0.0013695 0.0014904 -0.919 0.358666
low_inc -0.0728116 0.0710396 -1.025 0.305970
total_sessions:low_inc 0.0005118 0.0022121 0.231 0.817142
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.515 on 427 degrees of freedom
(763 observations deleted due to missingness)
Multiple R-squared: 0.004977, Adjusted R-squared: -0.002014
F-statistic: 0.7119 on 3 and 427 DF, p-value: 0.5453
#####################################Cluster-robust standard errors by schoolq5_pooled_by_low_inc_clustered =vcovCL( q5_model_pooled_by_low_inc,cluster =~ school_num)coeftest( q5_model_pooled_by_low_inc,vcov = q5_pooled_by_low_inc_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.19633255 0.06084849 3.2266 0.001349 **
total_sessions -0.00136950 0.00069307 -1.9760 0.048799 *
low_inc -0.07281162 0.01510821 -4.8193 2.005e-06 ***
total_sessions:low_inc 0.00051180 0.00171584 0.2983 0.765635
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_low_inc, type ="text", out ="q5_math_model_frpl.txt")
Reference group are students who are not low-income.
For non LC students with zero sessions, they observed a 0.2 point change in math z score, p<0.01. For non LC students, each additional session logged yielded a -0.001 point decrease in math z scores , p<0.05. For LC students with zero sessions, their change in math z scores was 0.07 less than non LC students with zero sessions, p<0.001. No significant changes in math z scores for low-income students using Live+AI.
Regression model - At-Risk
# Full modelq5_model_pooled_by_at_risk =lm( delta_math_z ~ total_sessions * at_risk, data = pooled_vt)summary(q5_model_pooled_by_at_risk) #before clustering by school
Call:
lm(formula = delta_math_z ~ total_sessions * at_risk, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-5.0249 -0.3360 0.0261 0.3792 3.6058
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.1357022 0.0462215 2.936 0.00343 **
total_sessions -0.0003874 0.0012116 -0.320 0.74928
at_risk -0.1276652 0.0801370 -1.593 0.11157
total_sessions:at_risk 0.0019666 0.0024655 0.798 0.42534
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.8148 on 738 degrees of freedom
(452 observations deleted due to missingness)
Multiple R-squared: 0.003554, Adjusted R-squared: -0.0004962
F-statistic: 0.8775 on 3 and 738 DF, p-value: 0.4523
#####################################Cluster-robust standard errors by schoolq5_pooled_by_at_risk_clustered =vcovCL( q5_model_pooled_by_at_risk,cluster =~ school_num)coeftest( q5_model_pooled_by_at_risk,vcov = q5_pooled_by_at_risk_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.3570e-01 1.8517e-02 7.3285 6.122e-13 ***
total_sessions -3.8736e-04 9.5267e-06 -40.6607 < 2.2e-16 ***
at_risk -1.2767e-01 7.8747e-02 -1.6212 0.1054
total_sessions:at_risk 1.9666e-03 1.9838e-03 0.9913 0.3219
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_at_risk, type ="text", out ="q5_math_model_at-risk.txt")
For students with no sessions logged, they saw a -0.06 point change in delta ELA z scores, p<0.05.For each additional session logged, students on average observed a 0.002 point change increase in delta ELA z scores, p<0.1.
Regression model - District
# Full modelq5_model_pooled_by_district =lm( delta_ela_z ~ total_sessions * district, data = pooled_vt)summary(q5_model_pooled_by_district) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * district, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-4.3431 -0.3252 0.0065 0.3337 2.6313
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -4.960e-02 3.240e-02 -1.531 0.1260
total_sessions 1.348e-03 1.138e-03 1.185 0.2362
districtKent 3.118e-01 1.367e-01 2.281 0.0227 *
districtPenn -8.037e-02 5.301e-02 -1.516 0.1298
total_sessions:districtKent 2.679e-03 3.605e-03 0.743 0.4576
total_sessions:districtPenn -7.821e-05 1.923e-03 -0.041 0.9676
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.6334 on 1024 degrees of freedom
(164 observations deleted due to missingness)
Multiple R-squared: 0.04489, Adjusted R-squared: 0.04023
F-statistic: 9.626 on 5 and 1024 DF, p-value: 5.411e-09
#####################################Cluster-robust standard errors by schoolq5_pooled_by_district_clustered =vcovCL( q5_model_pooled_by_district,cluster =~ school_num)coeftest( q5_model_pooled_by_district,vcov = q5_pooled_by_district_clustered #after clustering)
Reference group is Cumberland. Results after clustering:
For Cumberland students, for each additional session, delta ELA z scores would increase by an additional 0.001 points, p<0.001.
For Kent students, for each additional session, delta ELA z scores increased by an additional 0.002 points more when compared to Cumberland students logging sessions at the same rate, p<0.001 (3.1182e-01 represents the delta ELA score for Kent students with no sessions logged, p<0.001).
For Penn students, each additional session, delta ELA z scores decreased by an additional 0.00007 points less when compared to Cumberland students, p>0.9 (-8.0371e-02 represent the delta ELA score for Penn students with no sessions logged, p<0.05).
—- SUBGROUPS —-
Regression model - Male
# Full modelq5_model_pooled_by_male =lm( delta_ela_z ~ total_sessions * male, data = pooled_vt)summary(q5_model_pooled_by_male) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * male, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-4.3108 -0.3341 0.0018 0.3232 2.9199
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.0494346 0.0348837 -1.417 0.1567
total_sessions 0.0025215 0.0011777 2.141 0.0325 *
male -0.0352742 0.0514335 -0.686 0.4930
total_sessions:male 0.0002123 0.0017779 0.119 0.9050
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.6445 on 1026 degrees of freedom
(164 observations deleted due to missingness)
Multiple R-squared: 0.009079, Adjusted R-squared: 0.006182
F-statistic: 3.134 on 3 and 1026 DF, p-value: 0.02484
#####################################Cluster-robust standard errors by schoolq5_pooled_by_male_clustered =vcovCL( q5_model_pooled_by_male,cluster =~ school_num)coeftest( q5_model_pooled_by_male,vcov = q5_pooled_by_male_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.04943458 0.02304449 -2.1452 0.03217 *
total_sessions 0.00252148 0.00110103 2.2901 0.02222 *
male -0.03527425 0.03344898 -1.0546 0.29187
total_sessions:male 0.00021226 0.00232704 0.0912 0.92734
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_male, type ="text", out ="q5_ela_model_male.txt")
Reference group is female. Results after clustering:
For female students, each additional session yielded -0.0025 point change in math z scores, p<0.05. For treatment female students with zero sessions, delta ELA z score equaled -0.04, p<0.05.
For male students, no significant changes to ELA z scores were observed with or without sessions logged, when compared to females.
Regression model - Race
# Full modelq5_model_pooled_by_race =lm( delta_ela_z ~ total_sessions * race_pooled, data = pooled_vt)summary(q5_model_pooled_by_race) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * race_pooled, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-4.3010 -0.3379 0.0022 0.3312 2.9358
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -6.516e-02 9.770e-02 -0.667 0.505
total_sessions 2.654e-03 2.984e-03 0.889 0.374
race_pooledBlack -2.899e-02 1.023e-01 -0.283 0.777
race_pooledHispanic 1.788e-01 1.232e-01 1.451 0.147
race_pooledOther Races 9.529e-03 1.274e-01 0.075 0.940
total_sessions:race_pooledBlack -7.509e-05 3.198e-03 -0.023 0.981
total_sessions:race_pooledHispanic -2.413e-03 3.724e-03 -0.648 0.517
total_sessions:race_pooledOther Races 7.593e-04 3.753e-03 0.202 0.840
Residual standard error: 0.6434 on 1022 degrees of freedom
(164 observations deleted due to missingness)
Multiple R-squared: 0.01625, Adjusted R-squared: 0.009516
F-statistic: 2.412 on 7 and 1022 DF, p-value: 0.01881
#####################################Cluster-robust standard errors by schoolq5_pooled_by_race_clustered =vcovCL( q5_model_pooled_by_race,cluster =~ school_num)coeftest( q5_model_pooled_by_race,vcov = q5_pooled_by_race_clustered #after clustering)
White is the reference group. Significant results after clustering:
Black students with no sessions logged observed a 0.03 point change less than White students with no sessions logged, p<0.1. Hispanic students with no sessions logged observed a 0.2 point change more than White student with no sessions logged, p<0.01. For Multi-racial students, for each additional session logged, ELA z scores saw -0.005 point change, p<0.01. For PI students, for each additional session logged, ELA z scores saw 0.01 point increase more than White students using at the same rate, p<0.05.
Regression model - IEP
# Full modelq5_model_pooled_by_iep =lm( delta_ela_z ~ total_sessions * iep, data = pooled_vt)summary(q5_model_pooled_by_iep) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * iep, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-4.3282 -0.3306 0.0044 0.3295 2.9052
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.0670923 0.0275730 -2.433 0.01513 *
total_sessions 0.0026630 0.0009461 2.815 0.00498 **
iep 0.0076885 0.0747180 0.103 0.91806
total_sessions:iep -0.0003789 0.0026302 -0.144 0.88548
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.6447 on 1026 degrees of freedom
(164 observations deleted due to missingness)
Multiple R-squared: 0.008496, Adjusted R-squared: 0.005596
F-statistic: 2.93 on 3 and 1026 DF, p-value: 0.03269
#####################################Cluster-robust standard errors by schoolq5_pooled_by_iep_clustered =vcovCL( q5_model_pooled_by_iep,cluster =~ school_num)coeftest( q5_model_pooled_by_iep,vcov = q5_pooled_by_iep_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.06709227 0.02976551 -2.2540 0.0244 *
total_sessions 0.00266301 0.00162511 1.6387 0.1016
iep 0.00768854 0.03291664 0.2336 0.8154
total_sessions:iep -0.00037892 0.00171630 -0.2208 0.8253
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_iep, type ="text", out ="q5_ela_model_iep.txt")
Reference group is no-IEP. Results after clustering:
No observed significant changes to ELA z scores from MOY to EOY for IEP or nonIEP students using Live+AI.
Regression model - ELL
# Full modelq5_model_pooled_by_ell =lm( delta_ela_z ~ total_sessions * ell, data = pooled_vt)summary(q5_model_pooled_by_ell) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * ell, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-4.3228 -0.3275 0.0039 0.3338 2.9063
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.0727908 0.0263345 -2.764 0.00581 **
total_sessions 0.0027743 0.0009447 2.937 0.00339 **
ell 0.1354500 0.1184224 1.144 0.25298
total_sessions:ell -0.0023535 0.0028404 -0.829 0.40753
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.6443 on 1026 degrees of freedom
(164 observations deleted due to missingness)
Multiple R-squared: 0.009739, Adjusted R-squared: 0.006844
F-statistic: 3.364 on 3 and 1026 DF, p-value: 0.01818
#####################################Cluster-robust standard errors by schoolq5_pooled_by_ell_clustered =vcovCL( q5_model_pooled_by_ell,cluster =~ school_num)coeftest( q5_model_pooled_by_ell,vcov = q5_pooled_by_ell_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.07279076 0.02519819 -2.8887 0.003949 **
total_sessions 0.00277429 0.00127184 2.1813 0.029386 *
ell 0.13544998 0.11680400 1.1596 0.246467
total_sessions:ell -0.00235354 0.00077728 -3.0279 0.002524 **
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_ell, type ="text", out ="q5_ela_model_ell.txt")
For non ELLs with no sessions logged, they observed a -0.07 point change in ELA z score, p<0.01. For non ELLs, for each additional session, they observed 0.003 point change in ELA z scores, p<0.05. For ELLs with no sessions logged, ELA z scores did not change significantly compared to no ELLs with no sessions logged, p>0.2. For ELLs, for each additional session, they observed a 0.002 point change less compared to non ELLs using at the same rate, p<0.01.
Regression model - Low-Income
# Full modelq5_model_pooled_by_low_inc =lm( delta_ela_z ~ total_sessions * low_inc, data = pooled_vt)summary(q5_model_pooled_by_low_inc) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * low_inc, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-1.67264 -0.28749 0.00877 0.29235 2.79314
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.020920 0.061328 -0.341 0.7332
total_sessions 0.001840 0.001576 1.168 0.2436
low_inc -0.126439 0.074977 -1.686 0.0924 .
total_sessions:low_inc 0.005403 0.002336 2.313 0.0212 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.5435 on 430 degrees of freedom
(760 observations deleted due to missingness)
Multiple R-squared: 0.04367, Adjusted R-squared: 0.037
F-statistic: 6.545 on 3 and 430 DF, p-value: 0.0002459
#####################################Cluster-robust standard errors by schoolq5_pooled_by_low_inc_clustered =vcovCL( q5_model_pooled_by_low_inc,cluster =~ school_num)coeftest( q5_model_pooled_by_low_inc,vcov = q5_pooled_by_low_inc_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.0209202 0.1047096 -0.1998 0.841737
total_sessions 0.0018404 0.0021022 0.8754 0.381829
low_inc -0.1264393 0.0980183 -1.2900 0.197759
total_sessions:low_inc 0.0054029 0.0020389 2.6499 0.008348 **
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_low_inc, type ="text", out ="q5_ela_model_frpl.txt")
Reference group are students who are not low-income.
Non LC students’ ELA z scores did not change significantly, regardless of sessions logged. For LC students, each additional session saw a 0.002 point change less compared non LC students using at the same rate, p<0.01.
Regression model - At-Risk
# Full modelq5_model_pooled_by_at_risk =lm( delta_ela_z ~ total_sessions * at_risk, data = pooled_vt)summary(q5_model_pooled_by_at_risk) #before clustering by school
Call:
lm(formula = delta_ela_z ~ total_sessions * at_risk, data = pooled_vt)
Residuals:
Min 1Q Median 3Q Max
-4.3154 -0.3362 0.0134 0.3538 2.7239
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.079278 0.043465 -1.824 0.0686 .
total_sessions 0.002335 0.001382 1.689 0.0916 .
at_risk 0.116320 0.073798 1.576 0.1154
total_sessions:at_risk 0.002167 0.002567 0.844 0.3989
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.7029 on 679 degrees of freedom
(511 observations deleted due to missingness)
Multiple R-squared: 0.02216, Adjusted R-squared: 0.01784
F-statistic: 5.13 on 3 and 679 DF, p-value: 0.00163
#####################################Cluster-robust standard errors by schoolq5_pooled_by_at_risk_clustered =vcovCL( q5_model_pooled_by_at_risk,cluster =~ school_num)coeftest( q5_model_pooled_by_at_risk,vcov = q5_pooled_by_at_risk_clustered #after clustering)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.07927836 0.01962216 -4.0402 5.950e-05 ***
total_sessions 0.00233470 0.00050267 4.6446 4.093e-06 ***
at_risk 0.11631951 0.05171012 2.2495 0.0248 *
total_sessions:at_risk 0.00216666 0.00403549 0.5369 0.5915
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
stargazer(q5_model_pooled_by_at_risk, type ="text", out ="q5_ela_model_at-risk.txt")
Reference group is not at-risk. Results after clustering:
For non at-risk students with no sessions, ELA z score changed by -0.08 points, p<0.001. For non at-risk students, each additional session yielded -0.002 point change in math z score, p<0.001. For at-risk student with no sessions, ELA z score changed by 0.11 points more than non at-risk students with no sessions logged, p<0.05. At-risk students did not see significant changes to ELA z scores when using Live+AI.
Request from QA
How many students have MOY or EOY z-score with absolute value greater than 3?
Performed calculations among students who were offered Live+AI (pooled_vt, where VT ID was present for a student; N=1194).