library(tidyverse) # For ggplot, mutate(), filter(), and friends
library(broom) # For converting models to data frames
library(ggdag) # For drawing DAGs
library(scales) # For rescaling data with rescale()
library(truncnorm) # For truncated normal distributions
library(patchwork) # For customised plots
set.seed(1234) # Make any random stuff be the same every time you run this
# Turn off the message that happens when you use group_by() and summarize()
options(dplyr.summarise.inform = FALSE)TFA’s Ignite Fellowship creating learning sparks in students: Impact Evaluation
Introduction
Education is seen as one the most powerful tools to change ones live and livelihood. As remarked by the Nobel Laureate from Pakistan: Malala Yousafzai, “One child, one teacher, one book, one pen can change the world.”
Given the nearly undisputed importance of education, the global society has been committed to exploring and implementing educational solutions on a wide array of issues: equity, inclusivity, access, and quality of education. Despite the multi-faceted, multi-country, and multi-year efforts and initiatives around the world, UNESCO labeled the situation as a global learning crisis a little over a decade ago, which was only exacerbated with the nCovid-19 pandemic in early 2020 (UNESCO, World Bank, and United Nation’s Children Fund 2021). Similar instances were observed across the US with long school closures. Among the many innovative solutions in the US education space to combat the most damaging education crisis in history, ranging from remote learning with a variety of education technology to a high-dosage virtual tutoring, many organizations also put forth their efforts to combat the learning inequity.
One of these organizations was Teach For America’s Ignite Fellowship which focused its efforts towards virtual tutoring for students in different communities across the US. Given the unique yet imperative shift from the widely-acclaimed TFA’s Teacher Corps program to a virtual learning environment in the backdrop of the global pandemic, it is important and necessary to understand how impactful the modality and delivery of the program is in achieving the simple yet significant outcome of increasing student achievements. Having the empirical base to view the causal effect of this program will not only yield support to the case of nationwide tutoring program in the US by Kraft and Falken (2021) but also supplement the education-centric initiative to bridge the learning gap of students (Kane 2022).
Program overview
Teach For America, a nonprofit organization with the aim to improve education and opportunities for children in the US, launched a virtual tutoring program – Ignite – during the backdrop of COVID-19 in late 2020. The goal was to cover the learning losses of children due to weeks and months of school closures which adversely impacted the academic learning and performance of students, particularly those belonging to marginalized communities (“COVID-19 and the Widening Learning Gap | McKinsey,” n.d.). Currently, this virtual fellowship covers roughly 90 schools across 28 Teach for America groups (“Answers to Frequently Asked Questions \Textbar Teach For America” n.d.).
Ignite is a community-based, education-centric initiative sponsored by Teach for America where Fellows (primarily university students) lead a small group virtual learning session for K-8 students with the oversight of a school-based veteran educator. The time commitment is up to 5 hours per week during school hours with an age criterion of being at least 18 years old while authorized to work in the US. The learning sessions are spread across a semester either during the Fall or Spring for 4 months in one semester. The subject focus is on elementary reading or middle school mathematics whereby the Fellows provide students with high-impact tutoring customized to the needs of that school. Additionally, the Fellows are expected to provide individualized learning that drives students’ achievement and create a higher sense of belonging in the classroom. This is made possible by dedicated support from Teach for America to the Fellows who receive dedicated support and training to ensure that the educational goal of the fellowship with regards to equity, accessibility, and high learning quality are met. (“Ignite Fellowship Teach For America,” n.d.)
Program theory and implementation
Program theory and impact theory graph
The Ignite Fellowship was conceived as an attempt to to foster belonging among student while accelerating their learning, particularly given the context of learning loss during nCovid-19. As per a study on learning loss and student achievement, the impact of a well-trained tutor teaching a small group of students thrice a week was equivalent of roughly five months of academic learning (Kane 2022) . Additionally, tutoring was found to have a more significant impact relative to smaller class size, summer classes, or longer school year (Kraft and Falken 2021).
Beyond the academic performance, Future Ed (2021) highlighted that students in the Milwaukee Tutoring Program took six fewer absences in an academic year relative to their peers. Given the evidence for the benefits of high quality, customized, and small group tutoring, the Ignite Fellowship was envisaged to ensure that gaps in the educational landscape were addressed. The impact theory graph for Ignite Fellowship is shown below which highlights the direct links between virtual tutoring and improved educational outcomes.
Logic model
Inputs
Number of Fellows
Number of veteran educators
Number of schools on-board for the Ignite Fellowship
Technological platforms for learning
Curriculum resources for learning
Activities
Orientation sessions provided to Fellows
Teaching session provided by Fellows to students
Supervision by veteran educators
Outputs
Number of students taking the session
Number of weekly hours by Fellows
Number of online sessions taken by Fellows
Change in test scores of students
Outcomes
Decrease in nCovid19-induced learning loss
Better education achievement by students
Lesser student dropouts
Few repeat graders
Diagram
Outcome and causation
Main outcome
The main outcome of interest that will be the focus of this evaluation is test scores because TFA’s Ignite Fellowship has the primary aim of enhancing student academic outcomes, which are more often than not measured by the examination results of students. It also allows a wider comparison of this program with other regional education efforts and their associated impact reports.
Measurement
Outcome of interest: Better test scores in upcoming exams
Some defining attributes of our outcome of interest: Better test scores in upcoming exams are stated below:
- Better literacy score
- Better numeracy score
- Consistently good academic results
Attribute 1: Better literacy scores
- Measurable definition: The percentage marks achieved in the final assessment of literacy/reading for each grades subsequent to kindergarten.
- Ideal measurement: The students are tested in a standard and universal manner across Utah with negligible variations in other factors such as teaching quality, house factors, and school infrastructure. Thereafter, the marks achieved in the final exams are recorded.
- Feasible measurement: Differences in exam standards, teachers quality and school characteristics are bound to appear. Additionally, rolling out a standardized testing instrument will be difficult and time consuming. Therefore, only the comparison of literacy scores in final exams (despite these differences) can be conducted.
- Measurement of program effect: Comparison of literacy exam scores with students who did not attend the virtual Ignite tutoring.
Attribute 2: Better numeracy scores
- Measurable definition: The percentage marks achieved in the final assessment of numeracy/mathematics for each grades subsequent to kindergarten.
- Ideal measurement: The students are tested in a standard and universal manner across Utah with negligible variations in other factors such as teaching quality, house factors, and school infrastructure. Thereafter, the marks achieved in the final exams are recorded.
- Feasible measurement: Differences in exam standards, teachers quality and school characteristics are bound to appear. Additionally, rolling out a standardized testing instrument will be difficult and time consuming. Therefore, only the comparison of literacy scores in final exams (despite these differences) can be conducted.
- Measurement of program effect: Comparison of numeracy exam scores with students who did not attend the virtual Ignite tutoring.
Attribute 3: Consistently good academic results
- Measurable definition: Year to year variation in academic results in final assessment is less than 10%
- Ideal measurement: The difficulty level of the successive years of assessments should be as close to normal as possible without too easy or too hard of a test paper so as to allow a fair comparison of academic results.
- Feasible measurement: There is a possibility that certain years may have more difficult or easy paper than usual. Therefore, the scores may have to be weighted as per the difficulty level of that year.
- Measurement of program effect: Comparison of the percentage of students with consistently good academic results for both students with and without the virtual Ignite tutoring.
Causal theory
Students enrolled in virtual tutoring (via Ignite Fellowship) to avail virtual tutoring leads to better test scores in the exams. Variables that need to be controlled (not in our study as we utilize RCT):
Parental education: Higher educated parents may be able to provide education support to their children.
Household income: Better off families can afford private tutors.
Previous test scores: High achieving students will not be eligible for the virtual tutoring program.
Teacher quality: High quality teachers will have better students in academic terms.
Geographical region: Only the region (communities) where the Ignite Fellowship is active will see this program.
Subject: The tutoring is limited to literacy and numeracy.
Grade: The tutoring is for K8 schools.
Innate ability (unobserved): This is included for illustrative purposes only.
Hypotheses
The likely outcome of a virtual tutoring program on student’s test score is assumed to be positive for two main reasons:
TFA as an organization has largely been successful in generative positive results with its Teacher Corps program, and similar results are expected in this program.
Supplemental learning, especially in a focused and customized way, is expected to raise education outcomes due to better content understanding of subjects.
Data and methods
Identification strategy
The actual program effect of Ignite Fellowship - Virtual Tutoring will be measured using RCT where the students within a school will be randomly assigned to treatment (virtual tutoring) or control (no virtual tutoring) to see the differential impact of this program on test scores at the end of the academic year. Using RCT ensures that there are no confounding variables that influence both the treatment and the outcome, hence removing selection bias. Additionally, the issue of endogeneity is expected to minimize due to the random assignment of students and the existence of a control group, which means any difference in results should be attributed to the treatment itself.
This evaluation study will perform well in the case of internal validity with no issue of selection (due to random assignment); low likelihood of attrition in a school year; no maturation, secular trends, or seasonality (due to control group); testing (both groups will give the same test). Some aspects like intervening events could occur but they are likely effect both the groups equally on average. However, the external validity may be lacking because the generalizibility of program effects is dependent on contextual factors which may significantly influence results (for instance, virtual tutoring may not be as effective if ed-tech resources and internet connection is sub-par).
Data
The data will be collected from the schools administration where the program is in place. Data on student participation status and final examination score will be used to evaluate the impact of this program. Overall, this data should be easy to access as it is a standard protocol to have such information in a structured and updated format.
Synthetic analysis
Loading libraries
Nodes Specification
Test Scores: scale from 0-100, mostly around 75 ± 10, best to use normal distribution. Round to 1 decimal place.
Virtual Tutoring: binary 0/1, TRUE/FALSE variable, where 50% of the students were assigned to and participated (100% compliance assumed) the virtual tutoring sessions.
Relationship between the nodes
Test Scores ~ Virtual Tutoring: Assignment to virtual tutoring increases students test scores by 10 marks.
Data Generating Process
# Make this randomness consistent
set.seed(1234)
# Simulate 500 students
n_people <- 500
rct_data <- tibble(
# Make an ID column
id = 1:n_people,
) |>
# Randomly assign students to virtual tutoring
mutate(tutoring = rbinom(n_people, 1, 0.5)) |>
# Generate test score variable that is based on attending virtual tutoring sessions
mutate(score_base = rnorm(n_people, mean = 75, sd = 10),
# Add the virtual tutoring effect
score_effect = (10 * tutoring),
# Final test score varible
test_score = score_base + score_effect + rnorm(n_people, 0, sd = 2.5),
# Rescale to keep it under 100
test_score = rescale(test_score, to = c(37, 99)),
# Round to 1 decimal place
test_score = round(test_score, 1))Check Balance
We studied 500 students for this RCT with random assignment to the Ignite Fellowship - Virtual Tutoring program. First we will check if the split between treatment and control is 50/50. The results below indicate that the program was evenly divided, which is great!
rct_data |>
count(tutoring) |>
mutate(prop = n / sum(n))# A tibble: 2 × 3
tutoring n prop
<int> <int> <dbl>
1 0 243 0.486
2 1 257 0.514We could have checked the pre-treatment test scores but they were not incorporated in the RCT design. This concludes the diagnostics test.
Relationship Verification via plot and model
Visually, the spread of the treatment group is different from that of the control group, as integrated in the DGP.
# Visualizing test score and virtual tutoring using overlapping densities
ggplot(rct_data, aes(x = test_score, fill = factor(tutoring))) +
geom_density(alpha = 0.5)
The regression model shows that participation in the virtual tutoring program causes an increase in the final examination score of students by 10.4 marks on average (which is what we expect as indicated in Relationship between the nodes sub-section!). Additionally, this effect is statistically significant. Notice how confounding variables are not included in the DGP or regression model because the evaluation strategy is RCT, so we don’t need to worry about selection bias.
# Regression Model
lm(test_score ~ tutoring, data = rct_data) |> tidy()# A tibble: 2 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) 62.5 0.649 96.3 0
2 tutoring 10.4 0.905 11.5 2.12e-27rct_data <- rct_data |>
mutate(tutoring_factor = factor(tutoring, levels = c(0,1) ,
labels = c("No Tutoring", "Tutoring")))
ggplot(rct_data, aes(x = tutoring_factor, y = test_score, color = tutoring_factor)) +
stat_summary(geom = "pointrange", fun.data = "mean_se", fun.args = list(mult = 1.96)) +
guides(color = "none") +
labs(x = NULL, y = "Test Scores")
Cleaning the data set and saving as csv file
# Using relevant columns in dataset
rct_data_final <- rct_data |>
select(id, tutoring, test_score)
head(rct_data_final)# A tibble: 6 × 3
id tutoring test_score
<int> <int> <dbl>
1 1 0 67.2
2 2 1 83.6
3 3 1 80.3
4 4 1 79.7
5 5 1 60
6 6 1 70.3# Saving the data set as csv file
write_csv(rct_data_final, "data/ignite_tutoring_rct.csv")Conclusion
Students who received virtual tutoring as part of Ignite Fellowship by TFA will see a rise in final examination results by roughly 10 marks. This causal effect is statistically significant and adds to the empirical efficacy of the tutoring program to positively impact student achievement. Although a cost-benefit analysis is warranted, it can be safely assumed that the program cost is likely to be lower than other in-person initiative (which may generate similar academic outcomes) given the low-cost and tech-oriented nature of the virtual tutoring. Hence, the causal relationship justifies the continued operation of this program with more avenues for evidence-backed initiatives and further improvements in the program by experimenting with different “add-on” options (for instance, virtual tutoring with more versus less days or group size, a hybrid setting for in-person socialization, addition of an incentive for students, etc).
References
“COVID-19 and the Widening Learning Gap | McKinsey.” n.d. https://www.mckinsey.com/industries/education/our-insights/covid-19-and-education-an-emerging-k-shaped-recovery.
Future Ed. 2021. “Tutoring - FutureEd.” https://www.future-ed.org/wp-content/uploads/2021/06/Tutoring_Final.pdf.
“Ignite Fellowship Teach For America.” n.d. https://www.teachforamerica.org/ignite-fellowship.
Kane, Thomas. 2022. “Kids Are Far, Far Behind in School.” https://www.theatlantic.com/ideas/archive/2022/05/schools-learning-loss-remote-covid-education/629938/.
Kraft, Matthew A., and Grace Falken. 2021. “A Blueprint for Scaling Tutoring Across Public Schools.” https://edworkingpapers.com/ai20-335.
UNESCO, World Bank, and United Nation’s Children Fund. 2021. “The State of the Global Education Crisis: A Path to Recovery - UNESCO Digital Library.” https://unesdoc.unesco.org/ark:/48223/pf0000380128.