B2SOF Sample Simulations

Summary

This notebook simulates the effect of the new (as of July 7th 2022) evaluation design on payments. I produced this pretty quickly and thus there isn’t much explanation for the formulas / calculations. See the earlier notebooks with standard error calculations and simulations for more background.

Evaluation Design

Simulation Approach

With this evaluation design, each annual impact estimate is independent (assuming we don’t use the grade 3 assessment in year 4 for payment calculations). Thus, to simulate the effect of uncertainty on payments, I simulate draw random from the impact estimates rather than each data collection rounds.

In the code below, I assume that the actual 2 year effect and the target for each cohort is .2 but this is easily changed by changing the parameters in the code.

Standard Error Formula

Since we are resampling 50% of schools in both the treatment and control groups the standard error will be the same as if we are resampling schools in treatment but tracking the same schools for control. The variance for the estimator would (see scenario 2 from the earlier notebook):

\[ V(\hat{\tau}) = 2A(1-R_J^2)+2B(1-R_K^2) + 2*(A+B)*(1-R^2_L)\]

Simulations

library(tidyverse)

# sampling inputs
J <- 50 # Number of schools per arm
K <- 25 # Number of students per school
rho <- 0.1 # ICC
rsj <- .2
rsk <- .2
rsl <- .63

# variance calculations
A <- rho/J
B <- (1-rho)/(J*K)
var <- 2*A*(1-rsj)+2*B*(1-rsk)+2*(A+B)*(1-rsl)
se = var^.5

sims <- 100000
true_impact = .2 # true impact for 2 years of exposure
target = .2 # target for each cohort

est_effect_y1 <- rnorm(sims, mean = true_impact, sd = se)
est_effect_y2 <- rnorm(sims, mean = true_impact, sd = se)
est_effect_y3 <- rnorm(sims, mean = true_impact, sd = se)
est_effect_y4 <- rnorm(sims, mean = true_impact, sd = se)

payment_y1 <- pmax(0, est_effect_y1/(4*target))
payment_y2 <- pmax(0, est_effect_y2/(4*target))
payment_y3 <- pmax(0, est_effect_y3/(4*target))
payment_y4 <- pmax(0, est_effect_y4/(4*target))

total_payment = payment_y1+payment_y2+payment_y3+payment_y4

mean(total_payment)

[1] 0.9994181

print("90% confidence interval for return")

[1] "90% confidence interval for return"

quantile(total_payment-1, c(.05,.95))

        5%        95% 
-0.3259126  0.3266359 

print("75% confidence interval for return")

[1] "75% confidence interval for return"

quantile(total_payment-1, c(.125,.875))

     12.5%      87.5% 
-0.2288104  0.2280068 

ggplot(tibble(p = total_payment-1), aes(x = p)) + 
  geom_histogram(binwidth = .02) +
  labs(x = "Return", y = "Frequency", caption = "Red line indicates what the investors should be paid", title = "Expected Return w/ Current Evaluation Design")