1 Overview and results.

This notebook contains preliminary power calculations for a potential quantitative impact evaluation of the n-Frnds service among warungs in Indonesia. These calculations suggest that with a sample size of 200 warungs (100 treatment + 100 control) would allow us to detect a 21% increase in profits and a sample size of 800 warungs (400 treatment + 400 control) would allow us to detect a 10% increase in profits. These calculations are very sensitive to our assumption regarding warung profit which we have relatively little information about and thus should be considered rough calculations at this stage.

2 Approach and code

We assume that the primary outcome of interest is warung profit. To estimate power, we use the standard formula for power for an individual-level RCT (see appendix) but assume that the sample size is only half of the actual size to correct for the non-experimental design. This arbitrary inflation factor reflects the potential loss in power due to correlation between our covariates and the treatment indicator. See here for more info on this approach.

We assume standard values for \(\alpha\) and \(\beta\) of .05 and .8 respectively and that we will use a two-sided test when conducting the analysis. We estimate the variance of the log of warung profits using data from Olahan and ??? (See code below.)

# import packages
library(tidyverse)
## Warning: package 'tidyverse' was built under R version 4.0.5
## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --
## v ggplot2 3.3.3     v purrr   0.3.4
## v tibble  3.1.2     v dplyr   1.0.6
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   1.4.0     v forcats 0.5.1
## Warning: package 'tidyr' was built under R version 4.0.5
## Warning: package 'dplyr' was built under R version 4.0.5
## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
# Calculate the variance of log profit from the two studies
# Data from the Olahan study, in million IDR
profit = c(2,3,4,5,6)
log_profit <- log(profit)
num = c(6, 8, 9, 7, 3)
mean_log_profit <- weighted.mean(log_profit, num)
squared_terms <- (log_profit-mean_log_profit)^2
olahan_variance <- weighted.mean(squared_terms, num) # We should be dividing by n-1 rather than n but ignoring that as it doesn't make a big difference.

# Data from Medan, in million IDR
medan_data <- c(3.5, 2.7, 3.9, 3.5, 4.0, 6.5, 6.0, 6.7, 5.8, 4.0, 5.0, 2.0, 2.0, 7.0, 5.5, 3.5, 7.0, 5.8, 5.0, 5.0)
medan_variance <- var(log(medan_data))
mean_variance <- (olahan_variance+medan_variance)/2


# Set input values
alpha <- .05
beta <- .8
sigma_y <- mean_variance
inflate <- 2 # Inflation factor which corrects for the fact that this is not an RCT and thus covariates may be correlated with treatment status.

# Create function to calculate factor from N
factor <- function(N) {
  factor <- qt(1-alpha/2, N) + qt(beta, N)
  return(factor)
}

# Create function to calculate mdi from N
mdi <- function(N) {
  mdi <- factor(N)*(2*sigma_y/(N/inflate))^.5
  return(mdi)
}

# Plot 
ggplot() + xlim(100, 400) + geom_function(fun = mdi) +
  labs(x = "Sample size for either treatment or control", y = "MDI as % increase in profit", title = "MDI vs. sample size")

3 Appendix - Formulas

The minimum detectable impact (MDI) – i.e. the smallest size impact that we would detect with probability \(\beta\) – for a randomized evaluation with treatment randomized at the individual level and one round of data collection is:

\[ MDI = (t_{1-\alpha/2, N}+t_{\beta, N})*\sqrt{\frac{2*\sigma^2_y}{N}} \] Where \(t_{x,\nu}\) is the student t distribution quantile function for quantile x with degrees of freedom \(\nu\), \(\sigma^2_y\) is the variance of the outcome measure and N is the sample size of the treatment or control group (which are assumed to be equal).