A Replication of Karlan and List (2007)

Author

Sangho Lee

Published

April 11, 2024

Introduction

Dean Karlan at Yale and John List at the University of Chicago conducted a field experiment to test the effectiveness of different fundraising letters. They sent out 50,000 fundraising letters to potential donors, randomly assigning each letter to one of three treatments: a standard letter, a matching grant letter, or a challenge grant letter. They published the results of this experiment in the American Economic Review in 2007. The article and supporting data are available from the AEA website and from Innovations for Poverty Action as part of Harvard’s Dataverse.

to do: expand on the description of the experiment.

This project seeks to replicate their results.

Data

Description

I will now load the data and preview the first few rows. Since the data is in .dta format, I’m going to use the haven package to import it into this environment. Then, I will display the top 10 rows as a sample.

library(haven)
library(tidyverse)
library(magrittr)
library(kableExtra)
library(reticulate)
data <- haven::read_dta("karlan_list_2007.dta")


# Create a table
data %>% 
  head(10) %>% 
  knitr::kable()
treatment control ratio ratio2 ratio3 size size25 size50 size100 sizeno ask askd1 askd2 askd3 ask1 ask2 ask3 amount gave amountchange hpa ltmedmra freq years year5 mrm2 dormant female couple state50one nonlit cases statecnt stateresponse stateresponset stateresponsec stateresponsetminc perbush close25 red0 blue0 redcty bluecty pwhite pblack page18_39 ave_hh_sz median_hhincome powner psch_atlstba pop_propurban
0 1 0 0 0 0 0 0 0 0 0 0 0 0 55 70 85 0 0 -45 45 0 2 4 0 31 1 0 0 0 5 4 4.5002995 0.0199468 0.0195024 0.0208062 -0.0013039 0.4900000 1 0 1 0 1 0.4464934 0.5277692 0.3175913 2.10 28517 0.4998072 0.3245278 1.0000000
0 1 0 0 0 0 0 0 0 0 0 0 0 0 25 35 50 0 0 -25 25 1 2 3 0 5 0 0 0 0 0 2 2.9822462 0.0260870 0.0278330 0.0224949 0.0053381 0.4646465 0 0 1 1 0 NA NA NA NA NA NA NA NA
1 0 1 0 0 3 0 0 1 0 1 1 0 0 55 70 85 0 0 -50 50 0 3 2 0 6 0 0 0 0 3 1 9.6070213 0.0230482 0.0221589 0.0247435 -0.0025846 0.4081633 0 0 1 0 1 0.9357064 0.0119484 0.2761282 2.48 51175 0.7219406 0.1926679 1.0000000
1 0 1 0 0 4 0 0 0 1 1 1 0 0 55 70 85 0 0 -25 50 1 15 8 1 1 0 0 0 0 1 2 3.2814682 0.0206687 0.0247027 0.0126812 0.0120215 0.4646465 0 0 1 1 0 0.8883309 0.0107604 0.2794118 2.65 79269 0.9204314 0.4121422 1.0000000
1 0 1 0 0 2 0 1 0 0 1 1 0 0 35 45 55 0 0 -15 25 1 42 95 1 24 1 1 0 0 1 1 2.3020148 0.0155979 0.0169713 0.0128866 0.0040847 0.5252525 0 1 0 0 1 0.7590141 0.1274209 0.4423889 1.85 40908 0.4160721 0.4399652 1.0000000
0 1 0 0 0 0 0 0 0 0 0 0 0 0 95 120 145 0 0 -45 90 0 20 10 1 3 0 0 0 0 0 0 0.5306204 0.0187970 0.0264550 0.0000000 0.0264550 0.5102041 1 1 0 1 0 0.8620533 0.0715718 0.3632385 2.92 61779 0.9413386 0.2008403 0.9623450
1 0 1 0 0 1 1 0 0 0 3 0 0 1 125 160 190 0 0 -50 100 0 12 8 1 4 0 1 0 0 0 1 1.1390384 0.0157618 0.0204604 0.0055556 0.0149048 0.6060606 0 1 0 1 0 0.8829117 0.0974081 0.3764462 2.10 54655 0.6958798 0.5545676 1.0000000
1 0 2 1 0 3 0 0 1 0 3 0 0 1 75 95 120 0 0 -65 65 0 13 16 1 4 0 0 0 0 4 3 17.3688412 0.0189503 0.0184932 0.0198814 -0.0013883 0.4444444 0 0 1 0 1 0.9032967 0.0194936 0.2120401 2.47 141527 0.8979058 0.7081225 1.0000000
1 0 2 1 0 4 0 0 0 1 2 0 1 0 250 315 375 0 0 -100 200 0 28 19 1 6 0 0 0 0 1 1 2.3020148 0.0155979 0.0169713 0.0128866 0.0040847 0.5252525 0 1 0 1 0 0.9278508 0.0047386 0.2494650 2.49 37017 0.7631508 0.2058281 0.5990523
1 0 1 0 0 1 1 0 0 0 2 0 1 0 150 190 225 0 0 -125 125 0 4 7 1 35 1 0 0 0 4 3 17.3688412 0.0189503 0.0184932 0.0198814 -0.0013883 0.4444444 0 0 1 0 1 NA NA NA NA NA NA NA NA

I am also going to load the Python packages that I will need to complete the work in this environment.

import pandas as pd
import numpy as np
import pyrsm as rsm
Variable Description
treatment Treatment
control Control
ratio Match ratio
ratio2 2:1 match ratio
ratio3 3:1 match ratio
size Match threshold
size25 $25,000 match threshold
size50 $50,000 match threshold
size100 $100,000 match threshold
sizeno Unstated match threshold
ask Suggested donation amount
askd1 Suggested donation was highest previous contribution
askd2 Suggested donation was 1.25 x highest previous contribution
askd3 Suggested donation was 1.50 x highest previous contribution
ask1 Highest previous contribution (for suggestion)
ask2 1.25 x highest previous contribution (for suggestion)
ask3 1.50 x highest previous contribution (for suggestion)
amount Dollars given
gave Gave anything
amountchange Change in amount given
hpa Highest previous contribution
ltmedmra Small prior donor: last gift was less than median $35
freq Number of prior donations
years Number of years since initial donation
year5 At least 5 years since initial donation
mrm2 Number of months since last donation
dormant Already donated in 2005
female Female
couple Couple
state50one State tag: 1 for one observation of each of 50 states; 0 otherwise
nonlit Nonlitigation
cases Court cases from state in 2004-5 in which organization was involved
statecnt Percent of sample from state
stateresponse Proportion of sample from the state who gave
stateresponset Proportion of treated sample from the state who gave
stateresponsec Proportion of control sample from the state who gave
stateresponsetminc stateresponset - stateresponsec
perbush State vote share for Bush
close25 State vote share for Bush between 47.5% and 52.5%
red0 Red state
blue0 Blue state
redcty Red county
bluecty Blue county
pwhite Proportion white within zip code
pblack Proportion black within zip code
page18_39 Proportion age 18-39 within zip code
ave_hh_sz Average household size within zip code
median_hhincome Median household income within zip code
powner Proportion house owner within zip code
psch_atlstba Proportion who finished college within zip code
pop_propurban Proportion of population urban within zip code

Balance Test

As an ad hoc test of the randomization mechanism, I provide a series of tests that compare aspects of the treatment and control groups to assess whether they are statistically significantly different from one another.

todo: test a few variables other than the key outcome variables (for example, test months since last donation) to see if the treatment and control groups are statistically significantly different at the 95% confidence level. Do each as a t-test and separately as a linear regression, and confirm you get the exact same results from both methods. When doing a t-test, use the formula in the class slides. When doing the linear regression, regress for example mrm2 on treatment and look at the estimated coefficient on the treatment variable. It might be helpful to compare parts of your analysis to Table 1 in the paper. Be sure to comment on your results (hint: why is Table 1 included in the paper).

Experimental Results

Charitable Contribution Made

First, I analyze whether matched donations lead to an increased response rate of making a donation.

todo: make a barplot with two bars. Each bar is the proportion of people who donated. One bar for treatment and one bar for control.

todo: run a t-test between the treatment and control groups on the binary outcome of whether any charitable donation was made. Also run a bivariate linear regression that demonstrates the same finding. (It may help to confirm your calculations match Table 2a Panel A.) Report your statistical results and interpret them in the context of the experiment (e.g., if you found a difference with a small p-value or that was statistically significant at some threshold, what have you learned about human behavior? Use mostly English words, not numbers or stats, to explain your finding.)

todo: run a probit regression where the outcome variable is whether any charitable donation was made and the explanatory variable is assignment to treatment or control. Confirm that your results replicate Table 3 column 1 in the paper.

Differences between Match Rates

Next, I assess the effectiveness of different sizes of matched donations on the response rate.

todo: Use a series of t-tests to test whether the size of the match ratio has an effect on whether people donate or not. For example, does the 2:1 match rate lead increase the likelihood that someone donates as compared to the 1:1 match rate? Do your results support the “figures suggest” comment the authors make on page 8?

todo: Assess the same issue using a regression. Specifically, create the variable ratio1 then regress gave on ratio1, ratio2, and ratio3 (or alternatively, regress gave on the categorical variable ratio). Interpret the coefficients and their statistical precision.

todo: Calculate the response rate difference between the 1:1 and 2:1 match ratios and the 2:1 and 3:1 ratios. Do this directly from the data, and do it by computing the differences in the fitted coefficients of the previous regression. what do you conclude regarding the effectiveness of different sizes of matched donations?

Size of Charitable Contribution

In this subsection, I analyze the effect of the size of matched donation on the size of the charitable contribution.

todo: Calculate a t-test or run a bivariate linear regression of the donation amount on the treatment status. What do we learn from doing this analysis?

todo: now limit the data to just people who made a donation and repeat the previous analysis. This regression allows you to analyze how much respondents donate conditional on donating some positive amount. Interpret the regression coefficients – what did we learn? Does the treatment coefficient have a causal interpretation?

todo: Make two plot: one for the treatment group and one for the control. Each plot should be a histogram of the donation amounts only among people who donated. Add a red vertical bar or some other annotation to indicate the sample average for each plot.

Simulation Experiment

As a reminder of how the t-statistic “works,” in this section I use simulation to demonstrate the Law of Large Numbers and the Central Limit Theorem.

Suppose the true distribution of respondents who do not get a charitable donation match is Bernoulli with probability p=0.018 that a donation is made.

Further suppose that the true distribution of respondents who do get a charitable donation match of any size is Bernoulli with probability p=0.022 that a donation is made.

Law of Large Numbers

to do: Make a plot like those on slide 43 from our first class and explain the plot to the reader. To do this, you will simulate 100,00 draws from the control distribution and 10,000 draws from the treatment distribution. You’ll then calculate a vector of 10,000 differences, and then you’ll plot the cumulative average of that vector of differences. Comment on whether the cumulative average approaches the true difference in means.

Central Limit Theorem

to do: Make 4 histograms like those on slide 44 from our first class at sample sizes 50, 200, 500, and 1000 and explain these plots to the reader. To do this for a sample size of e.g. 50, take 50 draws from each of the control and treatment distributions, and calculate the average difference between those draws. Then repeat that process 999 more times so that you have 1000 averages. Plot the histogram of those averages. Comment on whether zero is in the “middle” of the distribution or whether it’s in the “tail.”