library(haven)
library(tidyverse)
library(magrittr)
library(scales)A Replication of Karlan and List (2007)
Introduction
Dean Karlan at Yale and John List at the University of Chicago conducted a field experiment to test the effectiveness of different fundraising letters. They sent out 50,000 fundraising letters to potential donors, randomly assigning each letter to one of three treatments: a standard letter, a matching grant letter, or a challenge grant letter. They published the results of this experiment in the American Economic Review in 2007. The article and supporting data are available from the AEA website and from Innovations for Poverty Action as part of Harvard’s Dataverse.
to do: expand on the description of the experiment.
This project seeks to replicate their results.
Data
Description
I will now load the data and preview the first few rows. Since the data is in .dta format, I’m going to use the haven package to import it into this environment. Then, I will display the top 10 rows as a sample.
data <- haven::read_dta("karlan_list_2007.dta")
# Create a table
data %>%
head(10) %>%
knitr::kable() | treatment | control | ratio | ratio2 | ratio3 | size | size25 | size50 | size100 | sizeno | ask | askd1 | askd2 | askd3 | ask1 | ask2 | ask3 | amount | gave | amountchange | hpa | ltmedmra | freq | years | year5 | mrm2 | dormant | female | couple | state50one | nonlit | cases | statecnt | stateresponse | stateresponset | stateresponsec | stateresponsetminc | perbush | close25 | red0 | blue0 | redcty | bluecty | pwhite | pblack | page18_39 | ave_hh_sz | median_hhincome | powner | psch_atlstba | pop_propurban |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 55 | 70 | 85 | 0 | 0 | -45 | 45 | 0 | 2 | 4 | 0 | 31 | 1 | 0 | 0 | 0 | 5 | 4 | 4.5002995 | 0.0199468 | 0.0195024 | 0.0208062 | -0.0013039 | 0.4900000 | 1 | 0 | 1 | 0 | 1 | 0.4464934 | 0.5277692 | 0.3175913 | 2.10 | 28517 | 0.4998072 | 0.3245278 | 1.0000000 |
| 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 25 | 35 | 50 | 0 | 0 | -25 | 25 | 1 | 2 | 3 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 2 | 2.9822462 | 0.0260870 | 0.0278330 | 0.0224949 | 0.0053381 | 0.4646465 | 0 | 0 | 1 | 1 | 0 | NA | NA | NA | NA | NA | NA | NA | NA |
| 1 | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 55 | 70 | 85 | 0 | 0 | -50 | 50 | 0 | 3 | 2 | 0 | 6 | 0 | 0 | 0 | 0 | 3 | 1 | 9.6070213 | 0.0230482 | 0.0221589 | 0.0247435 | -0.0025846 | 0.4081633 | 0 | 0 | 1 | 0 | 1 | 0.9357064 | 0.0119484 | 0.2761282 | 2.48 | 51175 | 0.7219406 | 0.1926679 | 1.0000000 |
| 1 | 0 | 1 | 0 | 0 | 4 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 55 | 70 | 85 | 0 | 0 | -25 | 50 | 1 | 15 | 8 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 2 | 3.2814682 | 0.0206687 | 0.0247027 | 0.0126812 | 0.0120215 | 0.4646465 | 0 | 0 | 1 | 1 | 0 | 0.8883309 | 0.0107604 | 0.2794118 | 2.65 | 79269 | 0.9204314 | 0.4121422 | 1.0000000 |
| 1 | 0 | 1 | 0 | 0 | 2 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 35 | 45 | 55 | 0 | 0 | -15 | 25 | 1 | 42 | 95 | 1 | 24 | 1 | 1 | 0 | 0 | 1 | 1 | 2.3020148 | 0.0155979 | 0.0169713 | 0.0128866 | 0.0040847 | 0.5252525 | 0 | 1 | 0 | 0 | 1 | 0.7590141 | 0.1274209 | 0.4423889 | 1.85 | 40908 | 0.4160721 | 0.4399652 | 1.0000000 |
| 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 95 | 120 | 145 | 0 | 0 | -45 | 90 | 0 | 20 | 10 | 1 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0.5306204 | 0.0187970 | 0.0264550 | 0.0000000 | 0.0264550 | 0.5102041 | 1 | 1 | 0 | 1 | 0 | 0.8620533 | 0.0715718 | 0.3632385 | 2.92 | 61779 | 0.9413386 | 0.2008403 | 0.9623450 |
| 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 3 | 0 | 0 | 1 | 125 | 160 | 190 | 0 | 0 | -50 | 100 | 0 | 12 | 8 | 1 | 4 | 0 | 1 | 0 | 0 | 0 | 1 | 1.1390384 | 0.0157618 | 0.0204604 | 0.0055556 | 0.0149048 | 0.6060606 | 0 | 1 | 0 | 1 | 0 | 0.8829117 | 0.0974081 | 0.3764462 | 2.10 | 54655 | 0.6958798 | 0.5545676 | 1.0000000 |
| 1 | 0 | 2 | 1 | 0 | 3 | 0 | 0 | 1 | 0 | 3 | 0 | 0 | 1 | 75 | 95 | 120 | 0 | 0 | -65 | 65 | 0 | 13 | 16 | 1 | 4 | 0 | 0 | 0 | 0 | 4 | 3 | 17.3688412 | 0.0189503 | 0.0184932 | 0.0198814 | -0.0013883 | 0.4444444 | 0 | 0 | 1 | 0 | 1 | 0.9032967 | 0.0194936 | 0.2120401 | 2.47 | 141527 | 0.8979058 | 0.7081225 | 1.0000000 |
| 1 | 0 | 2 | 1 | 0 | 4 | 0 | 0 | 0 | 1 | 2 | 0 | 1 | 0 | 250 | 315 | 375 | 0 | 0 | -100 | 200 | 0 | 28 | 19 | 1 | 6 | 0 | 0 | 0 | 0 | 1 | 1 | 2.3020148 | 0.0155979 | 0.0169713 | 0.0128866 | 0.0040847 | 0.5252525 | 0 | 1 | 0 | 1 | 0 | 0.9278508 | 0.0047386 | 0.2494650 | 2.49 | 37017 | 0.7631508 | 0.2058281 | 0.5990523 |
| 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 2 | 0 | 1 | 0 | 150 | 190 | 225 | 0 | 0 | -125 | 125 | 0 | 4 | 7 | 1 | 35 | 1 | 0 | 0 | 0 | 4 | 3 | 17.3688412 | 0.0189503 | 0.0184932 | 0.0198814 | -0.0013883 | 0.4444444 | 0 | 0 | 1 | 0 | 1 | NA | NA | NA | NA | NA | NA | NA | NA |
I am also going to load the Python packages that I will need to complete the work in this environment.
import pandas as pd
import numpy as np
import pyrsm as rsm
import matplotlib.pyplot as plt
import statsmodels.api as smBalance Test
As an ad hoc test of the randomization mechanism, I provide a series of tests that compare aspects of the treatment and control groups to assess whether they are statistically significantly different from one another.
I’m going to conduct t-tests and linear regressions on several variables, such as months since last donation, to determine if there are statistically significant differences between the treatment and control groups at the 95% confidence level. For the t-tests, I will use the t.test function in R to analyze the data. For the linear regressions, I will utilize the pyrsm package in Python to examine the impact of the treatment by regressing variables like months since last donation against the treatment group and observing the coefficient of the treatment variable. This approach will allow me to confirm that both methods yield consistent results.
- mrm2 variable
t.test(mrm2 ~ treatment, data = data, conf.level = 0.95)
Welch Two Sample t-test
data: mrm2 by treatment
t = -0.11953, df = 33394, p-value = 0.9049
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.2381015 0.2107298
sample estimates:
mean in group 0 mean in group 1
12.99814 13.01183 lm = rsm.model.regress(
data = data,
rvar = "mrm2",
evar = "treatment"
)
lm.summary()Linear regression (OLS)
Data : Not provided
Response variable : mrm2
Explanatory variables: treatment
Null hyp.: the effect of x on mrm2 is zero
Alt. hyp.: the effect of x on mrm2 is not zero
coefficient std.error t.value p.value
Intercept 13.00 0.094 138.979 < .001 ***
treatment 0.01 0.115 0.119 0.905
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-squared: 0.0, Adjusted R-squared: -0.0
F-statistic: 0.014 df(1, 50080), p.value 0.905
Nr obs: 50,082- P-value: Both methods yield a p-value of approximately 0.905
- Interpretation: There are no significant differences between the treatment and control groups for “mrm2” (months since last donation). This suggests effective randomization, indicating that any observed treatment effects are likely due to the treatment itself rather than pre-existing group differences.
- pwhite variable
t.test(pwhite ~ treatment, data = data, conf.level = 0.95)
Welch Two Sample t-test
data: pwhite by treatment
t = 0.55912, df = 31876, p-value = 0.5761
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.002287046 0.004112602
sample estimates:
mean in group 0 mean in group 1
0.8202078 0.8192950 lm = rsm.model.regress(
data = data,
rvar = "pwhite",
evar = "treatment"
)
lm.summary()Linear regression (OLS)
Data : Not provided
Response variable : pwhite
Explanatory variables: treatment
Null hyp.: the effect of x on pwhite is zero
Alt. hyp.: the effect of x on pwhite is not zero
coefficient std.error t.value p.value
Intercept 0.82 0.001 616.281 < .001 ***
treatment -0.00 0.002 -0.560 0.575
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-squared: 0.0, Adjusted R-squared: -0.0
F-statistic: 0.314 df(1, 48215), p.value 0.575
Nr obs: 48,217- P-value: Both the t-test and linear regression show a p-value around 0.575.
- Interpretation: There is no significant difference in the proportion of white individuals between the treatment and control groups, indicating that the randomization was successful in creating comparable groups regarding racial composition.
- female variable
t.test(female ~ treatment, data = data, conf.level = 0.95)
Welch Two Sample t-test
data: female by treatment
t = 1.7535, df = 32451, p-value = 0.07952
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.0008888548 0.0159826921
sample estimates:
mean in group 0 mean in group 1
0.2826978 0.2751509 lm = rsm.model.regress(
data = data,
rvar = "female",
evar = "treatment"
)
lm.summary()Linear regression (OLS)
Data : Not provided
Response variable : female
Explanatory variables: treatment
Null hyp.: the effect of x on female is zero
Alt. hyp.: the effect of x on female is not zero
coefficient std.error t.value p.value
Intercept 0.28 0.004 80.688 < .001 ***
treatment -0.01 0.004 -1.758 0.079 .
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-squared: 0.0, Adjusted R-squared: 0.0
F-statistic: 3.092 df(1, 48970), p.value 0.079
Nr obs: 48,972- P-value: The t-test has a p-value of 0.07952 and the linear regression also shows a p-value of 0.079.
- Interpretation: The results suggest a marginal difference in the proportion of females between the treatment and control groups that approaches but does not reach conventional levels of statistical significance (p < 0.05).
Understanding the T-value and P-value
T-value (1.7535): The t-value measures the ratio of the difference between the group means to the standard error of the difference. A t-value of 1.7535 indicates that the difference between the group means (for the proportion of females in treatment versus control) is 1.7535 times the standard error away from zero. The sign (+) of the t-value suggests that the mean proportion of females in the treatment group is slightly lower than in the control group, given the negative coefficient in the regression output.
P-value (0.07952): The p-value tells us the probability of observing a t-value as extreme as 1.7535 (or more) if the null hypothesis were true which was there is no difference in the proportion of females between groups.
Experimental Results
Charitable Contribution Made
First, I analyze whether matched donations lead to an increased response rate of making a donation.
I’m going to create a bar plot that visualizes the proportion of people who donated, with two separate bars representing the treatment and control groups. This will allow for a clear comparison of the donation rates between the two groups
# Calculate the proportion of donations in the treatment and control groups using dplyr
plot_data <- data %>%
group_by(treatment) %>%
summarise(Proportion_Donated = mean(gave == 1, na.rm = TRUE)) %>%
mutate(Group = ifelse(treatment == 1, "Treatment", "Control")) %>%
select(Group, Proportion_Donated)
# Generate the bar plot using ggplot2
ggplot(plot_data, aes(x = Group, y = Proportion_Donated, fill = Group)) +
geom_bar(stat = "identity", position = "dodge", width = 0.5) +
geom_text(aes(label=scales::percent(Proportion_Donated)), vjust=-0.3, size=5) +
scale_y_continuous(labels = scales::percent_format()) +
labs(y = "Proportion of Donations", title = "Proportion of People Who Donated by Group") +
theme_classic() +
theme(
plot.title = element_text(hjust = 0.5, size = 22),
axis.title.y = element_text(size = 20),
axis.text.x = element_text(size = 20),
axis.title.x = element_blank(),
legend.position = "none"
) +
scale_fill_manual(values = c("Treatment" = "#FF5733", "Control" = "#33C1FF"))
I’m going to perform a t-test to compare the treatment and control groups based on whether a charitable donation was made. Additionally, I will conduct a linear regression to analyze the same data. After obtaining the statistical outcomes, I will interpret these results in the context of the experiment. Specifically, if the results show a statistically significant difference, I’ll discuss what these findings suggest about human behavior towards charitable giving, focusing more on qualitative insights rather than numerical data.
t.test(gave ~ treatment, data = data, var.equal = FALSE)
Welch Two Sample t-test
data: gave by treatment
t = -3.2095, df = 36577, p-value = 0.001331
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-0.006733310 -0.001627399
sample estimates:
mean in group 0 mean in group 1
0.01785821 0.02203857 lm = rsm.model.regress(
data = data,
rvar = "gave",
evar = "treatment"
)
lm.summary()Linear regression (OLS)
Data : Not provided
Response variable : gave
Explanatory variables: treatment
Null hyp.: the effect of x on gave is zero
Alt. hyp.: the effect of x on gave is not zero
coefficient std.error t.value p.value
Intercept 0.02 0.001 16.225 < .001 ***
treatment 0.00 0.001 3.101 0.002 **
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-squared: 0.0, Adjusted R-squared: 0.0
F-statistic: 9.618 df(1, 50081), p.value 0.002
Nr obs: 50,083The t-test and linear regression both show that the treatment group had a slightly higher donation rate (2.2%) compared to the control group (1.8%). This statistically significant difference (p-value = 0.0013 for the t-test) suggests that even a modest intervention can effectively increase the likelihood of donating. This demonstrates how small nudges, such as targeted reminders, can influence human behavior and enhance donation rates, which is crucial for organizations aiming to improve their fundraising outcomes.
I’m going to conduct a probit regression where the outcome variable is whether a charitable donation was made, and the explanatory variable is whether individuals were assigned to the treatment or control group.
# Create the dependent and independent variables
X = data[['treatment']] # independent variable
X = sm.add_constant(X) # adding a constant
y = data['gave'] # dependent variable
# Fit a probit model
probit_model = sm.Probit(y, X).fit()Optimization terminated successfully.
Current function value: 0.100443
Iterations 7# Print the summary of the model
print(probit_model.summary()) Probit Regression Results
==============================================================================
Dep. Variable: gave No. Observations: 50083
Model: Probit Df Residuals: 50081
Method: MLE Df Model: 1
Date: Fri, 12 Apr 2024 Pseudo R-squ.: 0.0009783
Time: 10:44:46 Log-Likelihood: -5030.5
converged: True LL-Null: -5035.4
Covariance Type: nonrobust LLR p-value: 0.001696
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
const -2.1001 0.023 -90.073 0.000 -2.146 -2.054
treatment 0.0868 0.028 3.113 0.002 0.032 0.141
==============================================================================- Probit Regression Explained: Probit regression is a type of regression used in statistics to analyze binary (two outcome) variables. In this context, it models the probability that someone made a charitable donation based on whether they were in the treatment group or not.
Key Elements of the Output:
Dependent Variable (gave): This is what we’re trying to predict: whether or not someone gave a donation (1 if yes, 0 if no).
Number of Observations (50,083): This is the total number of data points used in the model.
Model (Probit): The type of model used, which in this case uses a Probit link function. This function maps any input from minus infinity to infinity to a 0 to 1 range, fitting the binary nature of the dependent variable.
MLE (Maximum Likelihood Estimation): This is the method used to fit the model, optimizing to find the parameter values that maximize the likelihood of the data given the model.
Coefficients:
- const (-2.1001): This is the intercept, or baseline log odds of donating when all predictors are 0 (i.e., not in the treatment group). A negative value indicates a low baseline probability of donation.
- treatment (0.0868): This coefficient shows the effect of being in the treatment group. It suggests that being in the treatment group increases the likelihood of donating compared to the control group.
Standard Error (std err): These values (0.023 for const and 0.028 for treatment) measure the average distance that the estimated values (coefficients) fall from the actual value. Smaller standard errors suggest more precise estimates.
Z-value: This is the test statistic used to determine whether to reject the null hypothesis (that the coefficient is zero, implying no effect). For the treatment, a z-value of 3.113 indicates a statistically significant effect.
P-value: For treatment (0.002), this value is very small, indicating strong evidence against the null hypothesis. It suggests that the treatment effect is statistically significant and not due to random chance.
Confidence Interval: For treatment ([0.032, 0.141]), this range gives us a 95% confidence level that the true coefficient falls within this interval. It confirms that the effect is positive and fairly precise.
Differences between Match Rates
Next, I assess the effectiveness of different sizes of matched donations on the response rate.
todo: Use a series of t-tests to test whether the size of the match ratio has an effect on whether people donate or not. For example, does the 2:1 match rate lead increase the likelihood that someone donates as compared to the 1:1 match rate? Do your results support the “figures suggest” comment the authors make on page 8?
todo: Assess the same issue using a regression. Specifically, create the variable ratio1 then regress gave on ratio1, ratio2, and ratio3 (or alternatively, regress gave on the categorical variable ratio). Interpret the coefficients and their statistical precision.
todo: Calculate the response rate difference between the 1:1 and 2:1 match ratios and the 2:1 and 3:1 ratios. Do this directly from the data, and do it by computing the differences in the fitted coefficients of the previous regression. what do you conclude regarding the effectiveness of different sizes of matched donations?
Size of Charitable Contribution
In this subsection, I analyze the effect of the size of matched donation on the size of the charitable contribution.
todo: Calculate a t-test or run a bivariate linear regression of the donation amount on the treatment status. What do we learn from doing this analysis?
todo: now limit the data to just people who made a donation and repeat the previous analysis. This regression allows you to analyze how much respondents donate conditional on donating some positive amount. Interpret the regression coefficients – what did we learn? Does the treatment coefficient have a causal interpretation?
todo: Make two plot: one for the treatment group and one for the control. Each plot should be a histogram of the donation amounts only among people who donated. Add a red vertical bar or some other annotation to indicate the sample average for each plot.
Simulation Experiment
As a reminder of how the t-statistic “works,” in this section I use simulation to demonstrate the Law of Large Numbers and the Central Limit Theorem.
Suppose the true distribution of respondents who do not get a charitable donation match is Bernoulli with probability p=0.018 that a donation is made.
Further suppose that the true distribution of respondents who do get a charitable donation match of any size is Bernoulli with probability p=0.022 that a donation is made.
Law of Large Numbers
to do: Make a plot like those on slide 43 from our first class and explain the plot to the reader. To do this, you will simulate 100,00 draws from the control distribution and 10,000 draws from the treatment distribution. You’ll then calculate a vector of 10,000 differences, and then you’ll plot the cumulative average of that vector of differences. Comment on whether the cumulative average approaches the true difference in means.
Central Limit Theorem
to do: Make 4 histograms like those on slide 44 from our first class at sample sizes 50, 200, 500, and 1000 and explain these plots to the reader. To do this for a sample size of e.g. 50, take 50 draws from each of the control and treatment distributions, and calculate the average difference between those draws. Then repeat that process 999 more times so that you have 1000 averages. Plot the histogram of those averages. Comment on whether zero is in the “middle” of the distribution or whether it’s in the “tail.”