The LL data set in the cem package (type data(LL) to get it) contains data from from an observational study constructed by LaLonde (1986) based on a randomized experiment that evaluated the effect on earnings of a job training program called National Supported Work. The constructed observational study was formed by replacing the randomized control group with a comparison group formed using data from two natioanl public use surveys: the Current Population Survey (CPS) and the Panel Study in Income Dynamics. The dependent variable is earnings in 1978 (re78). The treatment indicator is treated.
1. On the entire data set, compute the difference-in-means of 1978 earnings between the treatment and control groups with the t.test() command. What does the result suggest about the effect of the NSW program? Is the effect statistically significant?.
## Load Data and Libraries ##
library(cem)
## Loading required package: nlme
## Loading required package: lattice
## Loading required package: randomForest
## randomForest 4.6-7
## Type rfNews() to see new features/changes/bug fixes.
## Loading required package: tcltk
## Loading required package: combinat
##
## Attaching package: 'combinat'
##
## The following object is masked from 'package:utils':
##
## combn
##
##
## How to use CEM? Type vignette("cem")
library(Zelig)
## Loading required package: MASS
## Loading required package: boot
##
## Attaching package: 'boot'
##
## The following object is masked from 'package:lattice':
##
## melanoma
## ##
## ## Zelig (Version 3.5.4, built: 2010-01-20)
## ## Please refer to http://gking.harvard.edu/zelig for full
## ## documentation or help.zelig() for help with commands and
## ## models supported by Zelig.
## ##
##
## ## Zelig project citations:
## ## Kosuke Imai, Gary King, and Olivia Lau. (2009).
## ## ``Zelig: Everyone's Statistical Software,''
## ## http://gking.harvard.edu/zelig.
## ## and
## ## Kosuke Imai, Gary King, and Olivia Lau. (2008).
## ## ``Toward A Common Framework for Statistical Analysis
## ## and Development,'' Journal of Computational and
## ## Graphical Statistics, Vol. 17, No. 4 (December)
## ## pp. 892-913.
##
## ## To cite individual Zelig models, please use the citation format printed with
## ## each model run and in the documentation.
## ##
library(MatchIt)
##
## Attaching package: 'MatchIt'
##
## The following object is masked from 'package:Zelig':
##
## user.prompt
data(LL)
str(LL)
## 'data.frame': 722 obs. of 12 variables:
## $ treated : num 1 1 0 1 0 0 0 0 1 0 ...
## $ age : int 33 20 39 49 26 38 28 27 33 44 ...
## $ education: int 12 12 12 8 8 10 12 12 12 9 ...
## $ black : int 0 0 1 0 0 1 0 1 1 1 ...
## $ married : int 1 1 1 1 1 1 0 1 1 1 ...
## $ nodegree : int 0 0 0 1 1 1 0 0 0 1 ...
## $ re74 : num 0 8644 19785 9715 37212 ...
## $ re75 : num 0 8644 6608 7286 36941 ...
## $ re78 : num 12418 11657 499 16717 30248 ...
## $ hispanic : int 0 0 0 0 0 0 0 0 0 0 ...
## $ u74 : num 1 0 0 0 0 0 1 1 0 0 ...
## $ u75 : num 1 0 0 0 0 0 0 0 0 0 ...
## Compute Difference-in-Means ##
t.test(re78 ~ treated, data = LL) # Null hypothesis is that the difference between the mean of the treatment group and the mean of the control group is zero
##
## Welch Two Sample t-test
##
## data: re78 by treated
## t = -1.815, df = 557.1, p-value = 0.06999
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -1845.25 72.64
## sample estimates:
## mean in group 0 mean in group 1
## 5090 5976
Because the p-value is statistically insignificant at the 0.01-level, we cannot reject the null hypothesis that there is no difference between the mean of the treatment and control groups. Therefore it seems that the effect of the NSW on 1978 earning is insignificant.
2. Match the data based on the other variables in the data. Do this using Mahalanobis distance (nearest neighbor), propensity score (nearest neighbor), and CEM.
# Estimate model on raw data (before matching)
m.raw <- zelig(re78 ~ treated + age + education + black + married + nodegree +
hispanic, model = "normal", data = LL, cite = FALSE)
# Imbalance of the raw data
imbalance.raw <- imbalance(group = LL$treated, data = LL, drop = c("treated",
"re74", "re75", "re78", "u74", "u75"))
## Match the Data ## Using Mahalanobis distance (nearest neighbor)
match.maha <- matchit(treated ~ age + education + black + married + nodegree +
hispanic, method = "nearest", distance = "mahalanobis", data = LL)
data.maha <- match.data(match.maha) # Extract data from matching
imbalance.maha <- imbalance(group = data.maha$treated, data = data.maha, drop = c("treated",
"re74", "re75", "re78", "u74", "u75", "weights", "distance"))
# Using propensity score (nearest neighbor)
match.ps <- matchit(treated ~ age + education + black + married + nodegree +
hispanic, method = "nearest", data = LL)
data.ps <- match.data(match.ps)
imbalance.ps <- imbalance(group = data.ps$treated, data = data.ps, drop = c("treated",
"re74", "re75", "re78", "u74", "u75", "weights", "distance"))
# Using coarsened exact matching (CEM)
match.cem <- matchit(treated ~ age + education + black + married + nodegree +
hispanic, method = "cem", data = LL)
##
## Using 'treat'='1' as baseline group
data.cem <- match.data(match.cem)
imbalance.cem <- imbalance(group = data.cem$treated, data = data.cem, drop = c("treated",
"re74", "re75", "re78", "u74", "u75", "weights", "distance"))
3. Comment on the balance of the data before and after each matching method: compare means, variances, and covariances of variables before and after matching, or compute the \( L_1 \) statistic with CEM. Also comment on how many observations were dropped with each method. Which method seems to have done the best job in creating balance?
# Summaries for match methods
summary(match.maha)
##
## Call:
## matchit(formula = treated ~ age + education + black + married +
## nodegree + hispanic, data = LL, method = "nearest", distance = "mahalanobis")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control Mean Diff eQQ Med
## age 24.626 24.447 6.590 0.179 0
## education 10.380 10.188 1.619 0.192 0
## black 0.801 0.800 0.400 0.001 0
## married 0.168 0.158 0.365 0.011 0
## nodegree 0.731 0.814 0.389 -0.083 0
## hispanic 0.094 0.113 0.317 -0.019 0
## eQQ Mean eQQ Max
## age 0.471 6
## education 0.246 2
## black 0.003 1
## married 0.010 1
## nodegree 0.081 1
## hispanic 0.017 1
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control Mean Diff eQQ Med
## age 24.626 24.411 6.218 0.215 0
## education 10.380 10.354 1.609 0.027 0
## black 0.801 0.801 0.400 0.000 0
## married 0.168 0.168 0.375 0.000 0
## nodegree 0.731 0.758 0.429 -0.027 0
## hispanic 0.094 0.094 0.293 0.000 0
## eQQ Mean eQQ Max
## age 0.532 6
## education 0.155 2
## black 0.000 0
## married 0.000 0
## nodegree 0.027 1
## hispanic 0.000 0
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## age -20.25 0 -12.86 0
## education 85.99 0 36.99 0
## black 100.00 0 100.00 100
## married 100.00 0 100.00 100
## nodegree 67.73 0 66.67 0
## hispanic 100.00 0 100.00 100
##
## Sample sizes:
## Control Treated
## All 425 297
## Matched 297 297
## Unmatched 128 0
## Discarded 0 0
summary(match.ps)
##
## Call:
## matchit(formula = treated ~ age + education + black + married +
## nodegree + hispanic, data = LL, method = "nearest")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control Mean Diff eQQ Med
## distance 0.418 0.407 0.049 0.011 0.005
## age 24.626 24.447 6.590 0.179 0.000
## education 10.380 10.188 1.619 0.192 0.000
## black 0.801 0.800 0.400 0.001 0.000
## married 0.168 0.158 0.365 0.011 0.000
## nodegree 0.731 0.814 0.389 -0.083 0.000
## hispanic 0.094 0.113 0.317 -0.019 0.000
## eQQ Mean eQQ Max
## distance 0.011 0.085
## age 0.471 6.000
## education 0.246 2.000
## black 0.003 1.000
## married 0.010 1.000
## nodegree 0.081 1.000
## hispanic 0.017 1.000
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control Mean Diff eQQ Med
## distance 0.418 0.417 0.054 0.001 0
## age 24.626 24.283 6.355 0.343 0
## education 10.380 10.337 1.728 0.044 0
## black 0.801 0.781 0.414 0.020 0
## married 0.168 0.165 0.372 0.003 0
## nodegree 0.731 0.741 0.439 -0.010 0
## hispanic 0.094 0.104 0.306 -0.010 0
## eQQ Mean eQQ Max
## distance 0.001 0.04
## age 0.586 6.00
## education 0.098 2.00
## black 0.020 1.00
## married 0.003 1.00
## nodegree 0.010 1.00
## hispanic 0.010 1.00
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 89.49 96.06 87.42 53.17
## age -91.64 0.00 -24.29 0.00
## education 77.23 0.00 60.27 0.00
## black -1400.00 0.00 -500.00 0.00
## married 68.54 0.00 66.67 0.00
## nodegree 87.90 0.00 87.50 0.00
## hispanic 45.88 0.00 40.00 0.00
##
## Sample sizes:
## Control Treated
## All 425 297
## Matched 297 297
## Unmatched 128 0
## Discarded 0 0
summary(match.cem)
##
## Call:
## matchit(formula = treated ~ age + education + black + married +
## nodegree + hispanic, data = LL, method = "cem")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control Mean Diff eQQ Med
## distance 0.418 0.407 0.049 0.011 0.005
## age 24.626 24.447 6.590 0.179 0.000
## education 10.380 10.188 1.619 0.192 0.000
## black 0.801 0.800 0.400 0.001 0.000
## married 0.168 0.158 0.365 0.011 0.000
## nodegree 0.731 0.814 0.389 -0.083 0.000
## hispanic 0.094 0.113 0.317 -0.019 0.000
## eQQ Mean eQQ Max
## distance 0.011 0.085
## age 0.471 6.000
## education 0.246 2.000
## black 0.003 1.000
## married 0.010 1.000
## nodegree 0.081 1.000
## hispanic 0.017 1.000
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control Mean Diff eQQ Med
## distance 0.414 0.413 0.053 0.000 0.003
## age 23.235 23.396 5.047 -0.161 0.000
## education 10.396 10.392 1.480 0.004 0.000
## black 0.839 0.839 0.368 0.000 0.000
## married 0.122 0.122 0.327 0.000 0.000
## nodegree 0.761 0.761 0.427 0.000 0.000
## hispanic 0.078 0.078 0.269 0.000 0.000
## eQQ Mean eQQ Max
## distance 0.010 0.095
## age 0.431 2.000
## education 0.176 1.000
## black 0.008 1.000
## married 0.000 0.000
## nodegree 0.075 1.000
## hispanic 0.004 1.000
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 99.62 38.28 10.034 -11.91
## age 10.39 0.00 8.487 66.67
## education 97.76 0.00 28.203 50.00
## black 100.00 0.00 -132.941 0.00
## married 100.00 0.00 100.000 100.00
## nodegree 100.00 0.00 7.794 0.00
## hispanic 100.00 0.00 76.706 0.00
##
## Sample sizes:
## Control Treated
## All 425 297
## Matched 360 255
## Unmatched 65 42
## Discarded 0 0
# L1 statistics for methods
imbalance.raw
##
## Multivariate Imbalance Measure: L1=0.431
## Percentage of local common support: LCS=32.6%
##
## Univariate Imbalance Measures:
##
## statistic type L1 min 25% 50% 75% max
## age 0.179204 (diff) 0.004706 0 1 0 -1 -6
## education 0.192236 (diff) 0.098118 1 0 1 1 2
## black 0.001347 (diff) 0.001347 0 0 0 0 0
## married 0.010703 (diff) 0.010703 0 0 0 0 0
## nodegree -0.083478 (diff) 0.083478 0 -1 0 0 0
## hispanic -0.018665 (diff) 0.018665 0 0 0 0 0
imbalance.maha
##
## Multivariate Imbalance Measure: L1=0.347
## Percentage of local common support: LCS=37.0%
##
## Univariate Imbalance Measures:
##
## statistic type L1 min 25% 50% 75% max
## age 0.21549 (diff) 0.003367 0 1 -1 -1 -6
## education 0.02694 (diff) 0.053872 1 0 0 1 2
## black 0.00000 (diff) 0.000000 0 0 0 0 0
## married 0.00000 (diff) 0.000000 0 0 0 0 0
## nodegree -0.02694 (diff) 0.026936 0 -1 0 0 0
## hispanic 0.00000 (diff) 0.000000 0 0 0 0 0
imbalance.ps
##
## Multivariate Imbalance Measure: L1=0.360
## Percentage of local common support: LCS=35.3%
##
## Univariate Imbalance Measures:
##
## statistic type L1 min 25% 50% 75% max
## age 0.343434 (diff) 0.003367 0 1 0 0 -6
## education 0.043771 (diff) 0.060606 1 0 0 0 2
## black 0.020202 (diff) 0.020202 0 0 0 0 0
## married 0.003367 (diff) 0.003367 0 0 0 0 0
## nodegree -0.010101 (diff) 0.010101 0 0 0 0 0
## hispanic -0.010101 (diff) 0.010101 0 0 0 0 0
imbalance.cem
##
## Multivariate Imbalance Measure: L1=0.338
## Percentage of local common support: LCS=53.7%
##
## Univariate Imbalance Measures:
##
## statistic type L1 min 25% 50% 75% max
## age 0.0702614 (diff) 0.0000000 0 0 -1 1 1
## education -0.1516340 (diff) 0.0784314 0 0 -1 0 0
## black 0.0107843 (diff) 0.0107843 0 0 0 0 0
## married 0.0006536 (diff) 0.0006536 0 0 0 0 0
## nodegree 0.0781046 (diff) 0.0781046 0 0 0 0 0
## hispanic 0.0049020 (diff) 0.0049020 0 0 0 0 0
## subclass -1.1926471 (diff) 0.0156863 0 -5 -8 1 0
Overall, the matching lowered the \( L_1 \) statistic compared to the raw data, achieving better balance between the treatment and control groups. The Mahalanobis (nearest neighbors) and CEM methods have the lowest \( L_1 \) statistics, 0.340 and 0.388 respectively for the multivariate imbalance measure. It seems the CEM method does the best job in creating balance (by a slight margin).
For both the Mahalanobis and propensity score methods, 128 control observations were dropped because the method is a 1-1 match. For CEM (which is more of a pruning technique), 65 controlled and 42 treated were dropped.
4: Compute the difference-in-means again using the data created by each matching method. Report the results in a \( \LaTeX \) table and interpret them in a few sentences. Are there big differences in the estimated treatment effect, or are the three matching methods similar? What do the results suggest about the effect of the NSW program?
## Compute Difference-in-Means for Matched Data ##
t.test(re78 ~ treated, data = data.maha)
##
## Welch Two Sample t-test
##
## data: re78 by treated
## t = -2.379, df = 562.1, p-value = 0.0177
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -2224.4 -212.4
## sample estimates:
## mean in group 0 mean in group 1
## 4758 5976
t.test(re78 ~ treated, data = data.ps)
##
## Welch Two Sample t-test
##
## data: re78 by treated
## t = -1.736, df = 563.3, p-value = 0.08311
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -1899.0 117.1
## sample estimates:
## mean in group 0 mean in group 1
## 5085 5976
t.test(re78 ~ treated, data = data.cem)
##
## Welch Two Sample t-test
##
## data: re78 by treated
## t = -1.763, df = 464.7, p-value = 0.07859
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -1979.8 107.4
## sample estimates:
## mean in group 0 mean in group 1
## 4851 5787
Looking at the treatment group (reference the \( \LaTeX \) document), the means are the same for the first two methods and the original data: 5976. It is slightly lower for the CEM method at 5787. None of the p-values are statistically significant, therefore we concluce there is no significant difference between the control and treatment groups, and that the effect of the NSW program is not significant on job earnings.