When including control variables in a linear regression of some outcome variable on a binary treatment indicator, these control variables address endogeneity concerns only to the extent that: (a) there are no other unobserved factors that affect the treatment and could therefore confound inferences; and (b) the relation between the control variable and the outcome and independent variable(s) of interest is linear. Matching methods alleviate the inferential concerns caused by the assumption of a linear functional form in standard linear regressions. They do not help control for unobserved confounding factors. The most common matching methods used in the literature are currently propensity score matching and entropy balancing.We focus on the former here.
We use the psmatch2 command from Stata to perform our
propensity score matching.
Because there are many degrees of freedom with the implementation of propensity-score matching,we will only illustrate how the command can be implemented.
The code below is based on Leung and Veenman (2018). The
dataset lvdata.dta, which is opened in the code below,
contains a treatment indicator variable. This variable
equals 1 for all “treatment” firm-quarters, and 0 for a control group of
firm-quarter observations. The goal of the matching procedure is to
match each treatment firm-quarter to a control firm-quarter, and then to
compare an outcome variable cfop1 for these
observations.
The matching is done by minimizing the average difference in values of a set of covariates between treatment and control observations.
Note that because not all of the treatment observations have a suitable match available, some observations remain unmatched.
To understand the basic idea of the propensity score, we can first estimate a logit regression in which we explain whether an observation is identified in the treatment group based on observable firm characteristics. Below is the output from this estimation.
use "lvdata.dta",clear
logit treatment timedum_* lnta btm lnage salesgr lnsdearn spdum rnd div cfo acc expsc inta depr ng_indusnote: timedum_36 omitted because of collinearity.
Iteration 0: log likelihood = -4583.162
Iteration 1: log likelihood = -3057.2911
Iteration 2: log likelihood = -2753.6509
Iteration 3: log likelihood = -2734.7023
Iteration 4: log likelihood = -2734.484
Iteration 5: log likelihood = -2734.4838
Logistic regression Number of obs = 9,259
LR chi2(49) = 3697.36
Prob > chi2 = 0.0000
Log likelihood = -2734.4838 Pseudo R2 = 0.4034
------------------------------------------------------------------------------
treatment | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
timedum_1 | 1.273881 .3507545 3.63 0.000 .586415 1.961347
timedum_2 | .9360534 .3489256 2.68 0.007 .2521718 1.619935
timedum_3 | 1.127757 .3371575 3.34 0.001 .4669405 1.788573
timedum_4 | .2841641 .3522427 0.81 0.420 -.4062188 .974547
timedum_5 | .9073774 .349327 2.60 0.009 .2227091 1.592046
timedum_6 | 1.038683 .342915 3.03 0.002 .3665819 1.710784
timedum_7 | .8660718 .3288931 2.63 0.008 .2214531 1.51069
timedum_8 | .5849642 .322862 1.81 0.070 -.0478336 1.217762
timedum_9 | 1.092302 .3076985 3.55 0.000 .4892244 1.69538
timedum_10 | 1.247833 .3085367 4.04 0.000 .6431119 1.852553
timedum_11 | 1.013415 .3096384 3.27 0.001 .4065349 1.620295
timedum_12 | 1.135317 .2892457 3.93 0.000 .568406 1.702228
timedum_13 | .9449195 .2965354 3.19 0.001 .3637209 1.526118
timedum_14 | .7785439 .301071 2.59 0.010 .1884556 1.368632
timedum_15 | .810315 .3048659 2.66 0.008 .2127888 1.407841
timedum_16 | 1.010489 .3002162 3.37 0.001 .4220763 1.598902
timedum_17 | .1255879 .3409337 0.37 0.713 -.5426298 .7938057
timedum_18 | .5942768 .3400123 1.75 0.080 -.0721351 1.260689
timedum_19 | .6958536 .3185705 2.18 0.029 .0714669 1.32024
timedum_20 | .7087685 .3103793 2.28 0.022 .1004362 1.317101
timedum_21 | .4724516 .3271578 1.44 0.149 -.1687659 1.113669
timedum_22 | .3140497 .3560453 0.88 0.378 -.3837864 1.011886
timedum_23 | .3208126 .3390056 0.95 0.344 -.3436261 .9852513
timedum_24 | .4445938 .3158938 1.41 0.159 -.1745467 1.063734
timedum_25 | .3744538 .3253519 1.15 0.250 -.2632241 1.012132
timedum_26 | .3473209 .3310122 1.05 0.294 -.3014511 .996093
timedum_27 | .3170427 .3091951 1.03 0.305 -.2889686 .923054
timedum_28 | .106522 .308592 0.35 0.730 -.4983072 .7113512
timedum_29 | .3733525 .3088679 1.21 0.227 -.2320174 .9787225
timedum_30 | .1482598 .3262718 0.45 0.650 -.4912212 .7877407
timedum_31 | -.1266367 .3187739 -0.40 0.691 -.751422 .4981485
timedum_32 | -.1176477 .3051106 -0.39 0.700 -.7156535 .4803582
timedum_33 | .0731411 .299406 0.24 0.807 -.5136839 .6599661
timedum_34 | -.0949706 .3072279 -0.31 0.757 -.6971263 .5071851
timedum_35 | -.3684925 .3153805 -1.17 0.243 -.986627 .249642
timedum_36 | 0 (omitted)
lnta | .5459653 .0291749 18.71 0.000 .4887836 .6031471
btm | .0522964 .0812177 0.64 0.520 -.1068873 .2114801
lnage | -.2490797 .0569752 -4.37 0.000 -.360749 -.1374104
salesgr | .0606946 .0664283 0.91 0.361 -.0695025 .1908917
lnsdearn | -.0438052 .0370056 -1.18 0.237 -.1163348 .0287244
spdum | .6638392 .0746219 8.90 0.000 .5175829 .8100954
rnd | 4.461092 1.532898 2.91 0.004 1.456668 7.465517
div | -22.04053 8.673314 -2.54 0.011 -39.03991 -5.041143
cfo | 20.68008 1.212625 17.05 0.000 18.30338 23.05678
acc | 7.285523 1.030672 7.07 0.000 5.265443 9.305602
expsc | 80.25899 6.616544 12.13 0.000 67.29081 93.22718
inta | 2.177683 .1764341 12.34 0.000 1.831879 2.523488
depr | -18.73584 4.174841 -4.49 0.000 -26.91838 -10.5533
ng_indus | 7.235906 .2597329 27.86 0.000 6.726839 7.744974
_cons | -9.171973 .4257103 -21.55 0.000 -10.00635 -8.337596
------------------------------------------------------------------------------The results suggest there are substantial and significant differences
in characteristics between the groups. To reduce these differences, we
can use propensity score matching, which is a procedure that identifies
observations in the control group (treatment==0) that have
a similar conditional probability of being treated, based on the
observed covariates, as observations in the treatment group
(treatment==1).The goal is to have a treatment sample and a
control sample for which we no longer observe significant differences in
the observed covariates. The only difference between the two groups
should be their treatment status.
We will use psmatch2 for the propensity score matching to balance the covariates in the treatment and control samples, but we can first compute the propensity scores ourselves using the fitted values of the logit estimation:
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
treatment | 9,259 .1961335 .3970921 0 1
pscore | 9,259 .1961335 .2570519 1.47e-07 .9903858We can see that the treatment indicator variable equals
1 for 19.6 percent of the sample, and 0 otherwise. The
pscore variable we just created has the same mean as
treatment, but it is a continuous variable that captures
the predicted probability of treatment based on the covariates included
in the logit estimation from above. When we evaluate these predicted
probabilities for the treatment and control groups, we can see they
clearly discriminate between observations that are actually treated
versus those that are not:
Summary for variables: pscore
Group variable: treatment
treatment | Mean
----------+----------
0 | .1135354
1 | .5346675
----------+----------
Total | .1961335
---------------------To reduce the differences in covariate values between the treatment
and control sample, psmatch2 computes the propensity score
in the same way (assuming we let the program use a logit estimation; the
alternative is probit). The following code uses the same logit
estimation as presented above and specifies that the outcome variable we
examine is cfop1. We further specify that we match each
treatment observation to its nearest neighbor in terms of the propensity
score (n(1)), we require common support, we allow a maximum
caliper distance in propensity scores between matched pairs
of 0.01, and allow each control observation to be matched with only one
treatment observation (noreplace):
psmatch2 treatment timedum_* lnta btm lnage salesgr lnsdearn spdum rnd div cfo acc expsc inta depr ng_indus, n(1) common caliper(0.01) logit noreplace outc(cfop1)note: timedum_36 omitted because of collinearity.
Logistic regression Number of obs = 9,259
LR chi2(49) = 3697.36
Prob > chi2 = 0.0000
Log likelihood = -2734.4838 Pseudo R2 = 0.4034
------------------------------------------------------------------------------
treatment | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
timedum_1 | 1.273881 .3507545 3.63 0.000 .586415 1.961347
timedum_2 | .9360534 .3489256 2.68 0.007 .2521718 1.619935
timedum_3 | 1.127757 .3371575 3.34 0.001 .4669405 1.788573
timedum_4 | .2841641 .3522427 0.81 0.420 -.4062188 .974547
timedum_5 | .9073774 .349327 2.60 0.009 .2227091 1.592046
timedum_6 | 1.038683 .342915 3.03 0.002 .3665819 1.710784
timedum_7 | .8660718 .3288931 2.63 0.008 .2214531 1.51069
timedum_8 | .5849642 .322862 1.81 0.070 -.0478336 1.217762
timedum_9 | 1.092302 .3076985 3.55 0.000 .4892244 1.69538
timedum_10 | 1.247833 .3085367 4.04 0.000 .6431119 1.852553
timedum_11 | 1.013415 .3096384 3.27 0.001 .4065349 1.620295
timedum_12 | 1.135317 .2892457 3.93 0.000 .568406 1.702228
timedum_13 | .9449195 .2965354 3.19 0.001 .3637209 1.526118
timedum_14 | .7785439 .301071 2.59 0.010 .1884556 1.368632
timedum_15 | .810315 .3048659 2.66 0.008 .2127888 1.407841
timedum_16 | 1.010489 .3002162 3.37 0.001 .4220763 1.598902
timedum_17 | .1255879 .3409337 0.37 0.713 -.5426298 .7938057
timedum_18 | .5942768 .3400123 1.75 0.080 -.0721351 1.260689
timedum_19 | .6958536 .3185705 2.18 0.029 .0714669 1.32024
timedum_20 | .7087685 .3103793 2.28 0.022 .1004362 1.317101
timedum_21 | .4724516 .3271578 1.44 0.149 -.1687659 1.113669
timedum_22 | .3140497 .3560453 0.88 0.378 -.3837864 1.011886
timedum_23 | .3208126 .3390056 0.95 0.344 -.3436261 .9852513
timedum_24 | .4445938 .3158938 1.41 0.159 -.1745467 1.063734
timedum_25 | .3744538 .3253519 1.15 0.250 -.2632241 1.012132
timedum_26 | .3473209 .3310122 1.05 0.294 -.3014511 .996093
timedum_27 | .3170427 .3091951 1.03 0.305 -.2889686 .923054
timedum_28 | .106522 .308592 0.35 0.730 -.4983072 .7113512
timedum_29 | .3733525 .3088679 1.21 0.227 -.2320174 .9787225
timedum_30 | .1482598 .3262718 0.45 0.650 -.4912212 .7877407
timedum_31 | -.1266367 .3187739 -0.40 0.691 -.751422 .4981485
timedum_32 | -.1176477 .3051106 -0.39 0.700 -.7156535 .4803582
timedum_33 | .0731411 .299406 0.24 0.807 -.5136839 .6599661
timedum_34 | -.0949706 .3072279 -0.31 0.757 -.6971263 .5071851
timedum_35 | -.3684925 .3153805 -1.17 0.243 -.986627 .249642
timedum_36 | 0 (omitted)
lnta | .5459653 .0291749 18.71 0.000 .4887836 .6031471
btm | .0522964 .0812177 0.64 0.520 -.1068873 .2114801
lnage | -.2490797 .0569752 -4.37 0.000 -.360749 -.1374104
salesgr | .0606946 .0664283 0.91 0.361 -.0695025 .1908917
lnsdearn | -.0438052 .0370056 -1.18 0.237 -.1163348 .0287244
spdum | .6638392 .0746219 8.90 0.000 .5175829 .8100954
rnd | 4.461092 1.532898 2.91 0.004 1.456668 7.465517
div | -22.04053 8.673314 -2.54 0.011 -39.03991 -5.041143
cfo | 20.68008 1.212625 17.05 0.000 18.30338 23.05678
acc | 7.285523 1.030672 7.07 0.000 5.265443 9.305602
expsc | 80.25899 6.616544 12.13 0.000 67.29081 93.22718
inta | 2.177683 .1764341 12.34 0.000 1.831879 2.523488
depr | -18.73584 4.174841 -4.49 0.000 -26.91838 -10.5533
ng_indus | 7.235906 .2597329 27.86 0.000 6.726839 7.744974
_cons | -9.171973 .4257103 -21.55 0.000 -10.00635 -8.337596
------------------------------------------------------------------------------
-------------------------------------------------------------------------------
> ---------
Variable Sample | Treated Controls Difference S.E.
> T-stat
----------------------------+--------------------------------------------------
> ---------
cfop1 Unmatched | .082819289 -.090824745 .173644034 .005924133
> 29.31
ATT | .077399289 .024039467 .053359822 .005203649
> 10.25
----------------------------+--------------------------------------------------
> ---------
Note: S.E. does not take into account that the propensity score is estimated.
psmatch2: | psmatch2: Common
Treatment | support
assignment | Off suppo On suppor | Total
-----------+----------------------+----------
Untreated | 0 7,443 | 7,443
Treated | 681 1,135 | 1,816
-----------+----------------------+----------
Total | 681 8,578 | 9,259 The first lines of results are the same as we had with the logit
estimation.The next blockof results provides us two different
tests.First we see a \(t\)-test on the
difference in means of cfop1 for the unmatched treatment
and control groups. Second, we see a \(t\)-test on the difference in means of
cfop1 for the treatment and control groups after matching
these groups on the covariates included in the logit estimation. The
results suggest that the difference in means between the matched
treatment and control groups equals 0.053 with a \(t\)-value of 10.25. This is an estimate of
the average treatment effect on the observations that actually receive
the treatment (the “ATT”). Importantly, note that this \(t\)-value is based on standard errors that
do not (yet) account for the effects of potential clustering. In
addition, as indicated by the note “S.E. does not take into account that
the propensity score is estimated”, the standard error might not be
correct because it ignores the fact that a matching step was taken
before the mean comparison between the matched treatment and control
observations.
In the below block of results, we further see information on the “common support”. The common support refers to the extent to which the predicted probabilities (as captured by the propensity scores) for the treatment observations are within the range of those in the control sample. Here, 681 of the 1816 treatment observations have propensity scores that fall outside of this range. We can see this more clearly by inspecting the propensity scores for the treatment and control observations that remain unmatched:
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
pscore | 681 .8000031 .1307566 .4820849 .9893873
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
pscore | 6,308 .0658553 .0866694 1.47e-07 .4721092For the 681 for which no match was found, we can see that their propensity scores do not overlap with the distribution of propensity scores of the remaining observations in the control sample. The lowest propensity score of the unmatched treatment group is 0.482, while the highest propensity score of the unmatched control group is 0.472. Because we match without replacement, the pool of potential control observation matches shrinks every time the program’s algorithm finds a match for one of the treatment observations. The matching takes place until there is no more overlap in the distributions of the propensity scores of the treatment and control groups.
When we inspect our dataset after running psmatch2,we
can also see that several new variables were created by the
program.These are:
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
_pscore | 9,259 .1961335 .2570519 1.47e-07 .9903858
_treated | 9,259 .1961335 .3970921 0 1
_support | 9,259 .9264499 .2610514 0 1
_weight | 2,270 1 0 1 1
_cfop1 | 1,135 .0240395 .1484834 -.6990907 .4217812
-------------+---------------------------------------------------------
_id | 9,259 4630 2672.987 1 9259
_n1 | 1,135 6337.086 1184.003 819 7443
_nn | 9,259 .1225834 .3279761 0 1
_pdif | 1,135 .0030939 .0040083 1.11e-09 .0099939Of these variables,_pscore is exactly the same
propensity score as we calculated ourselves after the logit estimation,
_treated is equal to the treatment variable, and
_support is an indicator variable that is equal to 1 for
observations in the control group or treatment observations with common
support.Variable _weight captures the number of times an
observation is used for a match.For treatment observations this value is
always 1,but for control observations this can be greater than 1 if we
would match with replacement. Because we match without replacement
here,all values equal 1.
Variable _cfop1 is a variable that, for each treatment
observation, contains the value of the outcome variable of the matched
control observation. Variable _id is a variable that
uniquely identifies each observation in the sample. This variable is
useful because it links to variable _n1, which contains the
identifier of the nearest-neighbor match for a treatment
observation.
The variable _nn contains the number of observations
matched to a treatment observation, which is 1 here given the choice to
match only to the first nearest neighbor. Finally, for each matched
treatment observation, variable _pdif contains the
difference in propensity scores between matched treatment and control
observations.
For example, the highest value of 0.0099939 (which is below the
caliper distance of 0.01) belongs to the treatment observation with
_id == 9103, which was matched to the control observation
with _id == 7434. The propensity scores for these
observations are 0.91661393 and 0.92660782, respectively, which
translates into the difference of 0.0099939.
It is important that we not only examine the differences in the means of the outcome variable between the matched treatment and control observations, but also that we assess the success of the matching procedure in balancing the treatment and control groups in terms of the observed covariates. One simple solution is to redo the logit estimation for the treatment indicator, but now only for the matched treatment and control observations. In the output shown below, we can see that none of the covariates is significantly related to the treatment status after matching:
gen treat_psm=0 if _weight!=. & treatment==0
replace treat_psm=1 if _weight!=. & treatment==1
logit treat_psm timedum_* lnta btm lnage salesgr lnsdearn spdum rnd div cfo acc expsc inta depr ng_indus(8,124 missing values generated)
(1,135 real changes made)
note: timedum_36 omitted because of collinearity.
Iteration 0: log likelihood = -1573.4441
Iteration 1: log likelihood = -1566.2955
Iteration 2: log likelihood = -1566.295
Iteration 3: log likelihood = -1566.295
Logistic regression Number of obs = 2,270
LR chi2(49) = 14.30
Prob > chi2 = 1.0000
Log likelihood = -1566.295 Pseudo R2 = 0.0045
------------------------------------------------------------------------------
treat_psm | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
timedum_1 | .0000751 .429216 0.00 1.000 -.8411728 .841323
timedum_2 | .2352228 .4530822 0.52 0.604 -.652802 1.123248
timedum_3 | -.1544237 .4116886 -0.38 0.708 -.9613186 .6524712
timedum_4 | .0647871 .4396591 0.15 0.883 -.796929 .9265031
timedum_5 | -.3536509 .4267346 -0.83 0.407 -1.190035 .4827335
timedum_6 | -.2232423 .4081204 -0.55 0.584 -1.023144 .5766589
timedum_7 | .140271 .4076743 0.34 0.731 -.6587559 .9392979
timedum_8 | -.0765405 .3943007 -0.19 0.846 -.8493558 .6962747
timedum_9 | .0329149 .3719496 0.09 0.929 -.6960928 .7619227
timedum_10 | -.1468621 .3734208 -0.39 0.694 -.8787534 .5850293
timedum_11 | -.119388 .3757083 -0.32 0.751 -.8557627 .6169867
timedum_12 | -.045745 .3482933 -0.13 0.896 -.7283874 .6368974
timedum_13 | -.0297335 .3538331 -0.08 0.933 -.7232336 .6637667
timedum_14 | .1755499 .3707573 0.47 0.636 -.5511212 .9022209
timedum_15 | -.1334426 .3731748 -0.36 0.721 -.8648518 .5979666
timedum_16 | .0077959 .3660923 0.02 0.983 -.7097318 .7253236
timedum_17 | .01368 .4247273 0.03 0.974 -.8187702 .8461302
timedum_18 | -.1844412 .4171329 -0.44 0.658 -1.002007 .6331243
timedum_19 | -.0396841 .3836351 -0.10 0.918 -.7915951 .7122268
timedum_20 | -.059817 .3795308 -0.16 0.875 -.8036837 .6840497
timedum_21 | -.1028496 .3924284 -0.26 0.793 -.8719951 .6662959
timedum_22 | -.1468376 .4629627 -0.32 0.751 -1.054228 .7605527
timedum_23 | -.4616144 .406546 -1.14 0.256 -1.25843 .3352011
timedum_24 | -.3181094 .3818262 -0.83 0.405 -1.066475 .4302562
timedum_25 | .0122734 .3942684 0.03 0.975 -.7604785 .7850253
timedum_26 | -.0901802 .41155 -0.22 0.827 -.8968033 .716443
timedum_27 | -.0990496 .3752038 -0.26 0.792 -.8344356 .6363364
timedum_28 | .00079 .3720623 0.00 0.998 -.7284387 .7300187
timedum_29 | -.2965307 .3630628 -0.82 0.414 -1.008121 .4150594
timedum_30 | -.1405657 .4126521 -0.34 0.733 -.949349 .6682176
timedum_31 | .1326658 .3993126 0.33 0.740 -.6499726 .9153042
timedum_32 | .0894299 .3813735 0.23 0.815 -.6580485 .8369082
timedum_33 | .0046732 .3573383 0.01 0.990 -.6956969 .7050434
timedum_34 | .05745 .3745345 0.15 0.878 -.6766241 .7915241
timedum_35 | -.091291 .3801281 -0.24 0.810 -.8363283 .6537463
timedum_36 | 0 (omitted)
lnta | .0051809 .035437 0.15 0.884 -.0642744 .0746362
btm | -.0226109 .1021104 -0.22 0.825 -.2227435 .1775218
lnage | .0514345 .0691098 0.74 0.457 -.0840183 .1868873
salesgr | .0589519 .0844805 0.70 0.485 -.1066268 .2245305
lnsdearn | .0424737 .0439864 0.97 0.334 -.043738 .1286854
spdum | -.0354987 .0931525 -0.38 0.703 -.2180742 .1470768
rnd | 1.209932 2.030382 0.60 0.551 -2.769543 5.189407
div | -7.231397 9.833041 -0.74 0.462 -26.5038 12.04101
cfo | -1.367198 1.533419 -0.89 0.373 -4.372644 1.638249
acc | -.3149529 1.212049 -0.26 0.795 -2.690526 2.06062
expsc | -5.45401 8.649604 -0.63 0.528 -22.40692 11.4989
inta | .0402169 .213122 0.19 0.850 -.3774945 .4579283
depr | 1.95448 5.189355 0.38 0.706 -8.216468 12.12543
ng_indus | -.0606578 .31979 -0.19 0.850 -.6874347 .5661191
_cons | .1093349 .5219786 0.21 0.834 -.9137243 1.132394
------------------------------------------------------------------------------We can also test the differences in means for each variable individually by regressing the variable on the matched treatment indicator. This also allows us to adjust the standard errors for potential clustering,as follows:
tabstat lnta btm lnage salesgr lnsdearn spdum rnd div cfo acc expsc inta depr ng_indus, by(treat_psm)
reghdfe lnta treat_psm, cluster(cik qid) noabSummary statistics: Mean
Group variable: treat_psm
treat_psm | lnta btm lnage salesgr lnsdearn spdum
----------+------------------------------------------------------------
0 | 6.367659 .5907604 2.555425 .1549692 -3.929477 .5832599
1 | 6.370766 .584871 2.574921 .1617725 -3.882153 .5814978
----------+------------------------------------------------------------
Total | 6.369213 .5878157 2.565173 .1583709 -3.905815 .5823789
-----------------------------------------------------------------------
treat_psm | rnd div cfo acc expsc inta
----------+------------------------------------------------------------
0 | .02179 .001273 .0159721 -.0437311 .0063702 .1926536
1 | .0226218 .0011436 .0145904 -.0431276 .0063768 .1948018
----------+------------------------------------------------------------
Total | .0222059 .0012083 .0152813 -.0434294 .0063735 .1937277
-----------------------------------------------------------------------
treat_psm | depr ng_indus
----------+--------------------
0 | .0136364 .5122608
1 | .0136619 .5129756
----------+--------------------
Total | .0136492 .5126182
-------------------------------
(MWFE estimator converged in 1 iterations)
HDFE Linear regression Number of obs = 2,270
Absorbing 1 HDFE group F( 1, 35) = 0.00
Statistics robust to heteroskedasticity Prob > F = 0.9832
R-squared = 0.0000
Adj R-squared = -0.0004
Number of clusters (cik) = 932 Within R-sq. = 0.0000
Number of clusters (qid) = 36 Root MSE = 1.6705
(Std. err. adjusted for 36 clusters in cik qid)
------------------------------------------------------------------------------
| Robust
lnta | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
treat_psm | .0031065 .1464974 0.02 0.983 -.294299 .300512
_cons | 6.367659 .1198772 53.12 0.000 6.124296 6.611023
------------------------------------------------------------------------------Finally, to perform our actual tests of differences in the outcome
variables between the matched treatment and control observations, we can
estimate a simple linear regression of cfop1 on the
treat_psm variable.
Without adjusting the standard errors, we get a coefficient of
0.0533598 with a \(t\)-value of 10.25,
which is exactly the same as for the ATT shown by psmatch2.
Alternatively, we can use the regression to adjust the standard errors
and/or to add additional control variables in the model. In the
following output, we can see the difference in average future cash flows
between treated and control observations with standard errors that are
clustered by firm and time:
Source | SS df MS Number of obs = 2,270
-------------+---------------------------------- F(1, 2268) = 105.15
Model | 1.61582608 1 1.61582608 Prob > F = 0.0000
Residual | 34.8517811 2,268 .015366747 R-squared = 0.0443
-------------+---------------------------------- Adj R-squared = 0.0439
Total | 36.4676071 2,269 .016072105 Root MSE = .12396
------------------------------------------------------------------------------
cfop1 | Coefficient Std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
treat_psm | .0533598 .0052036 10.25 0.000 .0431554 .0635642
_cons | .0240395 .0036795 6.53 0.000 .0168239 .0312551
------------------------------------------------------------------------------
(MWFE estimator converged in 1 iterations)
HDFE Linear regression Number of obs = 2,270
Absorbing 1 HDFE group F( 1, 35) = 41.19
Statistics robust to heteroskedasticity Prob > F = 0.0000
R-squared = 0.0443
Adj R-squared = 0.0439
Number of clusters (cik) = 932 Within R-sq. = 0.0443
Number of clusters (qid) = 36 Root MSE = 0.1240
(Std. err. adjusted for 36 clusters in cik qid)
------------------------------------------------------------------------------
| Robust
cfop1 | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
treat_psm | .0533598 .0083146 6.42 0.000 .0364803 .0702394
_cons | .0240395 .0067449 3.56 0.001 .0103467 .0377322
------------------------------------------------------------------------------Similarly, we can estimate this regression with all of the matching
covariates included as well.This form of “doubly-robust” estimation
helps to eliminate any residual (random) differences in covariates
across the treated and control observations. The idea is that the
matching is never perfect (i.e., _pdif > 0 for all
observations).
reghdfe cfop1 timedum_* indusdum_* cfo acc_gaap lnta btm lnage salesgr lnsdearn spdum rnd div expsc inta depr ng_indus treat_psm, cluster(cik qid) noab (MWFE estimator converged in 1 iterations)
note: timedum_36 omitted because of collinearity
note: indusdum_2 omitted because of collinearity
note: indusdum_3 omitted because of collinearity
note: indusdum_7 omitted because of collinearity
note: indusdum_10 omitted because of collinearity
note: indusdum_12 omitted because of collinearity
note: indusdum_31 omitted because of collinearity
note: indusdum_32 omitted because of collinearity
note: indusdum_36 omitted because of collinearity
note: indusdum_42 omitted because of collinearity
note: indusdum_44 omitted because of collinearity
note: indusdum_57 omitted because of collinearity
note: indusdum_58 omitted because of collinearity
note: indusdum_61 omitted because of collinearity
Warning: VCV matrix was non-positive semi-definite; adjustment from Cameron, Ge
> lbach & Miller applied.
warning: missing F statistic; dropped variables due to collinearity or too few
> clusters
HDFE Linear regression Number of obs = 2,270
Absorbing 1 HDFE group F( 98, 35) = .
Statistics robust to heteroskedasticity Prob > F = .
R-squared = 0.3656
Adj R-squared = 0.3369
Number of clusters (cik) = 932 Within R-sq. = 0.3656
Number of clusters (qid) = 36 Root MSE = 0.1032
(Std. err. adjusted for 36 clusters in cik qid)
------------------------------------------------------------------------------
| Robust
cfop1 | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
timedum_1 | .0849939 .0136648 6.22 0.000 .057253 .1127349
timedum_2 | .0762377 .014242 5.35 0.000 .0473249 .1051505
timedum_3 | .1125561 .0133715 8.42 0.000 .0854106 .1397016
timedum_4 | .0418084 .0107962 3.87 0.000 .019891 .0637258
timedum_5 | .0593946 .0108264 5.49 0.000 .0374159 .0813733
timedum_6 | .0734642 .0105373 6.97 0.000 .0520724 .0948561
timedum_7 | .0563082 .0098797 5.70 0.000 .0362513 .076365
timedum_8 | .0468361 .0099462 4.71 0.000 .0266443 .0670279
timedum_9 | .068859 .0098891 6.96 0.000 .048783 .088935
timedum_10 | .0500091 .0085908 5.82 0.000 .0325688 .0674494
timedum_11 | .0628898 .0094592 6.65 0.000 .0436867 .0820929
timedum_12 | .0750848 .0104313 7.20 0.000 .0539082 .0962614
timedum_13 | .0976534 .0079855 12.23 0.000 .0814418 .1138649
timedum_14 | .0674046 .0072407 9.31 0.000 .0527052 .082104
timedum_15 | .0754358 .009598 7.86 0.000 .0559509 .0949207
timedum_16 | .0603559 .0084664 7.13 0.000 .0431681 .0775436
timedum_17 | .0746495 .0073834 10.11 0.000 .0596604 .0896386
timedum_18 | .0477049 .0120886 3.95 0.000 .0231637 .0722461
timedum_19 | .0488181 .0088523 5.51 0.000 .030847 .0667892
timedum_20 | .0187014 .0074291 2.52 0.017 .0036194 .0337833
timedum_21 | .0536613 .0081346 6.60 0.000 .0371473 .0701753
timedum_22 | .0168733 .0120445 1.40 0.170 -.0075784 .041325
timedum_23 | .0157901 .0072071 2.19 0.035 .001159 .0304213
timedum_24 | .0369831 .0067826 5.45 0.000 .0232136 .0507525
timedum_25 | .0315401 .0064821 4.87 0.000 .0183808 .0446994
timedum_26 | .0233141 .0059623 3.91 0.000 .01121 .0354183
timedum_27 | .0396758 .0059177 6.70 0.000 .0276622 .0516893
timedum_28 | .038837 .0066958 5.80 0.000 .0252438 .0524303
timedum_29 | .0400785 .0057748 6.94 0.000 .0283552 .0518019
timedum_30 | .0387584 .0082173 4.72 0.000 .0220764 .0554404
timedum_31 | .0160104 .0081684 1.96 0.058 -.0005722 .0325931
timedum_32 | .0146486 .0069134 2.12 0.041 .0006137 .0286836
timedum_33 | .0372344 .0061299 6.07 0.000 .0247901 .0496787
timedum_34 | .0384652 .005897 6.52 0.000 .0264936 .0504368
timedum_35 | .0378944 .0066945 5.66 0.000 .0243038 .051485
timedum_36 | 0 (omitted)
indusdum_1 | .020047 .0210122 0.95 0.347 -.0226101 .0627041
indusdum_2 | 0 (omitted)
indusdum_3 | 0 (omitted)
indusdum_4 | -.0754228 .0339098 -2.22 0.033 -.1442634 -.0065822
indusdum_5 | -.0401419 .0315331 -1.27 0.211 -.1041574 .0238736
indusdum_6 | -.0099137 .0150169 -0.66 0.513 -.0403996 .0205722
indusdum_7 | 0 (omitted)
indusdum_8 | -.0015319 .0269095 -0.06 0.955 -.0561612 .0530973
indusdum_9 | .0202878 .0159906 1.27 0.213 -.0121748 .0527504
indusdum_10 | 0 (omitted)
indusdum_11 | -.0150704 .0185073 -0.81 0.421 -.0526422 .0225014
indusdum_12 | 0 (omitted)
indusdum_13 | -.0090838 .0197947 -0.46 0.649 -.0492692 .0311016
indusdum_14 | -.018188 .0255881 -0.71 0.482 -.0701346 .0337585
indusdum_15 | -.0273003 .0424782 -0.64 0.525 -.1135357 .0589351
indusdum_16 | -.0264195 .0217122 -1.22 0.232 -.0704975 .0176586
indusdum_17 | -.0121096 .0195555 -0.62 0.540 -.0518094 .0275903
indusdum_18 | .0940384 .0379573 2.48 0.018 .016981 .1710958
indusdum_19 | -.0490312 .0213182 -2.30 0.028 -.0923095 -.0057529
indusdum_20 | .0441252 .0361422 1.22 0.230 -.0292473 .1174978
indusdum_21 | -.0619821 .0334217 -1.85 0.072 -.1298319 .0058676
indusdum_22 | -.0085004 .0547354 -0.16 0.877 -.1196191 .1026184
indusdum_23 | -.0781888 .0218834 -3.57 0.001 -.1226144 -.0337631
indusdum_24 | -.0391315 .0204119 -1.92 0.063 -.0805698 .0023069
indusdum_25 | .0103907 .0483377 0.21 0.831 -.08774 .1085215
indusdum_26 | -.0265911 .0222296 -1.20 0.240 -.0717196 .0185374
indusdum_27 | -.0269222 .0203357 -1.32 0.194 -.0682058 .0143614
indusdum_28 | -.0170284 .0281933 -0.60 0.550 -.0742638 .040207
indusdum_29 | -.0424107 .0175856 -2.41 0.021 -.0781113 -.0067101
indusdum_30 | .0252645 .0507241 0.50 0.622 -.077711 .1282399
indusdum_31 | 0 (omitted)
indusdum_32 | 0 (omitted)
indusdum_33 | -.0241045 .0240438 -1.00 0.323 -.0729161 .024707
indusdum_34 | -.0394001 .0115922 -3.40 0.002 -.0629334 -.0158667
indusdum_35 | -.0206011 .0229263 -0.90 0.375 -.067144 .0259418
indusdum_36 | 0 (omitted)
indusdum_37 | -.0075482 .0358138 -0.21 0.834 -.0802542 .0651578
indusdum_38 | -.0184074 .0178012 -1.03 0.308 -.0545457 .0177309
indusdum_39 | -.031101 .0159077 -1.96 0.059 -.0633953 .0011933
indusdum_40 | .0011673 .0246741 0.05 0.963 -.0489237 .0512584
indusdum_41 | -.0106368 .0348559 -0.31 0.762 -.0813981 .0601245
indusdum_42 | 0 (omitted)
indusdum_43 | -.0117064 .020102 -0.58 0.564 -.0525156 .0291028
indusdum_44 | 0 (omitted)
indusdum_45 | -.0495773 .0249388 -1.99 0.055 -.1002057 .0010511
indusdum_46 | -.0381762 .0354382 -1.08 0.289 -.1101196 .0337673
indusdum_47 | -.0236796 .023797 -1.00 0.327 -.0719901 .024631
indusdum_48 | -.030253 .0234751 -1.29 0.206 -.0779101 .017404
indusdum_49 | -.0146478 .0230977 -0.63 0.530 -.0615386 .0322429
indusdum_50 | -.0600502 .0121424 -4.95 0.000 -.0847005 -.0353999
indusdum_51 | .0419915 .0482419 0.87 0.390 -.0559447 .1399277
indusdum_52 | -.0157254 .0205713 -0.76 0.450 -.0574873 .0260365
indusdum_53 | -.0736516 .0397268 -1.85 0.072 -.1543012 .006998
indusdum_54 | -.0270345 .0158431 -1.71 0.097 -.0591977 .0051287
indusdum_55 | -.032207 .018887 -1.71 0.097 -.0705496 .0061355
indusdum_56 | -.0374863 .0196779 -1.90 0.065 -.0774347 .002462
indusdum_57 | 0 (omitted)
indusdum_58 | 0 (omitted)
indusdum_59 | -.0577832 .0167566 -3.45 0.001 -.0918008 -.0237655
indusdum_60 | -.0022788 .0488161 -0.05 0.963 -.1013807 .0968232
indusdum_61 | 0 (omitted)
cfo | .9514314 .1519487 6.26 0.000 .6429591 1.259904
acc_gaap | .4985704 .0963489 5.17 0.000 .3029718 .6941691
lnta | .0087056 .0026742 3.26 0.003 .0032768 .0141344
btm | -.0083251 .0069844 -1.19 0.241 -.0225041 .0058538
lnage | .0059662 .0048624 1.23 0.228 -.0039049 .0158374
salesgr | -.0197993 .0061148 -3.24 0.003 -.032213 -.0073856
lnsdearn | -.0066653 .0027437 -2.43 0.020 -.0122353 -.0010954
spdum | .0067357 .0047343 1.42 0.164 -.0028756 .0163469
rnd | -1.071397 .2639991 -4.06 0.000 -1.607344 -.5354509
div | 2.405209 .4109868 5.85 0.000 1.570861 3.239557
expsc | 1.990613 .9425911 2.11 0.042 .0770513 3.904175
inta | .009229 .0150586 0.61 0.544 -.0213415 .0397996
depr | 3.264426 .3485405 9.37 0.000 2.556851 3.972001
ng_indus | .046966 .0306436 1.53 0.134 -.0152438 .1091759
treat_psm | .0564843 .0064352 8.78 0.000 .0434201 .0695485
_cons | -.1523569 .0234643 -6.49 0.000 -.199992 -.1047218
------------------------------------------------------------------------------The example presented above focused on matching without replacement. It can also be useful, however, to match with replacement. Matching with replacement simply means that once a control observation is used in a match, it is “thrown back” into the potential pool of control observations for potential matching with another treatment observation. This could be useful when there is little common support and/or the pool of control observations is smaller, or not much larger, than the treatment sample.
Consider the following example of whether firms pay a fee premium for hiring a Big-4 auditor instead of a smaller audit firm, and what the magnitude of the fee premium is (if any).In a sample of US firms for the years 2002–2018, we can see that the pool of treatment observations (i.e., Big-4 clients) is substantially larger than the control sample:
use "data_auditfees.dta", clear
sum fyear
tabstat fyear, by(fyear) stats(N)
sum big4
sum big4 if big4==1
sum big4 if big4==0 Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
fyear | 36,115 2009.467 4.919834 2002 2018
Summary for variables: fyear
Group variable: fyear (Data Year - Fiscal)
fyear | N
---------+----------
2002 | 2594
2003 | 2561
2004 | 2359
2005 | 2340
2006 | 2327
2007 | 2274
2008 | 2231
2009 | 2132
2010 | 1964
2011 | 1972
2012 | 1963
2013 | 1924
2014 | 1972
2015 | 1946
2016 | 1923
2017 | 1891
2018 | 1742
---------+----------
Total | 36115
--------------------
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
big4 | 36,115 .777627 .4158462 0 1
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
big4 | 28,084 1 0 1 1
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
big4 | 8,031 0 0 0 0That is, about 78 percent of firm-year observations in this sample are treatment observations, leaving only 22 percent of observations available as potential matches in the control sample. This means that even when we do not impose restrictions such as common support or a caliper distance, the largest part of the treatment sample will not be matched if we decide to match without replacement. In this example, an additional concern is that there is very limited overlap in the distributions of propensity scores for the treatment and control samples.So this leaves even fewer control firm-observations as suitable matches.
In the code below, we first match without replacement using a simple set of firm characteristics. We can see that 8,031 firm-years in the Big-4 (treatment) sample were matched to an observation in the control sample, much less than the total number of 28,137 firm-years in the treatment sample. This is because the control sample consists of only 8,031 observations. Each of these observations is thus used in a match with a treatment observation.
psmatch2 big4 lnassets aturn curr lev roa salesgr, n(1) logit noreplace outc(lnauditfees)
sum _weight
sum big4 if big4==0 & _weight==.Logistic regression Number of obs = 36,115
LR chi2(6) = 11312.03
Prob > chi2 = 0.0000
Log likelihood = -13481.142 Pseudo R2 = 0.2956
------------------------------------------------------------------------------
big4 | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
lnassets | .9962023 .0126367 78.83 0.000 .9714347 1.02097
aturn | .156676 .021578 7.26 0.000 .1143839 .1989681
curr | .0536695 .0084254 6.37 0.000 .0371561 .070183
lev | -.7975957 .0782258 -10.20 0.000 -.9509154 -.644276
roa | -1.601652 .0914624 -17.51 0.000 -1.780915 -1.422389
salesgrowth | -.1897099 .0401083 -4.73 0.000 -.2683207 -.1110992
_cons | -4.454175 .0949572 -46.91 0.000 -4.640288 -4.268062
------------------------------------------------------------------------------
-------------------------------------------------------------------------------
> ---------
Variable Sample | Treated Controls Difference S.E.
> T-stat
----------------------------+--------------------------------------------------
> ---------
lnauditfees Unmatched | 14.1198827 12.6593561 1.46052659 .013189094
> 110.74
ATT | 13.0940304 12.6593561 .434674244 .014031483
> 30.98
----------------------------+--------------------------------------------------
> ---------
Note: S.E. does not take into account that the propensity score is estimated.
psmatch2: | psmatch2: Common
Treatment | support
assignment | Off suppo On suppor | Total
-----------+----------------------+----------
Untreated | 0 8,031 | 8,031
Treated | 20,053 8,031 | 28,084
-----------+----------------------+----------
Total | 20,053 16,062 | 36,115
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
_weight | 16,062 1 0 1 1
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
big4 | 0Importantly, further inspection suggests that the matching is extremely poor. For example, we can see that although the pre-matching difference in propensity scores between the treatment and control groups is reduced after the matching, the average propensity scores of 62.3 and 53.1 percent for the matched treatment and control observations are still far apart.
// Propensity score before matching
tabstat _pscore, by(big4)
// Propensity score after matching
tabstat _pscore if _weight!=., by(big4)Summary for variables: _pscore
Group variable: big4
big4 | Mean
---------+----------
0 | .53097
1 | .8481619
---------+----------
Total | .777627
--------------------
Summary for variables: _pscore
Group variable: big4
big4 | Mean
---------+----------
0 | .53097
1 | .6232409
---------+----------
Total | .5771055
--------------------Another important issue we observe when comparing the mean propensity
scores of the treatment group before and after matching is that the mean
propensity score of the treatment observations drops from 84.8 to 62.3
percent after matching. This means that out of all the treatment
observations in the original sample, the matched sample contains only
those observations that had a relatively low conditional probability of
hiring a Big-4 auditor. Given the importance of client size in
explaining the likelihood of having a Big-4 auditor (as evident from
logit results of psmatch2 above), this means that the
matched treatment firms are much smaller than the unmatched treatment
firms. (Variable lnassets is the natural logarithm of
firms’ total assets, a measure of firm size.)
gen big4_matched=0 if big4==1
replace big4_matched=1 if big4==1 & _weight!=.
tabstat lnassets, by(big4_matched)
drop big4_matched(8,031 missing values generated)
(8,031 real changes made)
Summary for variables: lnassets
Group variable: big4_matched
big4_matched | Mean
-------------+----------
0 | 7.84284
1 | 4.995169
-------------+----------
Total | 7.02851
------------------------The problem of the remaining difference in average propensity scores after matching can be addressed by imposing the caliper restriction (i.e., a maximum distance in propensity scores between matched pairs). Doing so helps balance the average propensity scores and remove the significant differences in firm characteristics between the matched treatment and control samples, but this also further reduces the number of treatment observations that are matched to 6,011.
// With common support:
psmatch2 big4 lnassets aturn curr lev roa salesgr, n(1) common logit noreplace outc(lnauditfees)
sum _weight
sum big4 if big4==0 & _weight==.
tabstat _pscore if _weight!=., by(big4)
// With common support and caliper:
psmatch2 big4 lnassets aturn curr lev roa salesgr, n(1) common caliper(0.01) logit noreplace outc(lnauditfees)
tabstat _pscore if _weight!=., by(big4)
reg lnauditfees big4 if _weight!=.
reg lnauditfees big4 if _weight!=., cluster(gvkey)
// Doubly-robust:
reg lnauditfee big4 lnassets aturn curr lev roa salesgr if _weight!=., cluster(gvkey)Logistic regression Number of obs = 36,115
LR chi2(6) = 11312.03
Prob > chi2 = 0.0000
Log likelihood = -13481.142 Pseudo R2 = 0.2956
------------------------------------------------------------------------------
big4 | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
lnassets | .9962023 .0126367 78.83 0.000 .9714347 1.02097
aturn | .156676 .021578 7.26 0.000 .1143839 .1989681
curr | .0536695 .0084254 6.37 0.000 .0371561 .070183
lev | -.7975957 .0782258 -10.20 0.000 -.9509154 -.644276
roa | -1.601652 .0914624 -17.51 0.000 -1.780915 -1.422389
salesgrowth | -.1897099 .0401083 -4.73 0.000 -.2683207 -.1110992
_cons | -4.454175 .0949572 -46.91 0.000 -4.640288 -4.268062
------------------------------------------------------------------------------
-------------------------------------------------------------------------------
> ---------
Variable Sample | Treated Controls Difference S.E.
> T-stat
----------------------------+--------------------------------------------------
> ---------
lnauditfees Unmatched | 14.1198827 12.6593561 1.46052659 .013189094
> 110.74
ATT | 13.0940304 12.6593561 .434674244 .014031483
> 30.98
----------------------------+--------------------------------------------------
> ---------
Note: S.E. does not take into account that the propensity score is estimated.
psmatch2: | psmatch2: Common
Treatment | support
assignment | Off suppo On suppor | Total
-----------+----------------------+----------
Untreated | 0 8,031 | 8,031
Treated | 20,053 8,031 | 28,084
-----------+----------------------+----------
Total | 20,053 16,062 | 36,115
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
_weight | 16,062 1 0 1 1
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
big4 | 0
Summary for variables: _pscore
Group variable: big4
big4 | Mean
---------+----------
0 | .53097
1 | .6232409
---------+----------
Total | .5771055
--------------------
Logistic regression Number of obs = 36,115
LR chi2(6) = 11312.03
Prob > chi2 = 0.0000
Log likelihood = -13481.142 Pseudo R2 = 0.2956
------------------------------------------------------------------------------
big4 | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
lnassets | .9962023 .0126367 78.83 0.000 .9714347 1.02097
aturn | .156676 .021578 7.26 0.000 .1143839 .1989681
curr | .0536695 .0084254 6.37 0.000 .0371561 .070183
lev | -.7975957 .0782258 -10.20 0.000 -.9509154 -.644276
roa | -1.601652 .0914624 -17.51 0.000 -1.780915 -1.422389
salesgrowth | -.1897099 .0401083 -4.73 0.000 -.2683207 -.1110992
_cons | -4.454175 .0949572 -46.91 0.000 -4.640288 -4.268062
------------------------------------------------------------------------------
-------------------------------------------------------------------------------
> ---------
Variable Sample | Treated Controls Difference S.E.
> T-stat
----------------------------+--------------------------------------------------
> ---------
lnauditfees Unmatched | 14.1198827 12.6593561 1.46052659 .013189094
> 110.74
ATT | 13.1068789 12.8982369 .208642045 .016916551
> 12.33
----------------------------+--------------------------------------------------
> ---------
Note: S.E. does not take into account that the propensity score is estimated.
psmatch2: | psmatch2: Common
Treatment | support
assignment | Off suppo On suppor | Total
-----------+----------------------+----------
Untreated | 0 8,031 | 8,031
Treated | 22,073 6,011 | 28,084
-----------+----------------------+----------
Total | 22,073 14,042 | 36,115
Summary for variables: _pscore
Group variable: big4
big4 | Mean
---------+----------
0 | .6192445
1 | .6130136
---------+----------
Total | .6161291
--------------------
Source | SS df MS Number of obs = 12,022
-------------+---------------------------------- F(1, 12020) = 152.12
Model | 130.833932 1 130.833932 Prob > F = 0.0000
Residual | 10338.1984 12,020 .860083061 R-squared = 0.0125
-------------+---------------------------------- Adj R-squared = 0.0124
Total | 10469.0323 12,021 .870895294 Root MSE = .92741
------------------------------------------------------------------------------
lnauditfees | Coefficient Std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
big4 | .208642 .0169166 12.33 0.000 .1754829 .2418012
_cons | 12.89824 .0119618 1078.28 0.000 12.87479 12.92168
------------------------------------------------------------------------------
Linear regression Number of obs = 12,022
F(1, 3507) = 40.81
Prob > F = 0.0000
R-squared = 0.0125
Root MSE = .92741
(Std. err. adjusted for 3,508 clusters in gvkey)
------------------------------------------------------------------------------
| Robust
lnauditfees | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
big4 | .208642 .0326587 6.39 0.000 .14461 .272674
_cons | 12.89824 .0281055 458.92 0.000 12.84313 12.95334
------------------------------------------------------------------------------
Linear regression Number of obs = 12,022
F(7, 3507) = 651.05
Prob > F = 0.0000
R-squared = 0.5259
Root MSE = .64279
(Std. err. adjusted for 3,508 clusters in gvkey)
------------------------------------------------------------------------------
| Robust
lnauditfees | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
big4 | .2397458 .0203788 11.76 0.000 .1997903 .2797013
lnassets | .5667943 .0095985 59.05 0.000 .5479751 .5856135
aturn | .1197562 .0144379 8.29 0.000 .0914486 .1480637
curr | -.0099505 .0049328 -2.02 0.044 -.0196219 -.0002791
lev | -.3848752 .0470814 -8.17 0.000 -.477185 -.2925654
roa | -1.012373 .0479707 -21.10 0.000 -1.106426 -.9183193
salesgrowth | -.0421716 .015676 -2.69 0.007 -.0729065 -.0114367
_cons | 9.965836 .0591581 168.46 0.000 9.849849 10.08182
------------------------------------------------------------------------------What happens when we match with replacement by removing the
noreplace option? We now see that 27,902 of the treatment
observations get matched, with only 182 remaining unmatched.
psmatch2 big4 lnassets aturn curr lev roa salesgr, n(1) common caliper(0.01) logit outc(lnauditfees)Logistic regression Number of obs = 36,115
LR chi2(6) = 11312.03
Prob > chi2 = 0.0000
Log likelihood = -13481.142 Pseudo R2 = 0.2956
------------------------------------------------------------------------------
big4 | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
lnassets | .9962023 .0126367 78.83 0.000 .9714347 1.02097
aturn | .156676 .021578 7.26 0.000 .1143839 .1989681
curr | .0536695 .0084254 6.37 0.000 .0371561 .070183
lev | -.7975957 .0782258 -10.20 0.000 -.9509154 -.644276
roa | -1.601652 .0914624 -17.51 0.000 -1.780915 -1.422389
salesgrowth | -.1897099 .0401083 -4.73 0.000 -.2683207 -.1110992
_cons | -4.454175 .0949572 -46.91 0.000 -4.640288 -4.268062
------------------------------------------------------------------------------
-------------------------------------------------------------------------------
> ---------
Variable Sample | Treated Controls Difference S.E.
> T-stat
----------------------------+--------------------------------------------------
> ---------
lnauditfees Unmatched | 14.1198827 12.6593561 1.46052659 .013189094
> 110.74
ATT | 14.108418 13.7265843 .381833746 .03721675
> 10.26
----------------------------+--------------------------------------------------
> ---------
Note: S.E. does not take into account that the propensity score is estimated.
psmatch2: | psmatch2: Common
Treatment | support
assignment | Off suppo On suppor | Total
-----------+----------------------+----------
Untreated | 0 8,031 | 8,031
Treated | 182 27,902 | 28,084
-----------+----------------------+----------
Total | 182 35,933 | 36,115 We can see that the matching with replacement produces only a very small difference in average propensity scores between the matched treatment and control samples. Importantly, we now also see that the mean propensity score of the treatment observations is now much higher than it was when matching without replacement.
Summary for variables: _pscore
Group variable: big4
big4 | Mean
---------+----------
0 | .8471856
1 | .8471862
---------+----------
Total | .8471859
--------------------In tabulating these means using tabstat, the weights in
variable _weights were used to obtain weighted means. This
is because the _weights variable contains information about
how often a particular control observation is used as a match with a
treatment observation. Note that for treatment observations the weight
is always equal to 1, and that it is equal to 1 for all observations
when matching without replacement. After matching with replacement, we
see a very different pattern:
psmatch2: weight of matched controls
-------------------------------------------------------------
Percentiles Smallest
1% 1 1
5% 1 1
10% 1 1 Obs 27,902
25% 1 1 Sum of wgt. 27,902
50% 1 Mean 1
Largest Std. dev. 0
75% 1 1
90% 1 1 Variance 0
95% 1 1 Skewness .
99% 1 1 Kurtosis .
psmatch2: weight of matched controls
-------------------------------------------------------------
Percentiles Smallest
1% 1 1
5% 1 1
10% 1 1 Obs 4,672
25% 1 1 Sum of wgt. 4,672
50% 2 Mean 5.972175
Largest Std. dev. 14.92343
75% 5 203
90% 12 243 Variance 222.7087
95% 22 314 Skewness 12.55932
99% 60 464 Kurtosis 272.3791
Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
_weight | 2,518 1.336378 .4725638 1 2The 27,902 treatment observations were matched to (only!) 4,672 control observations. Also, we observe huge variation in the weights. While a bit more than half of these control observations are used in a match only once or twice, several observations receive weights that are extremely large. One control observation is used even 464 times in a match with a treatment observation. Despite the benefits of matching with replacement, this extreme weighting is a clear drawback of matching with replacement because it makes the inferences much more dependent on a single observation.
Now, let’s interpret the results with respect to the outcome variable
lnauditfee.The psmatch2 results without
replacement suggested a difference of 0.2086 with a \(t\)-value of 12.33.This translates into a
fee premium of more than 23 percent for hiring a Big-4 auditor (\(e^{0.2086} − 1 = 0.2320\)).
The difference is much higher at 0.3818 with a \(t\)-value of 10.26 when matching with replacement, which highlights that the inferences here are very sensitive to the implementation of the matching procedure. There appears to be a clear ‘Big-4 premium’, but its magnitude is highly uncertain given the sensitivity to different design choices.
Linear regression Number of obs = 32,574
F(7, 5001) = 1667.99
Prob > F = 0.0000
R-squared = 0.6679
Root MSE = .64438
(Std. err. adjusted for 5,002 clusters in gvkey)
------------------------------------------------------------------------------
| Robust
lnauditfees | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
big4 | .2657333 .0220605 12.05 0.000 .222485 .3089816
lnassets | .5377854 .0059713 90.06 0.000 .526079 .5494919
aturn | .0986317 .0128063 7.70 0.000 .0735258 .1237376
curr | -.0032097 .0043102 -0.74 0.457 -.0116596 .0052403
lev | -.3001669 .0388904 -7.72 0.000 -.3764091 -.2239246
roa | -.8622942 .0422046 -20.43 0.000 -.9450337 -.7795547
salesgrowth | -.0478572 .0128421 -3.73 0.000 -.0730334 -.022681
_cons | 10.11482 .0477078 212.02 0.000 10.02129 10.20834
------------------------------------------------------------------------------The following regression results were obtained after matching with replacement:
(sum of wgt is 55,804)
Linear regression Number of obs = 32,574
F(7, 5001) = 460.90
Prob > F = 0.0000
R-squared = 0.5419
Root MSE = .70625
(Std. err. adjusted for 5,002 clusters in gvkey)
------------------------------------------------------------------------------
| Robust
lnauditfees | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
big4 | .3693434 .0622546 5.93 0.000 .2472971 .4913898
lnassets | .4771622 .0270295 17.65 0.000 .4241725 .5301519
aturn | .1399533 .0376796 3.71 0.000 .0660849 .2138218
curr | .0078425 .0106401 0.74 0.461 -.0130167 .0287017
lev | -.1971171 .1258805 -1.57 0.117 -.4438981 .0496639
roa | -.7540524 .0969409 -7.78 0.000 -.9440991 -.5640057
salesgrowth | -.2346081 .0890705 -2.63 0.008 -.4092254 -.0599908
_cons | 10.54304 .2020616 52.18 0.000 10.14691 10.93917
------------------------------------------------------------------------------We can see from these two estimations that after controlling for
other covariates, the estimates of the Big-4 premiums are 0.240 and
0.369, respectively. Interestingly, we also see that despite the
substantially larger sample when matching with replacement, the standard
error (and correspondingly, the \(t\)-value) is larger (smaller) for the
big4 coefficient in the second estimation (after matching
with replacement) than in the first estimation (after matching without
replacement).
This is an outcome of the duplication of information due to the reuse of control observations in the matching with replacement. This duplication — or, in other words, the fact that some control observations have extreme weights in determining the results — leads to a substantial reduction in the precision (i.e., increase in the variance) of the estimated treatment effect.