Difference-in-Differences in Edx

The Idea Behind Difference in Differences

Simple before and after studies often can’t account for changes over time that would have occurred even in the absence of a program. Things change, and not always in ways directly related to an intervention you might be studying.

Consider, for example, a program that provides nutritional supplements to poor schoolchildren. We certainly can’t compare the heights of kids who get the supplement with the heights of those who don’t get it as a means of assessing the effect of the program on child stature.

The program is deliberately targeted to poor kids who, in the absence of the supplements, are likely to be shorter than those who are ineligible. If we observed little difference in average heights between the treated and the untreated, it would be a mistake to infer that the supplements had no effect.

What to do? Well, let’s start with a before-and-after comparison of a group of people who take part in a program. And suppose we see a change in the average outcome over time. We know that some of the change could have been unrelated to the program itself. It would have happened in any case. But even so, the rest of any observed change over and above what would have happened could be attributed to the treatment.

We just don’t know how much if we are unable to find a group of untreated individuals who gain impacts. Within-without study– maybe we can construct one artificially. In particular, can we use the change in outcomes experienced by an untreated group to estimate the counterfactual for the treated group? This is the essence of a difference in differences approach to estimating the average impact of an intervention.

In regression

A contruction of the data before doing a diff and diff

Each row represents about in individual.

\(_i\) represents each individual

\(_t =\) O or 1 , \(_t\) representing the time of the treatment meaning = before or after the program.

With this data, we are ready to write an equation.

• Before the intervention, it was an RCT, we expect, before the intervention the outcome in treated and the control group to be the same. It is not the case in RCT. The coefficient \(\beta\) represents the outcome of after the intervention compared to before the intervention in the treated group, but in the absence of treatment

• The coefficient \(\gamma\),represents the outcome of after the intervention compared to before the intervention in the control group (it says what would have occured in time without the treatment)

• Therefore, the average outcome, before the intervention in the control group is \(\alpha\). After the intervention, the average outcome in the control group is \(\alpha + \gamma\).

• Also, there, the average outcome, before the intervention in the treated group is \(\alpha + \beta\). After the intervention, the average outcome in the treated group is \(\alpha + \beta + \gamma\), in the absence of the treatement. It is logical, since we need the difference in outcomes in the control group to tell us what would have occured in the absence of the treatement.

• But there’s one more term in the equation that combines the treatment and the post variables, \(\delta T * P\) . This is called the interaction term. To explain : the average outcome, in the treatment group after the intervention is \(\alpha + \beta + \gamma + \delta\).

• We can see that \(\delta\) captures the extent to which the outcome for treated individuals differs from what it would have been if the treatment had not taken place that equals \(\alpha + \beta + \gamma\) ,again, under the parallel trends assumption. This is on this value that we should a t-test : For example, if the value 0 lies outside the 95% confidence interval around \(\delta\) then we will reject the null hypothesis that the treatment has no effect at the 5% level.

The overal equation is written :

\(Y_{it} = \alpha + \beta T_{it} + \gamma P_{it} + \delta T_{it} * P_it + \epsilon_{it}\)

Difference in Differences Without Time

But the same technique can be used with cross sectional data. That is data collected at just one point in time. Example :

• We assume that level of school attendance in grades in between boys and girls is the same from 5th to 6th

• The intervention is : giving a bike to girls between 5th to 6th grade

• Then the difference-in-difference estimate of the effect of giving a bike on attendance is \(= (girl \ attendance \ in \ 6th \ grade - boy attendance in 6th grade) - (girl attendance in 5th grade- boy attendance in 5th grade)\)

Difference in Difference in Differences

An alternative approach is to use an otherwise comparable sample observed at the same two times before and after the treatment and made up of the same two kinds of people, boys and girls, but in which none of the individuals actually subject to the treatment. (like in a Placebo test)

Watch the video to know how to structure the dataset and the equation.

One problem with a triple difference approach is power. Every time you add another difference, you roughly double the size of the sample you need to get the same power. Alternatively, if you don’t double the sample size, then the estimate of program impact might be very imprecise.

Imperfect compliance and attrition

While the conceptual parallels are clear and instructive, there are many differences between such science experiments and the field experience in which we are interested. A big one shared by all clinical medical trials as well is that the interventions we assess involve people. And unlike test tubes, people get to make choices.

Amongst other things,

• they choose whether to sign up for a study,whether to comply with the treatment to which they are assigned,

• whether to answer questionnaires,

• and if so whether to answer truthfully, and whether to continue to take part in the study over time.

Imperfect compliance

Non compliance in the control group : some find a way to get treated eventhough they were assigned to treatment Non compliance in the treated group : some are absent during the treatment phase (after the baseline survey)

Intention to Treat :

• Impact it has on treated only because of the treatment (the LATE, the local average treatment effect)

• Impact it has on untreated despite the offer (we assume there is no impact)

• Impact it has on the people treated anyway (they would have been treated without being offered the treatment)

On overall, it measures the impact of being offered the treatment and taking it up if you desire

Measuring compliance

In both groups, they are the never treated : the idea is that they never get treated, whether they are offered the treatment as part of the study or not.

It’s important to recognize that this never treated group is not necessarily a random sample of the treatment group. The people in this group either choose not to be treated based on their own preferences, or are confronted by some kind of constraint to being treated that others did not face.

The likelyhood of never being treated is the same in the control and the treated groups (e.g. same chance to face a difficulty that makes the person unavailable)

There is the same logic for the always treated (it is for the same reason, for the same characteristics that they will be all the time treated in both groups)

The compliers : third group who take up the treatment because they are offered it under our program, but who wouldn’t have otherwise.

Compliers = 1 - never treated - always treated (or \(\phi_C = 1 - \phi_N -\phi_A\)). This is the fraction of people in the treatment group who are treated because of the existence of the program and the study. (Of course there is a similar fraction of people in the control group who would have been treated if they had been offered the treatment.) It is called the compliance rate in the sample of people from both groups.

Example : In an experiment in Kenya to test the impact of access to a widely available bank account accessible by mobile phone, about 33% of members of the control group had an account, while some 60% of the treatment group did. Thus \(\phi_A\) was about 0.33, \(\phi_N\) was about 0.4, and \(\phi_C\), the compliance rate, was about 0.27.

Measuring Impact with Imperfect Compliance – Calculating the ITT and LATE