Lab specs can be found here
Question: Recreate table 1 shown below,
inserting values appropriately for the three empty colums: (i) The
column labeled \(\delta_i\): please
enter each patient’s treatment effect (ii) The column labeled \(D\): the optimal treatment for this patient
(iii) The column labeled \(Y\): the
observed outcomes. Calculate the average treatment effect (ATE) and the
average treatment effect for the treated (ATT) when comparing the
outcome of the ventilators treatment with that of the
bedrest treatment and comment as to which type of
intervention is more effective on average. Finally, explain under which
conditions might SUTVA be violated
for treatments of covid-19 in the scenario described above.
# Write your code here
data <- tribble(
~patient, ~`Y(0)`, ~`Y(1)`, ~Age,
1, 10, 1, 29,
2, 5, 1, 35,
3, 4, 1, 19,
4, 6, 5, 45,
5, 1, 5, 65,
6, 7, 6, 50,
7, 8, 7, 77,
8, 10, 7, 18,
9, 2, 8, 85,
10, 6, 9, 96,
11, 7, 10, 77
)
data <- data %>%
mutate(
delta = `Y(1)` - `Y(0)`,
D = if_else(`Y(1)` > `Y(0)`, 1, 0),
Y = if_else(D == 1, `Y(1)`, `Y(0)`)
)
data %>% gt()| patient | Y(0) | Y(1) | Age | delta | D | Y |
|---|---|---|---|---|---|---|
| 1 | 10 | 1 | 29 | -9 | 0 | 10 |
| 2 | 5 | 1 | 35 | -4 | 0 | 5 |
| 3 | 4 | 1 | 19 | -3 | 0 | 4 |
| 4 | 6 | 5 | 45 | -1 | 0 | 6 |
| 5 | 1 | 5 | 65 | 4 | 1 | 5 |
| 6 | 7 | 6 | 50 | -1 | 0 | 7 |
| 7 | 8 | 7 | 77 | -1 | 0 | 8 |
| 8 | 10 | 7 | 18 | -3 | 0 | 10 |
| 9 | 2 | 8 | 85 | 6 | 1 | 8 |
| 10 | 6 | 9 | 96 | 3 | 1 | 9 |
| 11 | 7 | 10 | 77 | 3 | 1 | 10 |
result <- data %>% summarize(Average_treatment_effect = mean(`Y(1)` - `Y(0)`), .by = D) %>%
mutate(D = factor(D, levels = c(0, 1), labels = c("Untreated", "Treated")))
overall_mean <- data.frame(
D = "Overall",
Average_treatment_effect = mean(data$delta)
)
bind_rows(result, overall_mean) %>% gt()%>%
fmt_number(
columns = c(Average_treatment_effect),
decimals = 3
)| D | Average_treatment_effect |
|---|---|
| Untreated | −3.143 |
| Treated | 4.000 |
| Overall | −0.545 |
Answer:
\(\text{ATE} = E[Y(1)] - E[Y(0)] = \frac{1}{11} \sum_{i=1}^{11} (Y_i(1) - Y_i(0)) = -0.545\)
The overall ATE is -0.545. This suggests that, on average, the treatment (ventilators) has a slight negative effect on the survival outcome across all patients. The bedrest intervention is more effective on average.
\(\text{ATT} = E[Y(1) | D=1] - E[Y(0) | D=1] = \frac{1}{4} \sum_{i: D_i=1} (Y_i(1) - Y_i(0)) = 4\)
For the treated group, the intervention (ventilators) results in an average increase of 4 years of post-treatment survival.
\(\text{ATU} = E[Y(1) | D=0] - E[Y(0) | D=0] = \frac{1}{7} \sum_{i: D_i=0} (Y_i(1) - Y_i(0)) = -3.14\)
For the untreated group, the counterfactual scenario where they received the intervention would have resulted in a decrease of 3.14 years of post-treatment survival.
Violation of SUTVA
Question: Calculate the simple difference in
outcomes (SDO), showing the details of your calculation. Is
the SDO a good estimation for the ATE? Finally, check
whether the SDO is equal to the sum of the ATT
and the selection bias, \(E[Y(0)|T=1] -
E[Y(0)|T=0]\).
data %>% summarize(Mean_outcome = mean(Y), .by = D) %>% gt() %>%
fmt_number(
columns = c(Mean_outcome),
decimals = 3
)| D | Mean_outcome |
|---|---|
| 0 | 7.143 |
| 1 | 8.000 |
data %>% summarize(Expected_untreated_outcomes = mean(`Y(0)`), .by = D) %>% gt() %>%
fmt_number(
columns = c(Expected_untreated_outcomes),
decimals = 3
)| D | Expected_untreated_outcomes |
|---|---|
| 0 | 7.143 |
| 1 | 4.000 |
Answer:
Simple Difference in Outcomes (SDO)
\(\text{SDO} = E[Y | D=1] - E[Y | D=0]\)
Treated group (D = 1): \(\frac{1}{4}(5 + 8 + 9 + 10) = \frac{1}{4}(32) = 8\)
Untreated group (D = 0): \(\frac{1}{7}(10 + 5 + 4 + 6 + 7 + 8 + 10) = \frac{1}{7}(50) \approx 7.14\)
\(\text{SDO} = 8 - 7.14 \approx 0.86\)
Compared to the ATE of -0.545, the SDO differs significantly, indicating that the SDO is not a good estimate for the ATE. This discrepancy suggests the presence of confounding factors and selection bias.
Selection Bias
\(\text{Selection Bias} = E[Y(0) | D=1] - E[Y(0) | D=0]\)
Treated group: \(\frac{1}{4}(1 + 2 + 6 + 7) = \frac{1}{4}(16) = 4\)
Untreated group: \(\frac{1}{7}(10 + 5 + 4 + 6 + 7 + 8 + 10) = \frac{1}{7}(50) \approx 7.14\)
\(\text{Selection Bias} = 4 - 7.14 = -3.14\)
The selection bias of -3.14 indicates that the untreated group (patients who received bed rest) had better potential outcomes compared to the treated group (patients who received ventilators) before any treatment was administered. This means that patients who received ventilators were initially in worse condition.
SDO = ATT + Selection Bias
\(\text{ATT} + \text{Selection Bias} = 4 - 3.14 \approx 0.86 = SDO\)
The SDO of 0.86 is indeed equal to the sum of the ATT (4) and the selection bias (-3.14).
Question: Compare the treatment effect for both groups: for those treated with a ventilator and for those treated with bedrest. What explains the difference in the average effect? Now compare all four measures of effects. What are the advantages and disadvantages of each? Is the ATE equal to the mean of the ATU and the ATT? Why or why not?
data %>%
summarise(
ATT = mean(`Y(1)`[D == 1]) - mean(`Y(0)`[D == 1]),
ATU = mean(`Y(1)`[D == 0]) - mean(`Y(0)`[D == 0]),
ATE = (sum(D) / n() * (mean(`Y(1)`[D == 1]) - mean(`Y(0)`[D == 1]))) +
((n() - sum(D)) / n() * (mean(`Y(1)`[D == 0]) - mean(`Y(0)`[D == 0]))),
SDO = mean(Y[D == 1]) - mean(Y[D == 0]),
Mean_ATT_ATU = (mean(`Y(1)`[D == 1]) - mean(`Y(0)`[D == 1]) + mean(`Y(1)`[D == 0]) - mean(`Y(0)`[D == 0])) / 2
) %>% t() %>% as.data.frame() %>% rename(Value = V1) %>% rownames_to_column(var = "Measure") %>% gt() %>%
fmt_number(columns = c(Value), decimals = 3)| Measure | Value |
|---|---|
| ATT | 4.000 |
| ATU | −3.143 |
| ATE | −0.545 |
| SDO | 0.857 |
| Mean_ATT_ATU | 0.429 |
Answer:
Treatment effect for both groups
Treated with Ventilator:\(\text{ATT} = 4\)
Treated with Bedrest: \(\text{ATU} = -3.14\)
The difference between ATT and ATU can be explained by selection bias. Patients who received ventilators were initially in worse health conditions compared to those who received bedrest.
Compare all four measures of effects:
\(\text{ATE} = E[Y(1)] - E[Y(0)] = -0.545\)
Advantages: Provides a single measure of treatment effect across the whole population.
Disadvantages: Can be misleading if there is significant heterogeneity in treatment effects or selection bias; relies on strong assumptions for estimation, cannot be measured directly in real life.
\(\text{ATT} = E[Y(1) | D=1] - E[Y(0) | D=1] = 4\)
Advantages: Reflects actual effect on those who received the treatment.
Disadvantages: Not generalizable to the untreated population. Also cannot be measured directly in real life.
\(\text{ATU} = E[Y(1) | D=0] - E[Y(0) | D=0] = -3.14\)
Advantages: Provides insight into potential effects on untreated individuals.
Disadvantages: Not generalizable to the treated population, cannot be measured directly in real life.
\(\text{SDO} = E[Y | D=1] - E[Y | D=0] = 0.86\)
Advantages: Simple and straightforward to compute.
Disadvantages: Prone to bias and confounding, does not account for selection bias.
Is the ATE Equal to the Mean of the ATU and the ATT?
No, the ATE is not simply the mean of the ATU and ATT. Instead, it is a weighted average of ATT and ATU based on the proportions of treated and untreated individuals in the population
\(\text{Mean of ATT and ATU} = \frac{4 - 3.14}{2} = \frac{0.86}{2} = 0.43 \neq ATE\)
\(\text{ATE} = \frac{4}{11} \times \text{ATT} + \frac{7}{11} \times \text{ATU} = \left( \frac{4}{11} \times 4 \right) + \left( \frac{7}{11} \times (-3.14) \right) = \frac{16 - 21.98}{11} = -0.5436\)
The following exercises demonstrate that regression is a useful tool to estimate average outcomes and treatment effects in the different groups. Notice that in contrast to the role that regressions play in traditional statistics, here standard errors and significance are not of primary concern. Instead, we are interested in using regression to calculate average effects proper.
Question: Calculate the outcome, conditional on
getting the bedrest treatment \(\mathbb{E}[Y|D=0]\). Now estimate the
following regression, comparing the coefficients \(\alpha\) and \(\delta\) to the statistics you’ve
previously calculated. What did you find? How would you explain these
finding?
##
## Call:
## lm(formula = Y ~ D, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.143 -1.643 0.000 1.500 2.857
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.1429 0.8624 8.282 1.68e-05 ***
## D 0.8571 1.4302 0.599 0.564
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.282 on 9 degrees of freedom
## Multiple R-squared: 0.03838, Adjusted R-squared: -0.06847
## F-statistic: 0.3592 on 1 and 9 DF, p-value: 0.5637Answer:
\(\text{conditional_outcome_bedrest} = \mathbb{E}[Y | D=0] = \frac{1}{7} \sum_{i:D_i=0} Y_i = 7.14\)
Regression Model: \(Y_i = \alpha + \delta \cdot D_i + \epsilon_i\)
\(\alpha\): This value should be equal to the conditional outcome \(\mathbb{E}[Y | D=0]\) because it represents the intercept of the regression model (the average outcome for the untreated group).
\(\delta\): The slope represents the difference in outcomes between the treated (ventilators) and untreated (bedrest) groups, which should be equal to the SDO.
Question: Now estimate the same regression, but this time, controlling for age, again comparing, the coefficient \(\delta\) to the statistics you’ve previously calculated. What did you find? How do you explain these results?
##
## Call:
## lm(formula = Y ~ D + Age, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.7390 -1.6630 -0.0858 1.3839 3.2811
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.35543 1.90736 3.332 0.0104 *
## D 0.01419 2.33997 0.006 0.9953
## Age 0.02019 0.04309 0.469 0.6519
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.388 on 8 degrees of freedom
## Multiple R-squared: 0.06407, Adjusted R-squared: -0.1699
## F-statistic: 0.2738 on 2 and 8 DF, p-value: 0.7673Answer:
Regression Controlling for Age: \(Y_i = \alpha + \delta \cdot D_i + \gamma \cdot \text{Age}_i + \epsilon_i\)
Without controlling for age, the treatment effect (\(\delta\)) was 0.8571, same as SDO. Controlling for age, the treatment effect (\(\delta\)) dropped to 0.01419.
The significant reduction in \(\delta\) suggests that the initial observed treatment effect was largely due to age differences between the treatment groups. Older patients, who generally have worse health outcomes, might be more frequently assigned to the ventilator group.
Question: Estimate the following three regression models. The first model is the same as the one above. The second equation is the auxiliary regression of \(D\) onto \(X_{age}\). The third equation regresses \(Y\) onto \(\tilde{D}\) which is the residual from the second equation. Compare the coefficient on \(D\) from the first equation to the coefficient on \(\tilde{D}\) in the third equation. What does this tell you about how to interpret multivariate regressions?
##
## Call:
## lm(formula = Y ~ D, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.143 -1.643 0.000 1.500 2.857
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.1429 0.8624 8.282 1.68e-05 ***
## D 0.8571 1.4302 0.599 0.564
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.282 on 9 degrees of freedom
## Multiple R-squared: 0.03838, Adjusted R-squared: -0.06847
## F-statistic: 0.3592 on 1 and 9 DF, p-value: 0.5637##
## Call:
## lm(formula = D ~ Age, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.68663 -0.16289 0.04442 0.17433 0.48323
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.403323 0.236116 -1.708 0.12178
## Age 0.014155 0.003925 3.606 0.00569 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.3401 on 9 degrees of freedom
## Multiple R-squared: 0.591, Adjusted R-squared: 0.5455
## F-statistic: 13 on 1 and 9 DF, p-value: 0.005694data <- data %>% mutate(D_residual = residuals(model2))
model3 <- lm(Y ~ D_residual, data = data)
summary(model3)##
## Call:
## lm(formula = Y ~ D_residual, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.4565 -1.9522 0.5426 2.0429 2.5456
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.45455 0.70156 10.626 2.15e-06 ***
## D_residual 0.01419 2.28040 0.006 0.995
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.327 on 9 degrees of freedom
## Multiple R-squared: 4.302e-06, Adjusted R-squared: -0.1111
## F-statistic: 3.872e-05 on 1 and 9 DF, p-value: 0.9952Answer:
First Model: The initial estimate suggests that ventilators have a significant positive effect on survival. However, this model does not control for age, which can bias the results.
Second Model: The positive and statistically significant coefficient for age in the second regression shows that age is an important predictor of receiving the ventilator treatment. Older patients are more likely to receive ventilators.
Third Model: When we regress \(Y\) on the residuals from the second model (\(\tilde{D_i}\)), which controls for age, the treatment effect becomes almost zero. This indicates that the initial observed treatment effect was largely due to the confounding influence of age, failing to control for age inflates the apparent effect of ventilators on survival.
The coefficient on \(D_{residual}\) from the third regression (\(\delta_3 = 0.01419\)) matches the coefficient on \(D\) from the model controlling for age (\(\delta_{Age} = 0.01419\)), as both methods account for age’s confounding effect on treatment assignment.
Including age in the regression isolates the true effect of treatment \(D\), providing an accurate estimate \(\delta_{Age}\). By first regressing \(D\) on age, we remove age’s influence, leaving residuals (\(\tilde{D}\)) that represent the variation in \(D\) not explained by age. Regressing \(Y\) on these residuals also isolates the treatment effect, controlling for age, resulting in \(\delta_3\).
Properly controlling for confounding variables is crucial for valid and reliable estimates in multivariate regression analyses. Including confounders directly in the regression model or analyzing residuals after accounting for confounders both help ensure the estimated effect is not biased.