Research Intensive Day
Simulation Study
Examples of simulation studies
Only for the ATE
\(10,000\) reps
metrics: bias, variance and MSE
\(N\in \{1000,10000, 100000\}, \; n_{S=1}\in\{500,1000,5000,10000\}\)
8 simulation settings
Probability of selection was a constant, i.e. no confounding
They include the uncertainty of their simulations by providing the Monte Carlo error (empirical standard deviation) of the \(10,000\) simulations
Targeting the CATE, no generalizing
200 reps
Metrics: RMSE and the incremental gains curve.
For each of the 200 data-sets, they have a training and test set. One for training the Meta-Learner, and one for evaluating MSE and other metrics
4 outcome models
\[ Y^A = f(X) + \tau(X)A+\epsilon \]
\[\epsilon\sim N(0,\sigma_0^2)\]
Model Description Outcome model 1 No HTE \(\tau(X) = \delta\) 2 Linear \(\tau(X) = \gamma_0+\sum_{k=1}^{10}\gamma_kx_{ik}\) 3 Nonlinear + interactive \(\tau(X) = \gamma_0 +\gamma_1x_{i1}^3+\gamma_{2}\cos(x_{i2}x_{i3})\) 4 High-dimensional covariates Model 2 with 1000 covariates
These authors look at comparing methods for the PATE at varying amounts of sampling bias.
100 simulations for each combination of design factors (1000 simulations total)
Metrics: bias RMSE, (empirical) 95% confidence interval coverage
\(N\in\{1000,5000,9000,13000\}\)
\(\mathbb P(S=1|X_k)=(1+\exp(-(\beta_0+\beta_kf(X_k)))^{-1}\)
\(\beta_0\) was varied such that the percentage of target population included in the trial was 2%, 10%, 18%, 26%
| Model | Description | Outcome model |
|---|---|---|
| 1 | Linear | \(\tau(x) = 1+2.85X_1+2.98X_3\) |
| 2 | Quadratic | \(\tau(x) = 1+2.85X_1+2.98X_3-0.5X_7+1.5X_7^2\) |
Should we have training and test data, and how large should they be?
Do we want to investigate parametric vs non-parametric methods?
Do we want to investigate stabilized weights vs non-stabilized weights?
SuperLearner and/or random forests?
Extending Learners using Conformal Inference
(Angelopoulos and Bates, n.d.)
Useful tutorials: playlist
General steps:
Identify a heuristic notion of uncertainty using the pre-trained model
Define a score function \(s(x,y)\in \mathbb R\) . Larger scores means worse agreement between \(x\) and \(y\).
Compute \(\hat q\) as the \(\frac{\lceil(n+1)(1-\alpha)\rceil}{n}\) quantile of the calibration scores \(s_1=s(X_1, Y_1), \dots, s_n=s(X_n, Y_n)\)
Use this quantile to form the new prediction sets for new examples
\[ \mathcal C(X_{test})=\{y:s(X_{test},y)\le \hat q\} \]
The usefulness of the prediction sets is determined by the score function.
Conformalized Quantile Regression
The first step is to use a quantile regression as their base model. Since this model may provide inaccurate quantiles, we need to conformalize the procedure.
Score function:
\[ s(x,y)=\max\{ \hat t_{\alpha/2}(x)-y, y-\hat t_{1-\alpha/2}(x)+\hat q\} \]
Quantile regression can be added on top of any base model, even ML models, just by changing the loss function to a quantile loss, e.g. instead of MSE. It can also be thought of as a generalization of the L1-norm regression.
The quantile loss is:
\[ l(y, \hat t_\gamma) = \begin{cases} \gamma\cdot(y-\hat t_\gamma) & ,\hat y\le y\\ (1-\gamma)\cdot(\hat t_\gamma-y) &, \hat y>y \end{cases} \]
Examples have previously used quantile random forests and quantile neural networks. However, they can sometimes be overly conservative. The nominal quatiles can be tuned as additional hyper-parameters in cross validation.
Conformalizing Scalar Uncertainty Estimates
Choose to model \(Y_{test}|X_{test} = x \sim \mathcal N(\mu(x), \sigma(x))\) and we have \(\hat f(x)\) and \(\hat \sigma (x)\) trained to maximize the likelihood of the data with respect to \(\mathbb E[Y_{test}|X_{test}]\) and \(\sqrt{Var[Y_{test}|X_{test}=x]}\) respectively. \(\hat f(x)\) will be used as the point prediction and \(\hat \sigma (x)\) will be the uncertainty. Then use conformal prediction to turn this into a rigorous prediction intervals \(\hat f(x)\)
Conformal inference for ITEs
Conformal Inference of Counterfactuals and Individual Treatment Effects
From what I understand so far, \(W\) will be the IPW for settings with covariate shift due to treatment selection. In which case we can use IPSW (or similar) instead?
“In general, if \(Q_X\) is the covariate distribution in another population, as in the context of generalizability/transportability/external validity, then”
\[ w_1(x)=\frac{dQ_X(x)}{dP_{X|T=1(x)}(x)}=\frac{dQ_X(x)}{dP_X(x)}\frac{\mathbb P(T=1)}{\pi(x)} \]
Where \(Q_X\) is the covariate distribution of the target population.
However, their method is not guaranteed to cover the CATE, only the ITE. But their experiments cover both, although might be more conservative with the CATE. In this paper, their method was the only one to achieve coverage of CATEs and ITEs.
We could also use the BART method described in the since it seems to work well when errors are homoscedastic and it is guaranteed (theoretically, given assumptions) to cover both the CATE and the ITE.
Base learners: RF or neural net?
Fairness Metrics
Counterfactual Equalized Odds (potential outcome)
(Mishler, Kennedy, and Chouldechova 2021)
CEO is satisfied by a predictor of treatment \(D\) if \(D\perp\!\!\! \perp G|Y^0\)
If \(D,G,Y\) are binary then satisfying CEO is equivalent to having the counterfactual error rates to be the same between both groups. The error rates are defined as:
\[ cFPR(D,g) = \mathbb P(D=1|Y^0=0,G=g) \]
\[ cFNR(D,g) = \mathbb P(D=0|Y^0=1,G=g) \]
We can find estimators of \(cFPR\) and \(cFNR\)
\[ \widehat{cFPR}(D,g) = \frac{\mathbb P_n[I(G=g)D(1-\hat \phi)]}{P_n[I(G=g)(1-\hat\phi)]} \]
\[ \widehat{cFPR}(D,g) = \frac{\mathbb P_n[I(G=g)(1-D)\hat \phi]}{P_n[I(G=g)\hat\phi]} \]
where \(\hat \phi\) is an estimate of \(\phi = \frac{1-D}{1-\pi}(Y-\mu_0)+\mu_0\), the un-centered efficient influence function of \(Y^0\).
Currently no extension to continuous \(Y\).
Counterfactual Equalized Odds (causal contrast)
The above fairness metric targets error rates in comparison to the baseline potential outcome \(Y^0\), the outcome under no treatment. This metric might be more suitable when treatment is cheap so that we would want to treat (\(D=1\)) any of those that would have a worse outcome if they were to not be treated (e.g. \(Y^0=0\) if \(Y\) is an indicator of survival). However, this does not mean that these individuals will have an improved outcome if treated, their \(Y^1\) could be either 0 or 1.
Instead, we might want to target the error rate in comparison to the treatment effect \(\tau(X)=Y^1-Y^0\). This means we want to find the error rate of not giving the optimal treatment. So we should treat \((D=1)\) if we expect the effect of treatment to be positive (\(\tau(X)>0\)). This means we only target individuals that we believe to benefit from the treatment.
If we continue with the example of \(Y\) being a binary indicator of survival, we can look at pairs of \((Y^1,Y^0)\) such that we have 4 pairs: \((0,0), (0,1), (1,0), (1,1)\). The CEO with baseline \(Y^0\) would look to assign treatment to individuals with potential outcome pairs \((0,0)\) and \((1,0)\), however only those with potential outcome pair \((1,0)\) would actually benefit from treatment since their \(\tau(X)=Y^1 - Y^0 = 1\). The CEO based on the causal contrast aims to be a more useful metric when treatment decisions have to be more resource efficient.
We attempt to modify the CEO in (Mishler, Kennedy, and Chouldechova 2021) to target the fairness of decisions compared to the the continuous causal risk difference \(\tau(x)\). We convert a discrete or continuous \(\tau(x)\) to a binary treatment prediction that represents the true best treatment for each individual, denoted as \(D^\tau\).
\[ D^\tau_i=\begin{cases} 0 & \tau(X_i)\le\delta, \\ 1 & \tau(X_i) > \delta \end{cases} \] For now we assume \(\delta=0\).
The definition of the counterfactual error rates can now be rewritten as:
\[cFPR(D,g) = \frac{\mathbb E [I\{G=g\}D(1-D^\tau)]}{\mathbb E[I\{G=g\}(1-D^\tau)]}\] \[cFNR(D,g) = \frac{\mathbb E [I\{G=g\}(1-D)D^\tau]}{\mathbb E[I\{G=g\}D^\tau]}\] Question: Does \(\hat\phi = \hat\mu_1 - \hat\mu_0 + \frac{A(Y-\hat\mu_1)}{\hat\pi} - \frac{(1-A)(Y-\hat\mu_0)}{1-\hat\pi}\) identify \(D^\tau\)?
Fairness metric for continuous outcomes
“Fair and Robust Estimation of Heterogeneous Treatment Effects for Policy Learning.”
This paper uses notions of fairness introduced by (Mishler and Kennedy 2021) to find an approximation of \(\tau\) by finding a projection onto a finite-dimensional model space subject to specific fairness constraints. They wish to optimize the following:
\[ \text{minimize}_{\beta\in\mathbb R^k} \; \mathcal R_{MSE}(\beta;b):=\mathbb E[\{Y^1-Y^0-\beta^Tb(X)\}^2] \]
subject to \[ |\mathbb E \{uf_j(Z)\beta^Tb(X)\}|\le \delta_j\]
where \(uf_j:\mathcal Y^2\times \{0,1\}\rightarrow \mathbb R\) are fairness measures, \(\beta\) is the coefficients of the best-fitting function of \(\tau\) spanned by the basis functions \(b(W)\). For example, we can apply constraints for independence by having
\[ uf_j(G)=\frac{1-G}{\mathbb E(1-G)}- \frac{S}{\mathbb E(G)} \]
This leads to \(|\mathbb E[\beta^Tb(X)|S=0]-\mathbb E[\beta^Tb(X)|S=1]|\le \delta_j\)
We can also have balance for the positive class (consistent with equal opportunity, looks similar to CEO?)
\[ uf_j(G,X) = \frac{(1-S)I(\tau(X)>0)}{\mathbb E[(1-S)I(\tau(X)>0]}- \frac{SI(\tau(X)>0)}{\mathbb E[SI(\tau(X)>0)]} \]
Which leads to \(|\mathbb E[\beta^Tb(X)|S=0, \tau(X)>0]-\mathbb E[\beta^Tb(X)|S=1, \tau(X)>0]|\le \delta_j\).
“Fairness Measures for Regression via Probabilistic Classification”
Fairness for regression based on (Steinberg, Reid, and O’Callaghan 2020)
In contrast to the vast algorithmic fairness literature which has mainly focused on classification tasks, this paper looks at measuring unfairness in regression problems. They propose a method for determining independence, separation and sufficiency conditions between a regression prediction R and the sensitive attribution G by performing logistic regression.
| Independence | Separation | Sufficiency |
|---|---|---|
| \(R\perp G\) | \(R\perp G|Y\) | \(Y\perp G|R\) |
For example, in the case of independence, if the model is good at predicting \(G\) from \(R\) then there is discrimination occurring.
They rewrite the three conditional independence statements as density ratios. Wide applicability, such as mutual information measures.
Interpretations:
independence: how much more predictive of A is S over base distribution \(P(A)\)
Separation & sufficiency: how much more predictive of A is the joint distribution of Y and S vs just the marginals Y or S
Classifiers and data need to be sufficiently expressive and well calibrated to model \(P(A|\cdot)\) with little error. Other limitations include, limited extension to categorical A. Empirical estimation can also lead to instability because of vanishing probabilities in the denominator.
Alternatively, we can use normalized Mutual Information Estimation, which can handle categorical sensitive attributes, and improves upon the limitations mentioned above.
Note: This method may be useful for calculating the fairness functions in the above, using \(\hat\tau(X)\) instead of \(\beta^Tb(X)\)?
Fair Inference on Outcomes
Proposes a method for distinguishing between indirect and direct unfairness along certain causal pathways. They then develop a method for determining a fair outcome model through constrained optimization techniques.
An example of direct vs indirect effect: gender should not directly influence hiring decisions, but may influence decisions indirectly by being correlated with secondary influential characteristics.
Definitions
Mediation: denoted as \(Y(a,M(a'))\). The outcome \(Y\) if \(A\) were set to \(a\), but \(M\) were set to the value it would have been if \(A=a'\).
The natural direct effect (NDE): \(\mathbb E[Y(a,M(a'))] - \mathbb E[Y(a')]\)
The natural indirect effect (NIE): \(\mathbb E[Y(a)] - \mathbb E[Y(a,M(a'))]\).
Then \(ATE = NDE+NIE\).
Assumptions
To identify \(\mathbb P(Y(a,M(a'))\) we need the following assumptions
Consistency
sequential ignorability, \(Y(a,m)\perp M(a')|C\)
\(Y(a,m)\perp A,M|C\) and \(M(a')\perp A|C\)
Then the NDE can be identified using the mediation formula:
\[\sum_{C,M}\mathbb E[Y|a,M,C]-\mathbb E[Y|a',M,C]p(M|a',C)p(C)\]
They argue that fairness metrics such as equalized odds are associative and as can exhibit spurious correlations via other features.
Their intuition says that fair decisions are those that would stay the same if the only attribute that changed is the sensitive attribute. Similar to the notion of counterfactual fairness by (Kusner et al. 2017). However, in our case, race or gender might have a true direct effect on the outcome, which might not be unfair, and so would this notion of fairness work? Moreover, if the sensitive attribute has an effect on the outcome through a hidden proxy, is this identifiable? They then argue that this is a domain specific issue, and so we should appropriately consider all path-specific effects and determine which are fair or not.
Their fairness metric is to do with the path specific effect. Unsure whether it is appropriate for determining the fairness of policy rules.
Causal Concepts of Fairness and their Consequences
Not much attention has been made to the downstream consequences of causal fairness definitions. The authors consider two types of causal fairness. The first being the effect of decisions on counterfactual disparities, such as counterfactual equalized odds. The second type looking to ensure that decisions are not (indirectly or directly) dependent on an individual’s group membership, such as path specific fairness.
Definitions
Counterfactual predictive parity: \(Y(1)\perp A|D=0\)
Counterfactual equalized odds: \(D\perp A |Y(1)\)
Conditional principal fairness: \(D\perp A|Y(0), Y(1),W\)
Policies in which applicants are admitted randomly with probability \(b\) satisfies these 3 causal fairness definitions. This paper aims to consider causally fair policies that maximize utility.
Theorem 1 shows that any decision making policy that satisfies ceo, conditional principal fairness or path specific fairness is Pareto dominated. Meaning that there exists another policy that has higher utility. Pareto efficient is where the decisions \(d\) are feasible and not Pareto dominated.
However, this paper assumes that we wish to satisfy the fairness requirements exactly, without having a bound for which we want to keep the fairness metric within, plus they have a bound for budget, which we are not considering. Would these different assumptions change their results?
#Check what they propose as better alternatives? None really, I guess just to maximise both the diversity and welfare as a Pareto optimization?