Estimator under completely randomized experiment (CRE)

Jinsil Kim

2024.08.28

0. Notation

1. Estimator for the average treatment effect

2. The Sampling Variance of the Neyman Estimator

Theorem 2. The sampling variance of $\hat{\tau}^{\mathrm{dif}}=\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}$ is

$$ \mathbb{V}W\left(\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}\right)=\frac{S_c^2}{N{\mathrm{c}}}+\frac{S_t^2}{N_{\mathrm{t}}}-\frac{S_{t c}^2}{N} $$

Proof of Theorem 2 -> Appendix

3. Estimating the Sampling Variance

$$ \hat{\mathbb{V}}^{\text {neyman }}=\frac{s_c^2}{N_{\mathrm{c}}}+\frac{s_t^2}{N_{\mathrm{t}}} $$

Proof of Theorem 3 (brief version)

APPENDIX : Sampling Variance Calculations

First note that the average treatment effect is:

$$ \tau_{\mathrm{fs}}=\bar{Y}(1)-\bar{Y}(0)=\frac{1}{N} \sum_{i=1}^{N}\left(Y_{i}(1)-Y_{i}(0)\right) $$

And the standard estimator of $\tau_{\mathrm{fs}}$ is:

$$ \begin{aligned} \hat{\tau}^{\mathrm{dif}} & =\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}=\frac{1}{N_{\mathrm{t}}} \sum_{i=1}^{N} W_{i} \cdot Y_{i}^{\mathrm{obs}}-\frac{1}{N_{\mathrm{c}}} \sum_{i=1}^{N}\left(1-W_{i}\right) \cdot Y_{i}^{\mathrm{obs}} \ & =\frac{1}{N} \sum_{i=1}^{N}\left(\frac{N}{N_{\mathrm{t}}} \cdot W_{i} \cdot Y_{i}(1)-\frac{N}{N_{\mathrm{c}}} \cdot\left(1-W_{i}\right) \cdot Y_{i}(0)\right) \end{aligned} $$

For the variance calculations, define a centered treatment indicator $D_{i}$ as

$$ D_{i}=W_{i}-\frac{N_{\mathrm{t}}}{N}= \begin{cases}\frac{N_{\mathrm{c}}}{N} & \text { if } W_{i}=1 \ -\frac{\mathrm{N}{\mathrm{t}}}{N} & \text { if } W{i}=0\end{cases} $$

$\mathbb{E}\left[D_{i}\right]=0$, and $\mathbb{V}\left(D_{i}\right)=\mathbb{E}\left[D_{i}^{2}\right]=N_{\mathrm{c}} N_{\mathrm{t}} / N^{2}$.

Later we also need its cross moment, $\mathbb{E}\left[D_{i} \cdot D_{j}\right]$. For $i \neq j$ the distribution of this cross product is

$$ \operatorname{Pr}{W}\left(D{i} \cdot D_{j}=d\right)= \begin{cases}\frac{N_{\mathrm{t}} \cdot\left(N_{\mathrm{t}}-1\right)}{N \cdot(N-1)} & \text { if } d=N_{\mathrm{c}}^{2} / N^{2} \text{ (both treated}) \ 2 \cdot \frac{N_{\mathrm{t}} \cdot N_{\mathrm{c}}}{N \cdot(N-1)} & \text { if } d=-N_{\mathrm{t}} N_{\mathrm{c}} / N^{2} \text{ (only one treated)} \ \frac{N_{\mathrm{c}} \cdot\left(N_{\mathrm{c}}-1\right)}{N \cdot(N-1)} & \text { if } d=N_{\mathrm{t}}^{2} / N^{2} \text{ (both not treated}) \ 0 & \text { otherwise }\end{cases} $$

thereby leading to

$$ \mathbb{E}{W}\left[D{i} \cdot D_{j}\right]= \begin{cases}\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{2}} & \text { if } i=j \ -\frac{N_{\mathrm{t}} \cdot N_{\mathrm{c}}}{N^{2} \cdot(N-1)} & \text { if } i \neq j\end{cases} $$

In terms of $D_{i}$, our estimate of the average treatment effect is:

$$ \begin{align*} \bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}} & =\frac{1}{N} \sum_{i=1}^{N}\left(\frac{N}{N_{\mathrm{t}}} \cdot\left(D_{i}+\frac{N_{\mathrm{t}}}{N}\right) \cdot Y_{i}(1)-\frac{N}{N_{\mathrm{c}}} \cdot\left(\frac{N_{\mathrm{c}}}{N}-D_{i}\right) \cdot Y_{i}(0)\right) \ & =\frac{1}{N} \sum_{i=1}^{N}\left(Y_{i}(1)-Y_{i}(0)\right)+\frac{1}{N} \sum_{i=1}^{N} D_{i} \cdot\left(\frac{N}{N_{\mathrm{t}}} \cdot Y_{i}(1)+\frac{N}{N_{\mathrm{c}}} \cdot Y_{i}(0)\right) \ & =\tau_{f s}+\frac{1}{N} \sum_{i=1}^{N} D_{i} \cdot\left(\frac{N}{N_{\mathrm{t}}} \cdot Y_{i}(1)+\frac{N}{N_{\mathrm{c}}} \cdot Y_{i}(0)\right) \tag{A.1} \end{align*} $$

Because $\mathbb{E}{W}\left[D{i}\right]=0$ and all potential outcomes are fixed, the estimator $\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}$ is unbiased for the average treatment effect, $\tau_{\mathrm{fs}}=\bar{Y}(1)-\bar{Y}(0)$. Next, because the only random element in Equation (A.1) is $D_{i}$, the variance of $\hat{\tau}=$ $\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}$ is equal to the variance of the second term in Equation (A.1). Defining $Y_{i}^{+}=\left(N / N_{\mathrm{t}}\right) Y_{i}(1)+\left(N / N_{\mathrm{c}}\right) Y_{i}(0)$, the latter is equal to:

$$ \begin{equation*} \mathbb{V}{W}\left(\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}\right)=\mathbb{V}{W}\left(\frac{1}{N} \sum_{i=1}^{N} D_{i} \cdot Y_{i}^{+}\right)=\frac{1}{N^{2}} \cdot \mathbb{E}{W}\left[\left(\sum{i=1}^{N} D_{i} \cdot Y_{i}^{+}\right)^{2}\right] \tag{A.2} \end{equation*} $$

Expanding Equation (A.2), we get:

$$ \begin{aligned} & \mathbb{V}{W}\left(\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}\right)=\mathbb{E}{W}\left[\frac{1}{N^{2}} \sum_{i=1}^{N} \sum_{j=1}^{N} D_{i} D_{j} Y_{i}^{+} Y_{j}^{+}\right] \ & =\frac{1}{N^{2}} \sum_{i=1}^{N}\left(Y_{i}^{+}\right)^{2} \cdot \mathbb{E}{W}\left[D{i}^{2}\right]+\frac{1}{N^{2}} \sum_{i=1}^{N} \sum_{j \neq i} \mathbb{E}{W}\left[D{i} \cdot D_{j}\right] \cdot Y_{i}^{+} \cdot Y_{j}^{+} \ & =\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{4}} \sum_{i=1}^{N}\left(Y_{i}^{+}\right)^{2}-\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{4} \cdot(N-1)} \sum_{i=1}^{N} \sum_{j \neq i} Y_{i}^{+} \cdot Y_{j}^{+} \ & =\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{3} \cdot(N-1)} \sum_{i=1}^{N}\left(Y_{i}^{+}\right)^{2}-\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{4} \cdot(N-1)} \sum_{i=1}^{N} \sum_{j=1}^{N} Y_{i}^{+} \cdot Y_{j}^{+} \ & =\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{3} \cdot(N-1)} \sum_{i=1}^{N}\left(Y_{i}^{+}-\overline{Y^{+}}\right)^{2} \ & =\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{3} \cdot(N-1)} \sum_{i=1}^{N}\left(\frac{N}{N_{\mathrm{t}}} \cdot Y_{i}(1)+\frac{N}{N_{\mathrm{c}}} \cdot Y_{i}(0)-\left(\frac{N}{N_{\mathrm{t}}} \cdot \bar{Y}(1)+\frac{N}{N_{\mathrm{c}}} \cdot \bar{Y}(0)\right)\right)^{2}\ & = \frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{3} \cdot(N-1)} \sum_{i=1}^{N}\left(\frac{N}{N_{\mathrm{t}}} \cdot Y_{i}(1)-\frac{N}{N_{\mathrm{t}}} \cdot \bar{Y}(1)\right)^{2} \ &~~~ +\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{3} \cdot(N-1)} \sum_{i=1}^{N}\left(\frac{N}{N_{\mathrm{c}}} \cdot Y_{i}(0)-\frac{N}{N_{\mathrm{c}}} \cdot \bar{Y}(0)\right)^{2} \ &~~~ +\frac{2 \cdot N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{3} \cdot(N-1)} \sum_{i=1}^{N}\left(\frac{N}{N_{\mathrm{t}}} \cdot Y_{i}(1)-\frac{N}{N_{\mathrm{t}}} \cdot \bar{Y}(1)\right) \cdot\left(\frac{N}{N_{\mathrm{c}}} \cdot Y_{i}(0)-\frac{N}{N_{\mathrm{c}}} \cdot \bar{Y}(0)\right) \ &= \frac{N_{\mathrm{c}}}{N \cdot N_{\mathrm{t}} \cdot(N-1)} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)\right)^{2}+\frac{N_{\mathrm{t}}}{N \cdot N_{\mathrm{c}} \cdot(N-1)} \sum_{i=1}^{N}\left(Y_{i}(0)-\bar{Y}(0)\right)^{2} \ &~~~ +\frac{2}{N \cdot(N-1)} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)\right) \cdot\left(Y_{i}(0)-\bar{Y}(0)\right) \tag{A.3} \end{aligned} $$

Recall the definition of $S_{t c}^{2}$, which implies that

$$ \begin{aligned} & S_{t c}^{2}= \frac{1}{N-1} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)-\left(Y_{i}(0)-\bar{Y}(0)\right)\right)^{2} \ &= \frac{1}{N-1} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)\right)^{2}+\frac{1}{N-1} \sum_{i=1}^{N}\left(Y_{i}(0)-\bar{Y}(0)\right)^{2} \ & -\frac{2}{N-1} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)\right) \cdot\left(Y_{i}(0)-\bar{Y}(0)\right) \ &= S_{t}^{2}+S_{c}^{2}-\frac{2}{N-1} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)\right) \cdot\left(Y_{i}(0)-\bar{Y}(0)\right) \end{aligned} $$

Hence, the expression in (A.3) is equal to

$$ \begin{aligned} \mathbb{V}{W}\left(\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}\right)= & \frac{N{\mathrm{c}}}{N \cdot N_{\mathrm{t}}} \cdot S_{t}^{2}+\frac{N_{\mathrm{t}}}{N \cdot N_{\mathrm{c}}} \cdot S_{c}^{2} \ & +\frac{1}{N} \cdot\left(S_{t}^{2}+S_{c}^{2}-S_{t c}^{2}\right)=\frac{S_{t}^{2}}{N_{\mathrm{t}}}+\frac{S_{c}^{2}}{N_{\mathrm{c}}}-\frac{S_{t c}^{2}}{N} \end{aligned} $$

Now we investigate the bias of the Neyman estimator for the sampling variance, $\mathbb{V}{\text {neyman }}$, under the assumption of a constant treatment effect. Assuming a constant treatment effect, $S{t c}^{2}$ is equal to zero, so we need only find unbiased estimators for $S_{c}^{2}$ and $S_{t}^{2}$ to provide an unbiased estimator of the variance of $\bar{Y}{\mathrm{t}}^{\mathrm{obs}}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}$. Consider the estimator

$$ s_{t}^{2}=\frac{1}{N_{\mathrm{t}}-1} \sum_{i: W_{i}=1}\left(Y_{i}^{\mathrm{obs}}-\bar{Y}_{\mathrm{t}}^{\mathrm{obs}}\right)^{2} $$

The goal is to show that the expectation of $s_{t}^{2}$ is equal to

$$ S_{t}^{2}=\frac{1}{N-1} \sum_{i=1}^{N}\left(Y_{i}(1)-\bar{Y}(1)\right)^{2}=\frac{N}{N-1}\left(\overline{Y^{2}}(1)-(\bar{Y}(1))^{2}\right) $$

First,

$$ \begin{align*} s_{t}^{2} & =\frac{1}{N_{\mathrm{t}}-1} \sum_{i=1}^{N} 1\left{W_{i}=1\right} \cdot\left(Y_{i}^{\mathrm{obs}}-\bar{Y}{\mathrm{t}}^{\mathrm{obs}}\right)^{2} \ & =\frac{1}{N{\mathrm{t}}-1} \sum_{i=1}^{N} 1\left{W_{i}=1\right} \cdot\left(Y_{i}(1)-\bar{Y}{\mathrm{t}}^{\mathrm{obs}}\right)^{2} \ & =\frac{1}{N{\mathrm{t}}-1} \sum_{i=1}^{N} 1\left{W_{i}=1\right} \cdot Y_{i}^{2}(1)-\frac{N_{\mathrm{t}}}{N_{\mathrm{t}}-1}\left(\bar{Y}_{\mathrm{t}}^{\mathrm{obs}}\right)^{2} \tag{A.4} \end{align*} $$

Consider the expectation of the two terms in (A.4) in turn. Using again $D_{i}=\mathbf{1}{W{i}=1}-$ $N_{\mathrm{t}} / N$, with $\mathbb{E}\left[D_{i}\right]=0$, we have

$$ \begin{aligned} \mathbb{E}\left[\frac{1}{N_{\mathrm{t}}-1} \sum_{i=1}^{N} \mathbf{1}{W{i}=1} \cdot Y_{i}^{2}(1)\right] & =\frac{1}{N_{\mathrm{t}}-1} \sum_{i=1}^{N} \mathbb{E}\left[\left(D_{i}+\frac{N_{\mathrm{t}}}{N}\right) \cdot Y_{i}^{2}(1)\right] \ & =\frac{N_{\mathrm{t}}}{N_{\mathrm{t}}-1} \cdot \overline{Y^{2}}(1) \end{aligned} $$

Next, the expectation of the second factor in the second term in (A.4):

$$ \begin{aligned} \mathbb{E}{W} & {\left[\left(\bar{Y}{\mathrm{t}}^{\mathrm{obs}}\right)^{2}\right]=\mathbb{E}{W}\left[\frac{1}{N{\mathrm{t}}^{2}} \sum_{i=1}^{N} \sum_{j=1}^{N} W_{i} \cdot W_{j} \cdot Y_{i}^{\mathrm{obs}} \cdot Y_{j}^{\mathrm{obs}}\right] } \ = & \mathbb{E}{W}\left[\frac{1}{N{\mathrm{t}}^{2}} \sum_{i=1}^{N} \sum_{j=1}^{N} W_{i} \cdot W_{j} \cdot Y_{i}(1) \cdot Y_{j}(1)\right] \ = & \frac{1}{N_{\mathrm{t}}^{2}} \sum_{i=1}^{N} \sum_{j=1}^{N} \mathbb{E}{W}\left[\left(D{i}+\frac{N_{t}}{N}\right) \cdot\left(D_{j}+\frac{N_{\mathrm{t}}}{N}\right) \cdot Y_{i}(1) \cdot Y_{j}(1)\right] \ = & \frac{1}{N_{\mathrm{t}}^{2}} \sum_{i=1}^{N} \sum_{j=1}^{N} Y_{i}(1) \cdot Y_{j}(1) \cdot\left(\mathbb{E}\left[D_{i} \cdot D_{j}\right]+\frac{N_{\mathrm{t}}^{2}}{N^{2}}\right) \ = & \frac{1}{N_{\mathrm{t}}^{2}} \sum_{i=1}^{N} Y_{i}^{2}(1) \cdot\left(\mathbb{E}{W}\left[D{i}^{2}\right]+\frac{N_{\mathrm{t}}^{2}}{N^{2}}\right) \ & +\frac{1}{N_{\mathrm{t}}^{2}} \sum_{i=1}^{N} \sum_{j \neq i} Y_{i}(1) \cdot Y_{j}(1) \cdot\left(\mathbb{E}{W}\left[D{i} \cdot D_{j}\right]+\frac{N_{t}^{2}}{N^{2}}\right)\ = & \frac{1}{N_{\mathrm{t}}^{2}} \sum_{i=1}^{N} Y_{i}^{2}(1) \cdot\left(\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{2}}+\frac{N_{\mathrm{t}}^{2}}{N^{2}}\right) \ & +\frac{1}{N_{\mathrm{t}}^{2}} \sum_{i=1}^{N} \sum_{j \neq i} Y_{i}(1) \cdot Y_{j}(1) \cdot\left(-\frac{N_{\mathrm{c}} \cdot N_{\mathrm{t}}}{N^{2} \cdot(N-1)}+\frac{N_{\mathrm{t}}^{2}}{N^{2}}\right) \ = & \frac{1}{N_{\mathrm{t}}} \cdot \overline{Y^{2}}(1)+\frac{N_{\mathrm{t}}-1}{N \cdot(N-1) \cdot N_{\mathrm{t}}} \sum_{i=1}^{N} \sum_{j \neq i} Y_{i}(1) \cdot Y_{j}(1) \ = & \frac{1}{N_{\mathrm{t}}} \cdot \overline{Y^{2}}(1)-\frac{N_{\mathrm{t}}-1}{N \cdot(N-1) \cdot N_{\mathrm{t}}} \sum_{i=1}^{N} Y_{i}^{2}(1)+\frac{N_{\mathrm{t}}-1}{N \cdot(N-1) \cdot N_{\mathrm{t}}} \sum_{i=1}^{N} \sum_{j=1}^{N} Y_{i}(1) \cdot Y_{j}(1) \ = & \frac{1}{N_{\mathrm{t}}} \cdot \overline{Y^{2}}(1)-\frac{N_{\mathrm{t}}-1}{(N-1) \cdot N_{\mathrm{t}}} \cdot \overline{Y^{2}}(1)+\frac{\left(N_{\mathrm{t}}-1\right) \cdot N}{(N-1) \cdot N_{\mathrm{t}}}(\bar{Y}(1))^{2} \ = & \frac{N_{\mathrm{c}}}{N_{\mathrm{t}} \cdot(N-1)} \cdot \overline{Y^{2}}(1)+\frac{\left(N_{\mathrm{t}}-1\right) \cdot N}{(N-1) \cdot N_{\mathrm{t}}}(\bar{Y}(1))^{2} \cdot \end{aligned} $$

Hence, the expectation of the second term in (A.4) equals

$$ -\frac{N_{\mathrm{c}}}{\left(N_{\mathrm{t}}-1\right) \cdot(N-1)} \cdot \overline{Y^{2}}(1)+\frac{N}{(N-1)} \cdot(\bar{Y}(1))^{2} $$

and adding up the expectations of both terms in in (A.4) leads to

$$ \begin{aligned} \mathbb{E}{W}\left[s{t}^{2}\right] & =\frac{N_{\mathrm{t}}}{N_{\mathrm{t}}-1} \cdot \overline{Y^{2}}(1)-\frac{N_{\mathrm{c}}}{\left(N_{\mathrm{t}}-1\right) \cdot(N-1)} \cdot \overline{Y^{2}}(1)-\frac{N}{(N-1)} \cdot(\bar{Y}(1))^{2} \ & =\frac{N}{N-1} \cdot \overline{Y^{2}}(1)-\frac{N}{(N-1)} \cdot(\bar{Y}(1))^{2}=S_{t}^{2} \end{aligned} $$

Following the same argument,

$$ \mathbb{E}{W}\left[s{c}^{2}\right]=\frac{1}{N_{c}-1} \cdot \mathbb{E}{W}\left[\sum{i=1}^{N}\left(1-W_{i}\right) \cdot\left(Y_{i}^{\text {obs }}-\bar{Y}{\mathrm{c}}^{\mathrm{obs}}\right)^{2}\right]=S{c}^{2} $$

Hence, the estimators $s_{c}^{2}$ and $s_{t}^{2}$ are unbiased for $S_{c}^{2}$ and $S_{t}^{2}$, and can be used to create an unbiased estimator for the variance of $\bar{Y}{\mathrm{t}}^{\text {obs }}-\bar{Y}{\mathrm{c}}^{\text {obs }}$, our estimator of the average treatment effect under the constant treatment effect assumption.