Direct proof that \(\bar{X}\) and \(S^2\) are independent when sampling from the \(N(\mu, \sigma^2)\) distribution. Let \(X_1, X_2\) be independent \(N(\mu, \sigma^2)\) random variables.

  1. Show that \(Y_1 = X_1 + X_2\) and \(Y_2 = X_2 - X_1\) are independent.
  2. What is the distribution of \(Y_1?\) What is the distribution of \(Y_2?\)
  3. Show that \(W_1 = \frac{1}{2}Y_1\) and \(W_2 = \frac{1}{2}Y_2\) are independent.
  4. What is the distribution of \(W_1?\) What is the distribution of \(W_2\)
  5. Show that \(W_3 = X_1 - \bar{X}\) is also independent of \(W_1 = \bar{X}\)
  6. Argue that \(\bar{X}\) and \(S^2\) are independent for a given sample \(n=2\)
  7. What is the distribution of \(\bar{X}?\) What is the distribution of \(S^2?\)
  8. Develop the same results for \(n=3\)
  9. Develop the same result for sample size \(n\)

Solution

  1. To show this we will consider Theorem 7.5.7 (page 313) in the textbook. The thorem states that Within a MVN random vector, uncorrelated implies independent. So there are few things we must first show in order to use this theorem. Namely we must show that the joint distribution of \(Y_1\) and \(Y_2\) is a MVN. To show this we will use the uniqueness of MGF’s. First we show that \(Y_1\) is normal. \[M_{Y_1}(t) = M_{X_1}(t)M_{X_2}(t) = \exp\{\mu t + \frac{1}{2}\sigma^2t^2\}\exp\{\mu t + \frac{1}{2}\sigma^2t^2\}\] \[ = \exp\{2\mu t + \sigma^2t^2\}\] and so \(Y_1\sim N(2\mu, 2\sigma^2)\). Now we show that \(Y_2\) is normal using the same technique \[M_{Y_2}(t) = M_{X_2}(t)M_{X_1}(-t) = \exp\{0\mu + \sigma^2 t^2\} \sim N(0,2\sigma)\] And so now it suffices to look at the covariance of these two \[Cov(X_1 + X_2, X_2 - X_1) = Var(X_1) - Cov(X_1,X_2) + Cov(X_2,X_1) - Var(X_2)\] \[= Var(X_1) - Var(X_2) = 0\] and therefore independent.

  2. The distribution of \(Y_1\) is \(N(2\mu, 2\sigma^2)\) by uniqueness of the MGF, similarly the distribution of \(Y_2\) is \(N(0,2\sigma^2)\).

  3. We apply a similar technique that was used in part 1. Consider the vector \[(W_1,W_2) = (\bar{X}, X_2-\bar{X})\] Clearly this is MVN, so it suffices to look at the covariance term of these. \[Cov(\bar{X}, X_2 - \bar{X}) = Cov(\bar{X},X_2) - Cov(\bar{X},\bar{X})\] \[=Cov(\frac{1}{2}X_1 + \frac{1}{2}X_2,X_2) -Cov(\bar{X},\bar{X})\] \[=Cov(\frac{1}{2}X_2,X_2) - Cov(\bar{X},\bar{X})\] \[ = \frac{1}{2}Var(X_2) - Var(\bar{X}) = \frac{\sigma^2}{2} - \frac{1}{4}Var(X_1 + X_2)\] \[\frac{\sigma^2}{2} - \frac{1}{4}2\sigma^2 = \frac{\sigma^2}{2} - \frac{\sigma^2}{2} = 0\] and so indeed we have that \(W_1\) and \(W_2\) are independent.

  4. To find the distribution of \(W_1\) and \(W_2\) we will look at the MGF of these. First we look at \(W_1\) \[M_{W_1}(t) = E(e^{t\frac{1}{2}Y_1}) = M_{Y_1}(\frac{1}{2}t) = \exp[\mu t + \frac{1}{4}\sigma^2 t^2] \Rightarrow W_1 \sim N(\mu, 1/2\sigma^2)\] Similarly we have that \[M_{W_2} = \exp[\frac{1}{4}\sigma^2t^2] \Rightarrow W_2 \sim N(0,1/2\sigma^2)\]

  5. We basically repeat the process we did in part (3) \[(W_1,W_3) = (\bar{X}, X_1-\bar{X})\] Clearly this is MVN, so it suffices to look at the covariance term of these. \[Cov(\bar{X}, X_1 - \bar{X}) = Cov(\bar{X},X_1) - Cov(\bar{X},\bar{X})\] \[=Cov(\frac{1}{2}X_1 + \frac{1}{2}X_2,X_1) -Cov(\bar{X},\bar{X})\] \[=Cov(\frac{1}{2}X_1,X_1) - Cov(\bar{X},\bar{X})\] \[ = \frac{1}{2}Var(X_1) - Var(\bar{X}) = \frac{\sigma^2}{2} - \frac{1}{4}Var(X_1 + X_2)\] \[\frac{\sigma^2}{2} - \frac{1}{4}2\sigma^2 = \frac{\sigma^2}{2} - \frac{\sigma^2}{2} = 0\] and so indeed we have that \(W_1\) and \(W_3\) are independent.

  6. Using what we derived above, given \(n=2\) we have that \(S^2\) is just a function of the vector \((X_1 - \bar{X}, X_2 - \bar{X})\), so it follows that it too is independent of the \(\bar{X}\)

  7. The distribution of \(\bar{X}\) using the MGF is: \[M_{\bar{X}}(t) = E[e^{t\bar{X}}] = E[e^{\frac{1}{2}t X_1}e^{\frac{1}{2}t X_2}] \Rightarrow \bar{X} \sim N(\mu, \sigma^2/2)\] and \[M_{S^2}(t) \Rightarrow S^2 \sim N(0,\sigma^2)\]

  8. Consider a sample of \(n=3\) from \(N(\mu, \sigma^2)\), then as we formulated \(W_1, W_2\) and \(W_3\) we can create a fourth element say \(W_4 = X_3 - \bar{X}\). With exactly the same arguments this is independent of \(\bar{X}\). So once again it follows that \(S^2\) is independent of \(\bar{X}\). Moreover the distribution of \(\bar{X}\sim N(\mu, \sigma^2/3)\) and \(S^2 \sim N(0, \sigma^2/3)\) as one would expect.

  9. To develop this derivation for a sample of size \(n\), we would extend to create a MVN vector \((X_1 - \bar{X}, X_2 - \bar{X}, ..., X_n - \bar{X})\). Since MVN we have the liberty to prove independence by simply showing 0 correlation. So using the method we developed in (3) we can show that \[Cov(\bar{X}, X_j- \bar{X}) = 0 \;\;\text{for}\;\; j = 1,2, ...., n\] namely \[Cov(\bar{X}, X_j - \bar{X}) = Cov(\bar{X},X_j) - Cov(\bar{X},\bar{X})\] \[=Cov(\frac{1}{n}X_1 + ... + \frac{1}{n}X_n,X_j) -Cov(\bar{X},\bar{X})\] \[=Cov(\frac{1}{n}X_j,X_j) - Cov(\bar{X},\bar{X})\] \[ = \frac{1}{n}Var(X_j) - Var(\bar{X}) = \frac{\sigma^2}{n} - \frac{1}{n^2}Var(\sum X_i)\] \[\frac{\sigma^2}{n} - \frac{1}{n^2}n\sigma^2 = \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0\] And so \(\bar{X}\) is independent to all elements in the vector \((X_1 - \bar{X}, X_2 - \bar{X}, ..., X_n - \bar{X})\), and so since \(S^2\) is just a function of the elements in the vector then it too is independent of \(\bar{X}\).