5.6 - Variance of linear combinations and conditional expectation

Linear combinations

• A linear combination of random variables \(Y_1\), \(Y_2\),…,\(Y_n\) is simply a weighted sum:

\[U = w_1Y_1 + w_2Y_2 + ... + w_n Y_n = \sum_{i=1}^n w_i Y_i\]

• Especially important for when we move into sampling distribution thinking (see STAT 460)
• If \(n=2\), then we can simplify notation:

\[U = w_1X + w_2 Y\]

Mean and variance of linear combinations

• If \(Y_1\), \(Y_2\),…,\(Y_n\) are random variables with \(E(Y_i) = \mu_i\) and \(Var(Y_i) = \sigma^2_i\).
• If \(U = \sum_{i=1}^n w_i Y_i\), then:

\[E(U) = \sum_{i=1}^n w_i \mu_i\]

\[Var(U) = \sum_{i=1}^n w_i^2 \sigma^2_i + 2 \sum \sum_{1\le i < j \le n} w_iw_j Cov(Y_i, Y_j)\]

Proof of \(E(U)\)

\[E(U) = E\left(\sum_{i=1}^n w_i Y_i\right) =\sum_{i=1}^n E\left(w_i Y_i\right)\]

Proof of \(Var(U)\)

\[Var(U) = E\left[(U - E(U))^2\right]\]

Simple example

• Suppose \(X\) and \(Y\) are random variables with:
  • \(\sigma^2_X = 4\)
  • \(\sigma^2_Y = 6\)
  • \(Cov(X,Y) = 1\)
• Find \(Var(3X-4Y)\).

Example: independent exponentials

Suppose \(X\) and \(Y\) are independent \(EXP(\lambda)\) random variables. Define \(U = \frac{X+Y}{2}\) as the mean of the two random variables. Find \(E(U)\) and \(Var(U)\).

Actuary example

If:

• \(\rho(X,Y) = \frac{1}{3}\)
• \(\sigma^2_X = a\)
• \(\sigma^2_Y = 4a\)
• \(Z = 3X-4Y\), and \(\sigma^2_Z = 114\)

Find \(a\).

Conditional expectation

• Fundamental concept in regression: model mean of \(Y\) conditional on \(X\).
  
• Formal definition:

\[E(g(Y)|X = x) = \int_{-\infty}^\infty g(y) f_{Y|X}(y|x) dy\]

if \(X\) and \(Y\) are jointly continuous, and

\[E(g(Y)|X = x) = \sum_{all\ y} g(y) p_{Y|X}(y|x)\]

if \(X\) and \(Y\) are jointly discrete.

Conditional variance

• Conditional variance, in a regression setting, measures variability of \(Y\) about the regression line, i.e. variability of residuals
• Formally, just set \(g(Y) = Y^2\):

\[E(Y^2|X = x) = \int_{-\infty}^\infty y^2f_{Y|X}(y|x) dy\]

if \(X\) and \(Y\) are jointly continuous, and

\[E(Y^2|X = x) = \sum_{all\ y} y^2 p_{Y|X}(y|x)\]

if \(X\) and \(Y\) are jointly discrete.

• Then \(Var(Y|X=x) = E(Y^2|X=x) - E(Y|X=x)^2\).

Simple conditional expectation example

Simple conditional variance example

\(X\) and \(Y\) are jointly continuous with joint pdf given by

\[f(x,y) = \begin{cases} 1/2 & 0 \le y \le x \le 2 \\ 0 & otherwise \\ \end{cases}\]

Find \(Var(Y|X=1.5)\).

Marginal mean through iteration

• Iteration provides a very useful way of finding marginal means by averaging conditional means.
• Specifically:

\[E(g(Y)) = E_X\left[E_{Y|X}(g(Y)|X)\right]\]

• The subscripts indicate which density the expectation is with respect to.

Proof: Marginal mean through iteration

Example: Normal given uniform

• Suppose \(X\sim UNIF(0,10)\), and \(Y|X=x\sim N(x,1)\):

• Find \(E(Y)\) and \(Cov(X,Y)\).

\(E(Y)\): Two approaches

• Brute force:

\[E(Y) = \int_{-\infty}^\infty \int_{-\infty}^\infty y\cdot f_{Y|X}(y|x)\cdot f_X(x) \,dy\,dx\]

\[=\int_0^{10}\int_{-\infty}^\infty y\cdot \frac{1}{\sqrt{2\pi}}e^{-(y-x)^2/2}\cdot \frac{1}{10} \,dy\,dx\]

\(Cov(X,Y)\): Two approaches

• Brute force:

\[E(XY) = \int_{-\infty}^\infty \int_{-\infty}^\infty xy\cdot f_{Y|X}(y|x)\cdot f_X(x) \,dy\,dx\]

\[=\int_0^{10}\int_{-\infty}^\infty xy\cdot \frac{1}{\sqrt{2\pi}}e^{-(y-x)^2/2}\cdot \frac{1}{10} \,dy\,dx\]

ANOVA-like example

Solving ANOVA-like example

• Iteration:

Marginal variance through iteration

• We can also find marginal variance using conditional means and variances in a similar way:

\[Var(Y) = E_X(Var_{Y|X}(Y|X)) + Var_X(E_{Y|X}(Y|X))\] \[ \mbox{Total variance} = \mbox{Average conditional variance + Variability of conditional means}\]

Revisiting ANOVA-like example

Revisiting regression-like example

Regression with constant variance violated