We start by considering a linear function of two random variables and then extend that to a linear function of more than two random variables.
Let \(X\) be a random variable with mean \(\mu_x\) and standard deviation \(\sigma_x\). Let \(Y\) be a random variable with mean \(\mu_y\) and standard deviation \(\sigma_y\). Furthermore, let \(\sigma_{xy}\) and \(\rho\) be the covariance and correlation between \(X\) and \(Y\), respectively.
Then we can define a new random variable \[Z = aX + bY\] where \(a\) and \(b\) are constants.
The mean of \(Z\) is \[\mu_z = a\mu_x + b\mu_y\]
And the variance of \(Z\) is \[\sigma_z^2 = a^2 \sigma_x^2 + b^2 \sigma_y^2 + 2ab\sigma_{xy}\\=a^2 \sigma_x^2 + b^2 \sigma_y^2 + 2ab \rho \sigma_x \sigma_y\] There is a quick proof of these results at the end of this lesson. We can also check these results with a simple example.
Suppose we have two random variables \(X\) and \(Y\) with joint probability distribution
| \(x=1\) | \(x=2\) | |
|---|---|---|
| \(y=0\) | \(0.20\) | \(0.25\) |
| \(y=1\) | \(0.30\) | \(0.25\) |
It can be shown that \(\mu_x = 1.5\), \(\sigma_x^2 = 0.25\), \(\mu_y=0.55\), \(\sigma_y^2=0.2475\) and \(\sigma_{xy} = -0.025\). What is the mean and standard deviation of \(Z = 2X + 3Y\)?
We know by the result above that the mean of \(Z\) is \[\mu_z = 2\mu_x + 3\mu_y=2 \cdot 1.5 + 3 \cdot 0.55 = 4.65\] and the variance of \(Z\) is \[\sigma_z^2 = 2^2 \sigma_x^2 + 3^2 \sigma_y^2 + 2 \cdot 2 \cdot 3 \sigma_{xy}=2^2 \cdot 0.25 + 3^2 \cdot 0.2475 + 2 \cdot 2 \cdot 3 \cdot (-0.025) = 2.9275\]
We can also find the mean and variance of \(Z\) directly. First we need the probability distribution of \(Z\).
When \(X=1\) and \(Y=0\) then \(Z = 2 \cdot 1 + 3 \cdot 0 = 2\) and this occurs with probability \(0.20\). Continuing in this way for every value of \(X\) and \(Y\) leaves us with
| \(z\) | \(2\) | \(4\) | \(5\) | \(7\) |
|---|---|---|---|---|
| \(p(z)\) | \(0.20\) | \(0.25\) | \(0.30\) | \(0.25\) |
Now that we have the probability distribution of \(Z\) we can find the mean \[\mu_z = 2 \cdot 0.20 + 4 \cdot 0.25 + 5 \cdot 0.30 + 7 \cdot 0.25 = 4.65\] which is the same result we got before.
We can find the variance of \(Z\) from the probability distribution \[\sigma_z^2 = (2-4.65)^2 \cdot 0.20 + (4-4.65)^2 \cdot 0.25 + (5 - 4.65)^2 \cdot 0.30 + (7-4.65)^2 \cdot 0.25= 2.9275\] which is the same result we got before.
Suppose we have \(k\) independent random variables \(X_1,X_2,...,X_k\) with means \(\mu_1,\mu_2,...,\mu_k\) and variances \(\sigma_1^2, \sigma_2^2, ...., \sigma_k^2\). Furthermore suppose we have \(k\) constants \(a_1, a_2, ..., a_k\). Then \[Y=a_1 X_1 + a_2 X_2 + ... + a_k X_k\] has mean \[\mu_y = a_1 \mu_1 + a_2 \mu_2 + ... + a_k \mu_k\] and variance \[\sigma_y^2 = a_1^2 \sigma_1^2 + a_2^2 \sigma_2^2 + ... + a_k^2 \sigma_k^2\]
This result can be used to show that the mean and standard deviation of \(\bar X\) is \(\mu\) and \(\frac{\sigma}{\sqrt{n}}\) and that the mean and standard deviation of \(\hat p\) is \(p\) and \(\sqrt{\frac{p(1-p)}{n}}\).
Linear combination of normal random variables are normal. A result that will be important later in the course is
If \(X \sim N(\mu_x,\sigma_x)\) and \(Y \sim N(\mu_y,\sigma_y)\) and if \(X\) and \(Y\) are independent then \(Z = aX + bY \sim N(a\mu_x+b\mu_y,\sqrt{a^2\sigma_x^2 + b^2\sigma_y^2})\)
\[E(Z) = E(aX+bY)= aE(X) + bE(Y)\] since expectation is a linear operator
\[E(Z-\mu_z)^2 = E(aX+bY - (a\mu_x + b\mu_y))^2\\=E(a(X-\mu_x)+b(Y-\mu_y))^2\\=E[a^2(X-\mu_x)^2+2ab(X-\mu_x)(Y-\mu_y)+b^2(Y-\mu_y)^2]\\=a^2E(X-\mu_x)^2 + b^2E(Y-\mu_y)^2 + 2abE(X-\mu_x)(Y-\mu_y)\\=a^2\sigma_x^2 + b^2\sigma_y^2 + 2ab\sigma_{xy} \]