Consider a random sample of size \(n\) from an infinite population with a continuous density, and suppose we arrange the values \(X_1, X_2, \ldots, X_n\) according to size.
These \(Y\)’s are called order statistics. In particular, \(Y_r\) is called the \(r^{\text{th}}\) order statistic.
We restrict to infinite populations with continuous densities so there is zero probability that any two sample values are equal.
\[y_1 = x_1,\; y_2 = x_2 \quad \text{when } x_1 < x_2\] \[y_1 = x_2,\; y_2 = x_1 \quad \text{when } x_2 < x_1\]
| Condition | \(y_1\) | \(y_2\) | \(y_3\) |
|---|---|---|---|
| \(x_1 < x_2 < x_3\) | \(x_1\) | \(x_2\) | \(x_3\) |
| \(x_1 < x_3 < x_2\) | \(x_1\) | \(x_3\) | \(x_2\) |
| \(x_2 < x_1 < x_3\) | \(x_2\) | \(x_1\) | \(x_3\) |
For random samples of size \(n\) from an infinite population with pdf \(f(x)\), the probability density of the \(r^{\text{th}}\) order statistic \(Y_r\) is:
\[g_r(y_r) = \frac{n!}{(r-1)!\,(n-r)!} \left[\int_{-\infty}^{y_r} f(x)\,dx\right]^{r-1} f(y_r) \left[\int_{y_r}^{\infty} f(x)\,dx\right]^{n-r}\]
Proof:
Divide the real line into three intervals:
By the multinomial distribution, the probability that \(r-1\) sample values fall in interval 1, exactly one falls in interval 2, and \(n-r\) fall in interval 3 is:
\[\frac{n!}{(r-1)!\,1!\,(n-r)!} \left[\int_{-\infty}^{y_r} f(x)\,dx\right]^{r-1} \left[\int_{y_r}^{y_r+h} f(x)\,dx\right] \left[\int_{y_r+h}^{\infty} f(x)\,dx\right]^{n-r}\]
By the law of the mean from calculus: \[\int_{y_r}^{y_r+h} f(x)\,dx \approx f(y_r)\cdot h\]
Dividing by \(h\) and letting \(h \to 0\) yields the formula above. \(\blacksquare\)
\[g_1(y_1) = n\,f(y_1)\left[\int_{y_1}^{\infty} f(x)\,dx\right]^{n-1}\]
\[g_n(y_n) = n\,f(y_n)\left[\int_{-\infty}^{y_n} f(x)\,dx\right]^{n-1}\]
The sample median \(\tilde{X} = Y_{m+1}\), with sampling distribution:
\[h(\tilde{x}) = \frac{(2m+1)!}{m!\,m!} \left[\int_{-\infty}^{\tilde{x}} f(x)\,dx\right]^{m} f(\tilde{x}) \left[\int_{\tilde{x}}^{\infty} f(x)\,dx\right]^{m}\]
For samples of size \(n = 2m\), the median is defined as \(\dfrac{Y_m + Y_{m+1}}{2}\).
Show that for random samples of size \(n\) from an exponential population with parameter \(\theta\):
\[f(x) = \frac{1}{\theta}e^{-x/\theta}, \quad x > 0\]
the sampling distributions of \(Y_1\), \(Y_n\), and the median are as derived below.
\[\int_{y_1}^{\infty} f(x)\,dx = \int_{y_1}^{\infty} \frac{1}{\theta}e^{-x/\theta}\,dx = e^{-y_1/\theta}\]
Substituting into the formula for \(g_1\):
\[g_1(y_1) = n \cdot \frac{1}{\theta}e^{-y_1/\theta} \cdot \left(e^{-y_1/\theta}\right)^{n-1} = \frac{n}{\theta}e^{-ny_1/\theta}\]
\[\boxed{g_1(y_1) = \frac{n}{\theta}\,e^{-ny_1/\theta}, \quad y_1 > 0}\]
This is an exponential distribution with parameter \(\theta/n\).
\[\int_{0}^{y_n} f(x)\,dx = 1 - e^{-y_n/\theta}\]
Substituting into the formula for \(g_n\):
\[g_n(y_n) = n \cdot \frac{1}{\theta}e^{-y_n/\theta} \cdot \left(1 - e^{-y_n/\theta}\right)^{n-1}\]
\[\boxed{g_n(y_n) = \frac{n}{\theta}\,e^{-y_n/\theta}\!\left(1 - e^{-y_n/\theta}\right)^{n-1}, \quad y_n > 0}\]
Computing the required integrals:
\[\int_{-\infty}^{\tilde{x}} f(x)\,dx = 1 - e^{-\tilde{x}/\theta}, \qquad \int_{\tilde{x}}^{\infty} f(x)\,dx = e^{-\tilde{x}/\theta}\]
Substituting:
\[h(\tilde{x}) = \frac{(2m+1)!}{m!\,m!} \cdot \left(1 - e^{-\tilde{x}/\theta}\right)^m \cdot \frac{1}{\theta}e^{-\tilde{x}/\theta} \cdot \left(e^{-\tilde{x}/\theta}\right)^m\]
\[\boxed{h(\tilde{x}) = \frac{(2m+1)!}{m!\,m!}\,\frac{1}{\theta}\,e^{-(m+1)\tilde{x}/\theta}\!\left(1 - e^{-\tilde{x}/\theta}\right)^m, \quad \tilde{x} > 0}\]
For large \(n\), the sampling distribution of the median for random samples of size \(2n+1\) is approximately normal with:
\[\text{Mean} = \tilde{\mu}, \qquad \text{Variance} = \frac{1}{8n\,[f(\tilde{\mu})]^2}\]
where \(\tilde{\mu}\) is the population median satisfying \(\displaystyle\int_{-\infty}^{\tilde{\mu}} f(x)\,dx = \tfrac{1}{2}\).
Comparison with the Mean (Normal Population):
For samples of size \(2n+1\) from a normal population, \(\tilde{\mu} = \mu\) and \(f(\tilde{\mu}) = \dfrac{1}{\sigma\sqrt{2\pi}}\), giving:
| Statistic | Variance |
|---|---|
| Mean \(\bar{X}\) | \(\dfrac{\sigma^2}{2n+1}\) |
| Median \(\tilde{X}\) | \(\dfrac{\pi\sigma^2}{4n} \approx \dfrac{\sigma^2}{2n} \cdot \dfrac{\pi}{2}\) |
Conclusion: For large samples from normal populations, the mean is more reliable than the median — it has smaller variance (less sampling variability).
Find the sampling distributions of \(Y_1\) and \(Y_n\) for random samples of size \(n\) from a continuous uniform population on \([0, 1]\).
Find the sampling distribution of the median for random samples of size \(2m+1\) from the population of Exercise 1. \[h(\tilde{x}) = \frac{(2m+1)!}{m!\,m!}\,\tilde{x}^m(1-\tilde{x})^m, \quad 0 < \tilde{x} < 1\]
Find the mean and variance of \(Y_1\) for random samples of size \(n\) from the population of Exercise 1.
Find the sampling distributions of \(Y_1\) and \(Y_n\) for random samples of size \(n\) from a Beta distribution with \(\alpha = 3\), \(\beta = 2\). \[g_1(y_1) = 12n\,y_1^2(1-y_1)(1 - 4y_1^3 + 3y_1^4)^{n-1}, \quad 0 < y_1 < 1\]
Find the sampling distribution of the median for random samples of size \(2m+1\) from the population of Exercise 4.
Show that the joint density of \(Y_1\) and \(Y_n\) is: \[g(y_1, y_n) = n(n-1)\,f(y_1)\,f(y_n)\left[\int_{y_1}^{y_n} f(x)\,dx\right]^{n-2}, \quad y_1 < y_n\]
(a) Use this to find the joint density of \(Y_1\) and \(Y_n\) for an exponential population.
(b) Use this to find the joint density of \(Y_1\) and \(Y_n\) for the uniform population of Exercise 1.
Using part (b) of Exercise 6, find \(\text{Cov}(Y_1, Y_n)\). \[\text{Cov}(Y_1, Y_n) = \frac{1}{(n+1)^2(n+2)}\]
Use the joint density of \(Y_1\) and \(Y_n\) from Exercise 6 and the transformation technique to find an expression for the joint density of \(Y_1\) and the sample range \(R = Y_n - Y_1\).
Using Exercise 8 and part (a) of Exercise 6, find the sampling distribution of \(R\) for random samples of size \(n\) from an exponential population. \[f(R) = \frac{n(n-1)}{\theta^2}\,e^{-R/\theta}\!\left(1 - e^{-R/\theta}\right)^{n-2} \cdot \frac{1}{n-1}, \quad R > 0\]
Use Exercise 8 to find the sampling distribution of \(R\) for random samples of size \(n\) from the continuous uniform population of Exercise 1.
Use the result of Exercise 10 to find the mean and variance of \(R\): \[E(R) = \frac{n-1}{n+1}, \qquad \text{Var}(R) = \frac{2(n-1)}{(n+1)^2(n+2)}\]
In more advanced settings, the moment-generating function (MGF) is not always used because many distributions do not have MGFs. Instead, we use the characteristic function, which exists for every distribution.
Let \(i\) denote the imaginary unit and \(t\) an arbitrary real number.
If \(X\) is a random variable, its characteristic function is \(\varphi_X : \mathbb{R} \to \mathbb{C}\), defined as:
\[\varphi_X(t) = E\!\left(e^{itX}\right) = \begin{cases} \displaystyle\sum_x e^{itx} P(X = x) & \text{if } X \text{ is discrete} \\[8pt] \displaystyle\int_{-\infty}^{\infty} e^{itx} f(x)\,dx & \text{if } X \text{ is continuous} \end{cases}\]
This expectation exists for all \(t\) (since \(|e^{itx}| = 1\)).
\(X\) and \(Y\) have the same characteristic function if and only if they have the same distribution function.
If \(E(X^n)\) exists, then the \(n^{\text{th}}\) derivative of \(\varphi_X(t)\) exists and:
\[\varphi_X^{(n)}(t) = i^n\,E\!\left(X^n e^{itX}\right)\]
In particular, at \(t = 0\):
\[E(X^k) = \frac{\varphi_X^{(k)}(0)}{i^k}\]
Proof:
\[\frac{d}{dt}\varphi_X(t) = \frac{d}{dt}E\!\left(e^{itX}\right) = E\!\left(\frac{d}{dt}e^{itX}\right) = E\!\left(iX e^{itX}\right) = iE\!\left(Xe^{itX}\right)\]
\[\frac{d^2}{dt^2}\varphi_X(t) = i\frac{d}{dt}E\!\left(Xe^{itX}\right) = i\,E\!\left(iX^2 e^{itX}\right) = i^2 E\!\left(X^2 e^{itX}\right)\]
By induction, \(\varphi_X^{(n)}(t) = i^n E(X^n e^{itX})\). \(\blacksquare\)
If \(E(X^n)\) exists:
\[\varphi_X(t) = \sum_{j=0}^{n} \frac{(it)^j}{j!} E(X^j) + o_n(t)\]
where \(o_n(t)\) satisfies \(\displaystyle\lim_{t \to 0} \dfrac{o_n(t)}{t^n} = 0\).
From Property 2, evaluating derivatives at \(t = 0\):
\[\boxed{E(X) = \frac{\varphi_X'(0)}{i}}\]
\[\boxed{\text{Var}(X) = \frac{\varphi_X''(0)}{i^2} - \left[\frac{\varphi_X'(0)}{i}\right]^2 = -\varphi_X''(0) - \left[E(X)\right]^2}\]
Connection to MGF: For distributions that have an MGF \(M_X(t)\), we have \(\varphi_X(t) = M_X(it)\).
Let \(X \sim \text{Binomial}(n, p)\) with \(q = 1 - p\).
\[\varphi_X(t) = E\!\left(e^{itX}\right) = \sum_{x=0}^{n} e^{itx}\binom{n}{x}p^x q^{n-x}\]
\[= \sum_{x=0}^{n} \binom{n}{x}(pe^{it})^x q^{n-x} = \left(q + pe^{it}\right)^n\]
by the Binomial Theorem with \(a = q\) and \(b = pe^{it}\).
\[\boxed{\varphi_X(t) = \left(q + pe^{it}\right)^n}\]
First derivative:
\[\varphi_X'(t) = n\left(q + pe^{it}\right)^{n-1} \cdot pi e^{it}\]
At \(t = 0\), \(q + p = 1\):
\[\varphi_X'(0) = npi \implies E(X) = \frac{npi}{i} = np\]
Second derivative (using the product rule):
\[\varphi_X''(t) = n(n-1)\left(q + pe^{it}\right)^{n-2}(pie^{it})^2 + n\left(q + pe^{it}\right)^{n-1} \cdot pi^2 e^{it}\]
At \(t = 0\):
\[\varphi_X''(0) = n(n-1)p^2 i^2 + npi^2 = -n(n-1)p^2 - np\]
Variance:
\[\text{Var}(X) = \frac{\varphi_X''(0)}{i^2} - [E(X)]^2 = \frac{-n(n-1)p^2 - np}{-1} - n^2p^2\]
\[= n(n-1)p^2 + np - n^2p^2 = np - np^2 = np(1-p) = npq\]
\[\boxed{E(X) = np, \qquad \text{Var}(X) = npq}\]
Let \(X \sim \text{Exponential}(\theta)\) with \(f(x) = \dfrac{1}{\theta}e^{-x/\theta}\), \(x > 0\).
\[\varphi_X(t) = \int_0^{\infty} e^{itx} \cdot \frac{1}{\theta} e^{-x/\theta}\,dx = \frac{1}{\theta}\int_0^{\infty} e^{-x(1/\theta - it)}\,dx\]
\[= \frac{1}{\theta} \cdot \frac{1}{1/\theta - it} = \frac{1}{1 - i\theta t}\]
\[\boxed{\varphi_X(t) = \frac{1}{1 - i\theta t}}\]
First derivative:
\[\varphi_X'(t) = \frac{i\theta}{(1-i\theta t)^2}\]
At \(t = 0\): \(\varphi_X'(0) = i\theta\), so:
\[E(X) = \frac{i\theta}{i} = \theta\]
Second derivative:
\[\varphi_X''(t) = \frac{2(i\theta)^2}{(1-i\theta t)^3} = \frac{-2\theta^2}{(1-i\theta t)^3}\]
At \(t = 0\): \(\varphi_X''(0) = -2\theta^2\), so:
\[\text{Var}(X) = \frac{-2\theta^2}{i^2} - \theta^2 = 2\theta^2 - \theta^2 = \theta^2\]
\[\boxed{E(X) = \theta, \qquad \text{Var}(X) = \theta^2}\]
For each distribution below, (a) find the characteristic function, and (b) derive the mean and variance.
1. Discrete Uniform — \(P(X = x) = \dfrac{1}{N}\) for \(x = 0, 1, \ldots, N-1\).
\[\varphi_X(t) = \frac{1}{N} \cdot \frac{1 - e^{iNt}}{1 - e^{it}}, \qquad E(X) = \frac{N-1}{2}, \qquad \text{Var}(X) = \frac{N^2-1}{12}\]
2. Poisson — \(P(X = x) = \dfrac{e^{-\lambda}\lambda^x}{x!}\) for \(x = 0, 1, 2, \ldots\)
\[\varphi_X(t) = e^{\lambda(e^{it}-1)}, \qquad E(X) = \lambda, \qquad \text{Var}(X) = \lambda\]
3. Continuous Uniform — \(f(x) = \dfrac{1}{b-a}\) for \(a < x < b\).
\[\varphi_X(t) = \frac{e^{ibt} - e^{iat}}{it(b-a)}, \qquad E(X) = \frac{a+b}{2}, \qquad \text{Var}(X) = \frac{(b-a)^2}{12}\]
4. Chi-Squared — \(X \sim \chi^2_n\).
\[\varphi_X(t) = (1 - 2it)^{-n/2}, \qquad E(X) = n, \qquad \text{Var}(X) = 2n\]
5. Normal — \(X \sim N(\mu, \sigma^2)\).
\[\varphi_X(t) = e^{i\mu t - \sigma^2 t^2/2}, \qquad E(X) = \mu, \qquad \text{Var}(X) = \sigma^2\]
End of STA227 Weeks 10–11 Notes