8.3 - Consistency

Consistency: an asymptotic property

• An estimator \(\hat\theta\) of \(\theta\) is said to be a consistent estimator of \(\theta\) if, for any \(\epsilon>0\):

\[\lim_{n\rightarrow \infty} P(|\hat\theta-\theta| < \epsilon) = 1.\]

• An equivalent statement:

\[\lim_{n\rightarrow \infty} P(|\hat\theta-\theta| > \epsilon) = 0.\]

• Consistency is also known as convergence in probability. If \(\hat\theta\) is a consistent estimator of \(\theta\), \(\hat\theta\) is said to converge in probability to \(\theta\) and we can write:

\[\hat \theta \rightarrow_p \theta\]

• Intuitively: the estimator \(\hat \theta\) is arbitrarily close to \(\theta\) if \(n\) is large.

Example: consistency of sample mean of normals

• Suppose \(Y_1,...,Y_n\) are i.i.d. \(\sim N(\mu,\sigma^2)\). Show that \(\bar Y\) is a consistent estimator of \(\mu\).

Proof:

For any \(\epsilon >0\):

Simulating consistency

To demonstrate consistency via simulation study, we will:

• Show how \(\bar Y\) approaches \(\mu\) for one “run” of \(n \in \{2,4,6, ..., 1000\}\);
• Do this for many runs;
• For each \(n\), find the proportion of runs for which \(\bar Y\) is within \(\epsilon\) of \(\mu.\)
• One “run” of \(\bar Y\):

library(purrrfect)
library(tidyverse)

set.seed(1234)
mu <- 3
(ybars_one_run <- parameters(~n,
            seq(2,1000, by = 2))
 %>% mutate(y_bar = map_dbl(n, \(n) mean(rnorm(n, mu, sd = 1))),
            mu = mu,
           )
) %>% head()

# A tibble: 6 × 3
      n y_bar    mu
  <dbl> <dbl> <dbl>
1     2  2.54     3
2     4  2.92     3
3     6  2.32     3
4     8  3.04     3
5    10  2.61     3
6    12  2.39     3

Plotting one simulated “run”

ggplot(data = ybars_one_run, aes(x = n, y = y_bar))+
  geom_line()+
  geom_hline(yintercept = mu, color = "red") + 
  ylab(expression(bar(Y)))+
  theme_classic()

Simulating 5000 runs

• Now simulating 5000 runs, and determining whether each \(\bar Y\) is within \(\epsilon = 0.1\) of \(\mu=3\):

epsilon <- 0.1

(ybars_many_runs <- parameters(~n,   seq(2,1000, by = 2))
 %>% add_trials(5000)
 %>% mutate(y_bar = map_dbl(n, \(n) mean(rnorm(n, mu, sd = 1))),
            mu = mu,
           )
 %>% mutate(within_epsilon = ifelse(abs(y_bar-mu) < epsilon, 1, 0))
) %>% head

# A tibble: 6 × 5
      n .trial y_bar    mu within_epsilon
  <dbl>  <dbl> <dbl> <dbl>          <dbl>
1     2      1  2.70     3              0
2     2      2  2.95     3              1
3     2      3  1.78     3              0
4     2      4  2.80     3              0
5     2      5  3.51     3              0
6     2      6  3.18     3              0

Plotting the first 16 runs

Finding simulated \(P(|\bar Y - \mu|<\epsilon)\)

• For each \(n\), find the proportion of runs for which \(\bar Y\) is within \(\epsilon\) of \(\mu\)
• Plot vs \(n\):

(ybars_many_runs 
  %>% summarize(prob_within_eps = mean(within_epsilon), .by = n) 
) -> prop_within_epsilon

ggplot(data = prop_within_epsilon) + 
  geom_line(aes(x = n, y = prob_within_eps)) + 
  labs(x='n',
    y=expression(P(abs(bar(Y) - mu) < epsilon))
    ) +
  theme_classic(base_size = 16)

Not all statistics \(\rightarrow_p\) to a parameter

• A consistent estimator \(\hat \theta\) of \(\theta\) will converge in probability to \(\theta\).
  
• However, not all statistics converge in probability to a parameter.
• If \(\hat\theta\rightarrow_p c\) for some \(c\ne \theta\), then \(\hat\theta\) is clearly not a consistent estimator of \(\theta\).

Sample max of Betas converges in probability to a constant

• Let \(Y_{(n)}\) represent the maximum order statistic from a sample of size \(n\) drawn i.i.d. from a \(BETA(\alpha,1)\) parent population
• Parent population CDF:

\[F_Y(y) = \begin{cases} 0 & y < 0 \\ y^{\alpha} & 0\le y \le 1 \\ 1 & y > 1 \end{cases}\]

• Show that \(Y_{(n)}\rightarrow_p 1\), and thus that \(Y_{(n)}\) is not a consistent estimator of \(\alpha\).

Convergence of \(Y_{(n)}\) to 1

For \(\epsilon >0\):

\[P(|Y_{(n)}-1| < \epsilon)=P(-\epsilon< Y_{(n)}-1 < \epsilon)=P(1-\epsilon< Y_{(n)} < 1+\epsilon)\]

\[=F_{Y_{(n)}}(1+\epsilon) -F_{Y_{(n)}}(1-\epsilon) \]

Establishing consistency using asymptotic bias and variance

Example: Sample variance of normals

• Suppose \(Y_1,...,Y_n\) are i.i.d. \(\sim N(\mu,\sigma^2)\)
• Let:

\[\hat\sigma_1^2 = \frac{\sum_{i=1}^n (Y_i-\bar Y)^2}{n-1}\]

\[\hat\sigma_2^2 = \frac{\sum_{i=1}^n (Y_i-\bar Y)^2}{n}\]

• Show that both estimators are consistent for \(\sigma^2\).

Investigating \(\hat\sigma_1^2\)

• Note that:

\[\hat\sigma_1^2 = \frac{\sum_{i=1}^n (Y_i-\bar Y)^2}{n-1} = S^2\]

Investigating \(\hat\sigma_2^2\)

• Note that:

\[\scriptsize \hat\sigma_2^2 = \frac{\sum_{i=1}^n (Y_i-\bar Y)^2}{n} = \frac{n-1}{n}\hat\sigma^2_1\]

The Weak Law of Large Numbers

• A general law that establishes consistency of sample means
• Distribution agnostic; regardless of the parent population’s distribution (assuming mild regularity conditions)
• Informally: the sample mean converges in probability (is consistent for) the population mean, whatever it is.

Formal statement of WLLN

• If \(Y_1,...,Y_n\) are i.i.d. from a population with finite mean and variance, i.e. 
  • \(\mu< \infty\)
  • \(\sigma^2 < \infty\)
• Then \(\bar Y\) is a consistent estimator of \(\mu\); that is, \(\bar Y \rightarrow_p \mu\).

Application of WLLN: sample means of raw Gammas

• Suppose \(Y_1,Y_2,...,Y_n \stackrel{i.i.d}{\sim} GAM(\alpha,\lambda)\).
  
• Consider \(\bar Y\), the sample mean
• To what does \(\bar Y\) converge in probability?

Application of WLLN: sample means of squared Gammas