9.1 - MVUEs and the Rao-Blackwell theorem

Introduction

  • In section 8, we mostly discussed properties of estimators
    • Bias
    • Variance
    • MSE
    • Consistency
    • Sufficiency
  • In section 9, we will discuss methods of finding estimators
  • We will pick up with we left off with sufficiency, and discuss how we can use sufficient statistics to find optimal unbiased estimators

The MVUE

  • An expert archer:
    • hits the target on average
    • when they miss, they don’t miss by much
  • A high-quality estimator:
    • is unbiased
    • has small variance
  • An unbiased estimator \(\hat\theta\) is said to be the minimum variance unbiased estimator (MVUE) of \(\theta\) if, for any other unbiased estimator \(\tilde\theta\):

\[Var(\hat\theta) \le Var(\tilde \theta)\]

  • The MVUE is “optimal” among the class of unbiased estimators
  • It does not necessarily have the smallest MSE of all possible estimators! (why not?)

The Rao-Blackwell theorem

C.R. Rao; published initial results in Bulletin of the Calcutta Mathematical Society in 1945

Image source: Penn State University

David Blackwell; published the theorem independently of Rao in Annals of Mathematical Statistics in 1947

Image source: Wikipedia

The Rao-Blackwell theorem

  • Let \(\tilde\theta\) be an unbiased estimator for \(\theta\) with finite variance.
  • Let \(U\) be a sufficient statistic for \(\theta\), and define \(\hat\theta = E(\tilde\theta|U).\)
  • Then for all \(\theta:\)

\[E(\hat\theta) = \theta;\]

\[Var(\hat\theta) \leq Var(\tilde\theta).\]

  • I.e.:
    • \(\hat\theta\) is unbiased;
    • \(\hat\theta\) has a variance at least as small as any other unbiased estimator;
    • \(\hat\theta\) is the MVUE.*

* A more rigorous condition on the sufficient statistic \(U\) known as completeness is required for uniqueness of the MVUE, see the Lehmann-Sheffé theorem. The formal condition of completeness is beyond the scope of an undergraduate math-stat course, but as an example of an inlcomplete sufficient statistic, consider \(Y_{(n)}\) as an estimator of \(\theta\) when the population is \(UNIF(\theta,\theta+1)\). For all problems requiring you to find an MVUE, the sufficient statistic will be complete.

Proof of the Rao-Blackwell theorem

  • Let \(\tilde\theta\) be an unbiased estimator for \(\theta\) with finite variance.
  • Let \(U\) be a sufficient statistic for \(\theta\), and define \(\hat\theta = E(\tilde\theta|U).\)
  • Then for all \(\theta:\)

\[E(\hat\theta) = \theta;\]

\[Var(\hat\theta) \leq Var(\tilde\theta).\]

Proof:

\[E(\hat\theta) = E(E(\tilde\theta|U)) = E(\tilde\theta) = \theta\]

\[Var(\hat\theta) = Var(E(\tilde\theta|U))\]

\[\le Var(E(\tilde\theta|U))+E(\underbrace{Var(\tilde\theta|U)}_{\ge 0 \mbox{ always}}) = Var(\tilde\theta)\]

  • Aside from Casella & Berger, pg 343: “[This proof makes] no mention of sufficiency, so it might at first seem that conditioning on anything will result in an improvement. This is, in effect, true, but the problem is that the resulting quantity will probably depend on \(\theta\) and not be an estimator.”

Extension of Rao-Blackwell theorem

Rao-Blackwell theorem can be proved similarly for a more general case:

  • Let \(\tilde\tau(\theta)\) be an unbiased estimator for \(\tau(\theta)\) with finite variance.
  • Let \(U\) be a sufficient statistic for \(\theta\), and define \(\hat\tau(\theta) = E(\tilde\tau(\theta)|U).\)
  • Then for all \(\theta:\)

\[E(\hat\tau(\theta)) = \tau(\theta);\]

\[Var(\hat\tau(\theta)) \leq Var(\tilde\tau(\theta)).\]

  • I.e., any function of a sufficient statistic for \(\theta\) that is unbiased for \(\tau(\theta)\) has variance no larger than any other unbiased estimator of \(\tau(\theta)\).

Application 1: Rao-Blackwellization

Rao-Blackwellization is the process of improving upon an unbiased estimator by conditioning upon a sufficient statistic. For example:

  • Suppose \(Y_1,...,Y_n \stackrel{i.i.d.}{\sim} POI(\lambda)\).
    • Can show via factorization theorem that \(U = \sum_{i=1}^n Y_i\) is sufficient for \(\lambda\).
  • Interest lies in estimating \(\theta= \tau(\lambda) = P(Y=0) = e^{-\lambda}\).
  • Let:

\[\tilde\theta = \left\{\begin{array} {ll} 1 & Y_1 = 0 \\ 0 & Y_1 > 0\\ \end{array}\right.\]

We want to:

  • Prove that \(\tilde \theta\) is indeed unbiased for \(\theta\);
  • Find a better unbiased estimator of \(\theta.\)

Showing unbiasedness of \(\tilde\theta\)

\[\tilde\theta = \left\{\begin{array} {ll} 1 & Y_1 = 0 \\ 0 & Y_1 > 0\\ \end{array}\right.\]

\[E(\tilde\theta) = 1\cdot P(\tilde\theta=1) + 0\cdot P(\tilde\theta = 1)\]

\[ = P(Y_1 =0) = \frac{e^{-\lambda}\lambda^0}{0!} = e^{-\lambda}=\theta\]

Finding an improved estimator

With \(U= \sum_{i=1}^n Y_i \sim POI(n\lambda)\):

\[\hat\theta = E(\tilde\theta|U=u)= 1\cdot P(\tilde\theta =1 |U=u)+ 0\cdot P(\tilde\theta=0|U=u)\]

\[=P(Y_1=0|U=u)=\frac{P(Y_1=0, U=u)}{P(U=u)}\]

\[=\frac{P(Y_1=0,\sum_{i=2}^n Y_i=u)}{P(U=u)}, \mbox{ where } Y_1 \perp\!\!\!\perp \sum_{i=2}^n Y_i\sim POI((n-1)\lambda) \]

\[=\frac{\frac{e^{-\lambda}\lambda^0}{0!}\times\frac{e^{-(n-1)\lambda}[(n-1)\lambda]^u}{u!}}{\frac{e^{-n\lambda}(n\lambda)^u}{u!}} = \left(\frac{n-1}{n}\right)^u\]

  • \(\hat\theta = \left(\frac{n-1}{n}\right)^U\)
    • \(0 \le \hat\theta \le 1\), suitable for a probability estimate
    • Large \(U\) (lots of nonzero \(Y_i\)) \(\Rightarrow\) small \(\hat\theta\)
    • \(U=0\) (all the \(Y_i=0\)) \(\Rightarrow \hat\theta =1\)

Application 2: an MVUE for exponential rate

Suppose \(Y_1,...,Y_n\) are i.i.d. \(\sim EXP(\lambda)\) What is the MVUE of \(\lambda\)?

  • Process: find an unbiased estimator that is a function of the sufficient statistic.
  • Know:
    • \(U = \sum_{i=1}^n Y_i\) is sufficient for \(\lambda\) (factorization theorem)
    • \(U \sim GAM(n,\lambda)\) (MGF method)
    • \(E(U^k) = \frac{\Gamma(n+k)}{\lambda^k\Gamma(n)}\) if \(n+k>0\) (STAT 450) \(\Rightarrow E(U^{-1}) = \frac{\lambda}{n-1}\)
  • Thus, \(\hat\lambda = (n-1)U^{-1} = \frac{n-1}{\sum_{i=1}^n Y_i}\) is an unbiased function of the sufficient statistic \(U\), and by Rao-Blackwell, has a variance no larger than any other unbiased estimator \(\Rightarrow\) is MVUE.