9.2 - Method of moments estimators

Introduction

Karl Pearson in 1910

Image source: Wikipedia

  • One of the oldest methods of estimation is the method of moments
  • Heavily developed by Karl Pearson in the first decades of the 20th century

Short Pearson bio

  • Founded world’s first statistics department at University College London
  • Founded Biometrika, an important statistics journal, in 1902
  • Invented the correlation coefficient
  • Proud eugenicist

[M]y view – and I think it may be called the scientific view of a nation - is that of an organized whole, kept up to a high pitch of internal efficiency by insuring that its numbers are substantially recruited from the better stocks….

National Life from the Standpoint of Science, pp 43-44. 1901

Method of moments

Let \(Y_1,...,Y_n\) be a sample drawn i.i.d. from a population with moments:

\[\mu_1 = E(Y)\] \[\mu_2 = E(Y^2)\] \[\vdots\] \[\mu_k = E(Y^k)\]

Set up as many equations as there are population parameters.

Set these population moments equal to the sample moments:

\[ \mu_1 \stackrel{set}{=} \frac{\sum_{i=1}^n Y_i}{n} = \bar Y_1\] \[ \mu_2 \stackrel{set}{=} \frac{\sum_{i=1}^n Y_i^2}{n} = \bar Y_2\] \[ \vdots \] \[ \mu_k \stackrel{set}{=} \frac{\sum_{i=1}^n Y_i^k}{n} = \bar Y_k\]

Solving this system of equations for any parameters governing the population moments yields the method of moments estimators.

Brief algebraic review of sample variance/moments

  • The following relationships may be helpful.
  • \(\frac{\sum_{i=1}^n Y_i}{n} = \bar Y_1\)
  • \(\frac{\sum_{i=1}^n Y_i^2}{n} = \bar Y_2\)

If \(S^2\) represents the “over-\(n\)” sample variance estimator, then:

\[S^2 = \frac{\sum_{i=1}^n (Y_i -\bar Y_1)^2}{n} = \bar Y_2 - \bar Y_1^2\]

Proof: practice!

Example: MOM estimator for exponential rate

  • Suppose \(Y_1,...,Y_n \stackrel{i.i.d.}{\sim} EXP(\lambda)\).
  • Find the method-of-moments estimator of \(\lambda\).
  • Is it unbiased?

\[ \mu_1 = \frac{1}{\lambda} \stackrel{set}{=} \bar Y\]

\[\Rightarrow \hat \lambda_{MOM} = \frac{1}{\bar Y}\]

We know that \(\bar Y \sim GAM(n,n\lambda)\)

\[\Rightarrow E(\bar Y^{-1}) = \frac{\Gamma(n-1)}{(n\lambda)^{-1}\Gamma(n)} = \frac{n}{(n-1)} \lambda\]

\(\Rightarrow \hat \lambda_{MOM}\) biased for \(\lambda\).

Example: MOM estimators of gamma parameters

  • Suppose \(Y_1,...,Y_n \stackrel{i.i.d.}{\sim} GAM(\alpha, \lambda)\).
  • Find the method-of-moments estimators of \((\alpha, \lambda)\).

\[\mbox{Eq. 1: } \mu_1 = \frac{\alpha}{\lambda} \stackrel{set}{=} \bar Y_1\] \[\mbox{Eq. 2: } \mu_2 = E(Y)^2 + Var(Y) =\frac{\alpha^2}{\lambda^2} +\frac{\alpha}{\lambda^2} \stackrel{set}{=} \bar Y_2\]

Solve by substitution. From Eq. 1, \(\alpha = \lambda\bar Y_1\). Plug into Eq. 2:

\[ \frac{\lambda^2\bar Y_1^2}{\lambda^2} +\frac{\lambda \bar Y_1}{\lambda^2} = \bar Y_1^2+\frac{ \bar Y_1}{\lambda} = \bar Y_2\] \[ \Rightarrow \hat \lambda_{MOM} = \frac{\bar Y_1}{\bar Y_2 - \bar Y_1^2} = \frac{\bar Y_1}{S^2}\]

Plug \(\hat \lambda_{MOM}\) into Eq. 1 and solve for \(\alpha\):

\[\frac{\alpha}{\hat\lambda_{MOM}} = \bar Y_1\Rightarrow \hat\alpha_{MOM} = S^2\]

Example MOM estimators for beta parameters

  • Suppose \(Y_1,...,Y_n \stackrel{i.i.d.}{\sim} BETA(\alpha, \beta)\).
  • Find the method-of-moments estimators of \((\alpha, \beta)\).

\[\mbox{Eq. 1: } \mu_1 = \frac{\alpha}{\alpha + \beta} \stackrel{set}{=} \bar Y_1\] \[\mbox{Eq. 2: } \mu_2 = E(Y)^2 + Var(Y) =\left(\frac{\alpha}{\alpha + \beta}\right)^2 + \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)} \stackrel{set}{=} \bar Y_2\]

\[\mbox{Eq. 2 alt: } \mu_2- E(Y)^2= Var(Y) = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)} \stackrel{set}{=} \bar Y_2 - \bar Y_1^2 = S^2\]

Solving the beta equations

From Eq. 1, \(\frac{\alpha}{\alpha + \beta} = \bar Y_1 \Rightarrow \frac{\beta}{\alpha + \beta} = (1-\bar Y_1)\). Thus:

\[ \alpha = (\alpha +\beta) \bar Y_1\] \[ \beta = (\alpha +\beta) (1-\bar Y_1)\]

Plug into Eq. 2 alt:

\[\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}=\frac{(\alpha+\beta)\bar Y_1(\alpha +\beta)(1-\bar Y_1)}{(\alpha+\beta)^2(\alpha+\beta+1)}=\frac{\bar Y_1(1-\bar Y_1)}{(\alpha+\beta+1)} = S^2\] \[ \Rightarrow \alpha +\beta = \frac{\bar Y_1(1-\bar Y_1)}{S^2} -1 = \frac{\bar Y_1 - \bar Y_2}{S^2}\]

\[\Rightarrow \hat\alpha_{MOM} = \left( \frac{\bar Y_1 - \bar Y_2}{S^2}\right)\bar Y_1\]

\[\Rightarrow \hat\beta_{MOM} = \left( \frac{\bar Y_1 - \bar Y_2}{S^2}\right) (1-\bar Y_1)\]