2025-11-14

Goals

  • Define point estimators, bias, MSE, and consistency.
  • Compare Method of Moments and MLE.
  • Work through Normal and Bernoulli examples.
  • Provide simulations and visuals (including an interactive 3D seafoam likelihood).

Core definitions

Let \(X_1,\dots,X_n\) be i.i.d. with parameter \(\theta\).

  • Point estimator: \(\hat\theta = g(X_1,\dots,X_n)\).
  • Bias: \(\mathrm{Bias}(\hat\theta)=E[\hat\theta]-\theta\).
  • MSE: \(\mathrm{MSE}(\hat\theta)=E[(\hat\theta-\theta)^2]\).
  • Consistency: \(\hat\theta_n \xrightarrow{p}\theta\) as \(n\to\infty\).

MLE for Normal \((\mu,\sigma^2)\)

For \(X_i\sim N(\mu,\sigma^2)\): \[\ell(\mu,\sigma^2)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n (X_i-\mu)^2.\]

Solving gives \(\hat\mu=\bar X\) and \(\widehat{\sigma}^2_{\text{MLE}}=\frac{1}{n}\sum (X_i-\bar X)^2\).

Simulated Normal sample (summary)

n true_mu mle_mu mle_sigma sample_sd
100 2 1.9921 1.5226 1.5303

Histogram & density (seafoam theme)

Sampling distribution of \(\overline{X}\)

MLE calculation (code shown)

mle_mu <- mean(df$x)
mle_sigma2 <- sum((df$x - mle_mu)^2)/n
mle_mu; sqrt(mle_sigma2)
## [1] 1.992087
## [1] 1.52261

Simulation: bias & MSE comparison

n bias_mean mse_mean bias_mle_var bias_unbiased_var
5 0.007946 0.437348 -0.428725 0.026594
10 -0.010729 0.232209 -0.207005 0.019994
30 0.000919 0.073336 -0.083430 -0.008721
100 -0.000737 0.023555 -0.018564 0.003976

3D Likelihood Surface

Practical recommendations

  • For small samples prefer unbiased estimators when available.
  • For parametric models, MLE is recommended for asymptotic optimality.
  • Always simulate to check finite-sample performance.