Bayes Estimator for a Binomial Proportion Using a Beta Prior

Introduction

We derive the Bayes estimator for a binomial proportion \(\theta\) under squared error loss, using a Beta\((\alpha,\beta)\) prior distribution.

Model Setup

Likelihood

Let \(Y \mid \theta \sim \text{Binomial}(n,\theta)\). The probability mass function is:

\[ P(Y=y \mid \theta) = \binom{n}{y} \theta^y (1-\theta)^{n-y}, \quad y = 0,1,\dots,n \]

Prior Distribution

Let \(\theta \sim \text{Beta}(\alpha,\beta)\). The probability density function is:

\[ p(\theta) = \frac{\theta^{\alpha-1} (1-\theta)^{\beta-1}}{B(\alpha,\beta)}, \quad 0<\theta<1 \]

where \(B(\alpha,\beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\) is the beta function.

Loss Function

We use squared error loss:

\[ L(\theta, \hat{\theta}) = (\theta - \hat{\theta})^2 \]

Step 1: Bayes Estimator Formula

For squared error loss, the Bayes estimator is the posterior mean:

\[ \hat{\theta}_{\text{Bayes}} = \mathbb{E}[\theta \mid Y = y] \]

Thus, we need to find the posterior distribution \(p(\theta \mid y)\) and compute its mean.

Step 2: Apply Bayes’ Theorem

Bayes’ theorem states:

\[ p(\theta \mid y) = \frac{p(y \mid \theta)p(\theta)}{p(y)} \propto p(y \mid \theta) \times p(\theta) \]

We only need the kernel (proportionality) since the denominator \(p(y)\) is a normalizing constant.

Step 3: Multiply Likelihood and Prior

Likelihood kernel:

\[ p(y \mid \theta) \propto \theta^y (1-\theta)^{n-y} \]

Prior kernel:

\[ p(\theta) \propto \theta^{\alpha-1} (1-\theta)^{\beta-1} \]

Multiplying:

\[ p(\theta \mid y) \propto \theta^y (1-\theta)^{n-y} \times \theta^{\alpha-1} (1-\theta)^{\beta-1} \]

Combine exponents:

\[ p(\theta \mid y) \propto \theta^{(y+\alpha-1)} (1-\theta)^{(n-y+\beta-1)} \]

Step 4: Recognize the Posterior Distribution

The kernel

\[ \theta^{y+\alpha-1} (1-\theta)^{n-y+\beta-1} \]

is exactly the kernel of a Beta distribution with parameters:

• First parameter: \(a = y + \alpha\)
• Second parameter: \(b = n - y + \beta\)

Therefore:

\[ \theta \mid y \sim \text{Beta}(y + \alpha, \, n - y + \beta) \]

The full posterior density is:

\[ p(\theta \mid y) = \frac{\theta^{y+\alpha-1} (1-\theta)^{n-y+\beta-1}}{B(y+\alpha, \, n-y+\beta)} \]

Step 5: Compute the Posterior Mean

For a \(\text{Beta}(a,b)\) distribution, the mean is:

\[ \mathbb{E}[\theta] = \frac{a}{a+b} \]

Applying this to our posterior:

\[ \mathbb{E}[\theta \mid y] = \frac{y + \alpha}{(y+\alpha) + (n-y+\beta)} \]

Simplify the denominator:

\[ \mathbb{E}[\theta \mid y] = \frac{y + \alpha}{n + \alpha + \beta} \]

Step 6: State the Bayes Estimator

Under squared error loss, the Bayes estimator is:

\[ \boxed{\hat{\theta}_{\text{Bayes}} = \frac{y + \alpha}{n + \alpha + \beta}} \]

where:

• \(y\) = observed number of successes
• \(n\) = number of trials
• \(\alpha,\beta\) = prior hyperparameters

Interpretation: Weighted Average

The estimator can be rewritten as a weighted average of:

• Prior mean: \(\frac{\alpha}{\alpha+\beta}\)
• Sample proportion (MLE): \(\frac{y}{n}\)

\begin{align*}

\hat{\theta} &= \frac{y + \alpha}{n + \alpha + \beta} \\

&= \frac{\alpha + \beta}{n + \alpha + \beta} \cdot \frac{\alpha}{\alpha + \beta} + \frac{n}{n + \alpha + \beta} \cdot \frac{y}{n} \\

&= \left(\frac{\alpha + \beta}{n + \alpha + \beta}\right) \cdot \text{Prior Mean} + \left(\frac{n}{n + \alpha + \beta}\right) \cdot \text{Sample Proportion}

\end{align*}