Problem Statement

Let \(Y_1, Y_2, \ldots, Y_n \stackrel{\text{i.i.d.}}{\sim} N(\theta, 1)\).
The prior distribution for \(\theta\) is Cauchy with density

\[ \pi(\theta) = \frac{1}{\pi} \cdot \frac{1}{1+\theta^2}, \quad -\infty < \theta < \infty. \]

Tasks:

  1. Derive, up to a constant of proportionality, the posterior density of \(\theta\).
  2. Using the prior as the proposal distribution, find the acceptance probability for the rejection method.
  3. Using the prior as the proposal distribution, find the acceptance probability for the Metropolis-Hastings independence sampler.

Step 1: Likelihood Function

Let \(\bar{y} = \frac{1}{n} \sum_{i=1}^n y_i\). Then

\[ \sum_{i=1}^n (y_i - \theta)^2 = \sum_{i=1}^n (y_i - \bar{y})^2 + n(\bar{y} - \theta)^2 = S + n(\bar{y} - \theta)^2. \]

Thus the likelihood is

\[ L(\theta) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}} e^{-\frac{(y_i - \theta)^2}{2}} = \frac{1}{(2\pi)^{n/2}} e^{-S/2} \cdot e^{-\frac{n}{2}(\bar{y} - \theta)^2}. \]

Since \(\frac{e^{-S/2}}{(2\pi)^{n/2}}\) is constant with respect to \(\theta\), we have

\[ L(\theta) \propto e^{-\frac{n}{2}(\bar{y} - \theta)^2}. \]

Step 2: Posterior Density (up to proportionality)

Prior:

\[ \pi(\theta) = \frac{1}{\pi(1+\theta^2)}. \]

By Bayes’ theorem:

\[ \pi(\theta \mid y) \propto L(\theta) \cdot \pi(\theta) \propto e^{-\frac{n}{2}(\bar{y} - \theta)^2} \cdot \frac{1}{\pi(1+\theta^2)}. \]

Ignoring the constant \(1/\pi\):

\[ \pi(\theta \mid y) \propto \frac{e^{-\frac{n}{2}(\bar{y} - \theta)^2}}{1+\theta^2}. \]

This is the unnormalized posterior density (target distribution).

Step 3: Rejection Sampling with Prior as Proposal

Definitions

  • Target distribution (what we want): \[ f(\theta) = \pi(\theta \mid y) \propto \frac{e^{-\frac{n}{2}(\bar{y} - \theta)^2}}{1+\theta^2} \]

  • Proposal distribution (what we sample from): \[ g(\theta) = \pi(\theta) = \frac{1}{\pi(1+\theta^2)} \]

Ratio of target to proposal

Using unnormalized densities (constants cancel in the ratio):

\[ \frac{f(\theta)}{g(\theta)} \propto \frac{ \frac{e^{-\frac{n}{2}(\bar{y} - \theta)^2}}{1+\theta^2} }{ \frac{1}{1+\theta^2} } = e^{-\frac{n}{2}(\bar{y} - \theta)^2}. \]

Thus, up to a constant factor:

\[ \frac{f(\theta)}{g(\theta)} = e^{-\frac{n}{2}(\bar{y} - \theta)^2}. \]

Finding the bounding constant \(M\)

We need \(M\) such that \(\frac{f(\theta)}{g(\theta)} \leq M\) for all \(\theta\).

Since \(e^{-\frac{n}{2}(\bar{y} - \theta)^2} \leq 1\) for all \(\theta\), with maximum \(1\) at \(\theta = \bar{y}\), we can take

\[ M = 1. \]

Rejection sampling algorithm

  1. Generate \(X \sim g(\theta)\) (sample from the prior).
  2. Generate \(U \sim \text{Uniform}(0,1)\).
  3. Accept \(X\) if

\[ U \leq \frac{f(X)}{M g(X)} = \frac{e^{-\frac{n}{2}(\bar{y} - X)^2}}{1} = e^{-\frac{n}{2}(\bar{y} - X)^2}. \]

Conditional acceptance probability

Given a proposed value \(X = x\), the probability of accepting it is

\[ P(\text{accept} \mid X = x) = \exp\left[-\frac{n}{2}(x - \bar{y})^2\right]. \]

This is the author’s “acceptance probability of the rejection method” (conditional on the proposed value).

Step 4: Metropolis-Hastings Independence Sampler

Setup

  • Target: \(f(\theta) \propto \frac{e^{-\frac{n}{2}(\bar{y} - \theta)^2}}{1+\theta^2}\)
  • Proposal: \(q(\theta' \mid \theta) = g(\theta') = \frac{1}{\pi(1+\theta'^2)}\) (independent of current \(\theta\))

MH acceptance ratio

\[ \alpha(\theta, \theta') = \min\left(1, \frac{f(\theta') / q(\theta' \mid \theta)}{f(\theta) / q(\theta \mid \theta')}\right) = \min\left(1, \frac{f(\theta') / g(\theta')}{f(\theta) / g(\theta)}\right). \]

From Step 3, \(\frac{f(\theta)}{g(\theta)} \propto e^{-\frac{n}{2}(\bar{y} - \theta)^2}\). Therefore

\[ \frac{f(\theta') / g(\theta')}{f(\theta) / g(\theta)} = \frac{e^{-\frac{n}{2}(\bar{y} - \theta')^2}}{e^{-\frac{n}{2}(\bar{y} - \theta)^2}} = \exp\left[\frac{n}{2}\left((\bar{y} - \theta)^2 - (\bar{y} - \theta')^2\right)\right]. \]

Thus the acceptance probability is

\[ \alpha(\theta, \theta') = \min\left(1, \exp\left[\frac{n}{2}\left((\bar{y} - \theta)^2 - (\bar{y} - \theta')^2\right)\right]\right). \]

Note that the Cauchy prior cancels completely; the acceptance ratio depends only on the likelihood.

Summary of Results

Quantity Expression
Posterior (up to proportionality) \(\pi(\theta \mid y) \propto \frac{e^{-\frac{n}{2}(\bar{y} - \theta)^2}}{1+\theta^2}\)
Rejection sampling (proposal = prior) \(M = 1\), conditional accept probability \(= \exp\left[-\frac{n}{2}(x - \bar{y})^2\right]\)
MH independence sampler (proposal = prior) \(\alpha(\theta, \theta') = \min\left(1, \exp\left[\frac{n}{2}\left((\bar{y} - \theta)^2 - (\bar{y} - \theta')^2\right)\right]\right)\)

Key Observations

  • The posterior is proportional to a normal kernel (centered at \(\bar{y}\)) times a Cauchy prior.
  • When the prior is used as the proposal in rejection sampling, the \(1/(1+\theta^2)\) terms cancel, leaving an exponential acceptance probability.
  • In the MH independence sampler, the proposal also cancels exactly, so the acceptance ratio depends only on the likelihood ratio.
  • This cancellation is special to the case where the proposal equals the prior.