Conjugate Priors
- Conjugate priors permit the posterior distribution to stay in the same family as the prior.
- Same distribution but altered parameters
- Only easy case for conjugate pairs is the exponential family:
\[ f(x|\theta) = h(x)g(\theta)\exp(t(x)c(x)) \]
\[ \pi(\theta | x)\propto \pi(\theta)g(\theta)^n\exp\{c(\theta)\sum t(x_i)\}\]
- If we choose:
\[ \pi(\theta)\propto g(\theta)^d\exp\{bc(\theta)\} \]
- We will have prior in the form:
\[ \pi(\theta |x)\propto g(\theta)^{d+n}\exp\{c(\theta)[\sum t(x_i)+b]\}\]
Common Examples
Improper Priors
- We can have priors that do not integrate to 1
- If this results in a proper posterior, this is fine. However, if it results in a improper posterior, we cannot use this.
\[\int_\theta\pi(\theta)d\theta = \infty\]
Example
- A common case is the normal distribution
\[ x_i\sim N(\theta,\sigma^2),\quad\theta\sim N(b,d^2)\]
\[ \theta|x\sim N\left(\frac{cb+n\tau\bar{x}}{c+n\tau},\frac{1}{c+n\tau}\right)\]
- As the prior \(d\rightarrow\infty\), it becomes \(N(b,\infty)\). However, the posterior is still proper:
- This improper prior is common as it is uniformative in the form of normal distribution.
\[ \theta|x\sim N\left(\bar{x},\frac{\sigma^2}{n}\right)\]
Jeffery’s Prior
- Jeffery’s priors ensure prior ignorance is consistent across \(1-1\) transformations
\[ \pi_j(\theta)\propto |I(\theta)|^{1/2}\]
- Where the fisher information is calculated as:
\[ I(\theta) = -\mathbb{E}\left[\frac{d^2\log(L(x|\theta))}{d\theta^2}\right]\] * Noting:
\[ det\left(\begin{matrix}a\quad b\\c\quad d\end{matrix}\right) = da-bc\]
Objections
- Depends on the form of the data through \(L(x|\theta)\)
- Prior distribution shouldn’t be influenced by the data to be collected.
- Priors are inconsistent in certain cases
- For example the normal case:
\[ \pi_J(\mu)*\pi_J(\sigma)\neq\pi(\mu,\sigma)\]
Inversion Sampling
- Inversion sampling is a way to simulate from a univariate distribution \(F(x)\)
\[ X\sim F(X),\text{ then }F(X)\sim U(0,1)\]
\[ U\sim U(0,1),\text{ then }F^{-1}(U)=X\]
Method
- To draw samples from \(F(x)\)
- Draw \(u_1,...,u_n\sim U(0,1)\)
- Set \(x_i=F^{-1}(u_i)\)
- Return \(x_1,...,x_n\) as i.i.d samples from \(F(x)\)
Example
- Consider \(f(x)=3x^2\)
\[ F(x) = x^3\]
- Set \(F(x)=u\), solve for \(x\):
\[ x^3 = u,\quad\therefore\quad x=u^{1/3} \]
- Now generate \(u\sim U(0,1)\) and set
\[ x=u^{1/3}\sim f(x)\]