Mixture Priors
We may require more flexibility in our priors
- Mixture of distributions can add flexibility
- Particularly useful to mix conjugate distributions
- Mixture of distributions can add flexibility
Consider conjugate priors \(\pi_1(\theta),...,\pi_k(\theta)\), leading to posteriors \(\pi_1(\theta),...,\pi_k(\theta|x)\)
We consider the mixture distribution as our new prior:
\[ \pi(\theta) = \sum^k_{i=1}w_i\pi_i(\theta)\]
- Then the posterior is in the same conjugate mixture family:
\[ \pi(\theta|x)\propto\pi(\theta)L(x|\theta)\] \[ =\sum^k_{i=1}w_i\pi_i(\theta)L(x|\theta)\propto\sum^k_{i=1}w_i^*\pi_i(\theta|x)\]
Prior Predictive Checking
- Note the prior predictive distribution:
- We can generate samples from \(\pi(y)\) to see if it is what we expect of our prior
\[ \pi(y) = \int\pi(y|\theta)\pi(\theta)d\theta = \int\pi(y,\theta)d\theta\]
Method
- Generate samples from \(\pi(y)\) given choice of prior \(\pi(\theta)\)
- Can be done through Monte Carlo, or algebraically
- Check if \(\pi(y)\) represents our beliefs about possible data you may observe.
- Alter prior if needed, repeat process.
Multivariate Models
- Can extend univariate analysis to multivariate
\[ \pi(\mathbf{\theta}|x)=\frac{L(x|\mathbf{\theta})\pi(\mathbf{\theta})}{\int L(x|\mathbf{\theta})\pi(\mathbf{\theta})d\mathbf{\theta}}\]
- Can obtain the marginal distribution of parameters after:
\[ \pi(\theta_1|x) = \int_{\theta_2}...\int_{\theta_d}\pi(\theta|x)d\theta_2 d\theta_d\]
Example
- Consider \(Y_1,Y_2\) which are independent given \(\alpha,\beta\)
\[ Y_1\sim Poi(\alpha\beta),\quad Y_2\sim Poi((1-\alpha)\beta)\]
\[ \alpha\sim Beta(p,q),\quad \beta\sim Gamma(p+q,1)\]
- Likelihood:
\[ \pi(y_1,y_2|\alpha,\beta) = \underbrace{\frac{\exp(-\alpha\beta)(\alpha\beta)^{y_1}}{y_1!}}_{Poi(\alpha\beta)}*\underbrace{\frac{\exp(-(1-\alpha)\beta)[(1-\alpha)\beta]^{y_2}}{y_2!}}_{Poi((1-\alpha)\beta)}\]
- Joint Prior:
\[ \pi(\alpha.\beta) = \underbrace{\frac{\Gamma(p+q)}{\Gamma(p)\Gamma(q)}\alpha^{p-1}(1-\alpha)^{q-1}}_{ Beta(p,q)}*\underbrace{\frac{\exp(-\beta)\beta^{p+q-1}}{\Gamma(p+q)}}_{Gamma(p+q,1)}\] ### Sampling from Joint Posterior
\[ \pi(\mu,\sigma^2|y) = \pi(\mu|\sigma^2,y)\pi(\sigma^2|y)\]
Monte Carlo Integration
- Almost all Bayesian inference can be expressed as integrals
- This is very easy to do but has high variance
- We can decrease this variance through importance sampling
- This is very easy to do but has high variance
- Monte Carlo integration focuses on mapping an integral to \([0,1]\) such that we can simulate from \(u\sim U(0,1)\) to calculate the expectation:
\[ \int_x g(x)f(x) dx,\quad f(x)=1\]
- For a finite range integral, use:
\[ u = (x-a)/(b-a),\quad x= a+u(b-a),\quad dx=(b-a)du \]
\[ \int^b_a g(x)dx = \int^1_0 (b-a)g(a+u(b-a))du\]
- Can do the same for infinite range integral, use:
\[ u=1/(x+1),\quad x=\frac{1}{u}-1\quad dx=-\frac{1}{u^2} du\]
\[ \int^\infty_0 g(x) = \int^0_1-\frac{1}{u^2}g\left(\frac{1}{u}-1\right)du = \int^1_0\frac{1}{u^2}g\left(\frac{1}{u}-1\right)du\]
Multiple Dimension
- Consider \(U_1,U_2\sim U(0,1)\) with joint density:
\[ f(u_1,u_2)=\begin{cases}1,\quad\text{if }U_1,U_2\in [0,1]\\ 0,\quad\text{otherwise}\end{cases}\]
- We can therefore approximate:
\[ \int^1_0\int^1_0 g(u_1,u_2)du_1du_2 = \mathbb{E}[g(U_1,U_2)]\approx \frac{1}{N}\sum^N_{i=1}g(u^{(i)}_1,u^{(i)}_2)\]
Rejection Sampling
- If sampling from \(f(X)\) is difficult, we can
- Sample from \(x^*\sim g(x)\) which is easy to simulate.
- Accept \(x^*\) with some probability, p.
Method
- Generate \(x^*\) from \(g(x)\)
- Accept \(x^*\) with probability \(\frac{f(x*)}{Kg(x^*)}\)
- Generate \(U\sim U(0,1)\), if \(U\leq\frac{f(x*)}{Kg(x^*)}\), accept.
- To increase efficiency, we want to reduce \(K\) as much as possible (aka we want the highest acceptance probability) subject to \(f(x)\leq Kg(x)\)
- We calculate this bu maximising the following:
\[ K=\max_x \frac{f(x)}{g(x)}\]