Suppose we have data with likelihood function \(phi(x|\theta)\) depending on a hypothesized parameter \(\theta\). Also suppose the prior distribution for \(\theta\) is one of a family of parametrised distributions. If the posterior distribution for \(\theta\) is in this family (matches the prior distribution family) then we say the the family (of priors) are conjugate priors for the likelihood.

With conjugate priors for a given likelihood Bayesian inference is computationally much simpler, as we can directly write down the posterior distribution in terms of updated hyperparameters

Beta Prior Conjugate to Binomial Likelihood:

Hypothesis (unknown) data PRIOR PDF likelihood POSTERIOR PDF
Continuous: \[\theta\] Discrete: \[x\] \[\text{Beta}(\theta,a,b)\\f(\theta)=c_1\theta^{a-1}(1-\theta)^{b-1}\] \[\text{Bin}(N,\theta)\\p(x|\theta)=c_2\theta^{x}(1-\theta^{N-x})\] \[\text{Beta}(\theta,a+x,b+N-x)\\f(\theta|x)=c_3 \theta^{a+x-1}(1-\theta)^{b+N-x-1}\]

\[\text{where}\quad c_1=\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)},\quad c_2=\binom{N}{x},\quad c_3=\frac{\Gamma(a+b+N)}{\Gamma(a+x)\Gamma(b+N-n)}\]

Beta Prior Conjugate to Bernoulli Likelihood:

hypothesis: \(\theta\)

PRIOR PDF likelihood POSTERIOR PDF
\[\text{Beta}(\theta,a,b)\] \[\text{Ber}(\theta)\] \[\text{Beta}(\theta,a+1,b)\quad\text{or}\quad\text{Beta}(\theta,a,b+1)\]
\[f(\theta)=c\theta^{a-1}(1-\theta)^{b-1}\] \[p(1|\theta)=\theta\] \[f(\theta|1)=c_1\theta^{a}(1-\theta)^{b-1}\]
\[f(\theta)=c\theta^{a-1}(1-\theta)^{b-1}\] \[p(0|\theta)=1-\theta\] \[f(\theta|0)=c_0\theta^{a-1}(1-\theta)^{b}\]

\[\text{where}\quad c=\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)},\quad c_1=\frac{\Gamma(a+b+1)}{\Gamma(a+1)\Gamma(b)},\quad c_0=\frac{\Gamma(a+b+1)}{\Gamma(a)\Gamma(b+1)}\]

Beta Prior Conjugate to Geometric Likelihood:

Watch - there are two variations of the geometric function. I’m doing the upto and including

Hypothesis (unknown) data PRIOR PDF likelihood POSTERIOR PDF
Continuous: \[\theta\] Discrete: \[x\] \[\text{Beta}(\theta,a,b)\\f(\theta)=c_1\theta^{a-1}(1-\theta)^{b-1}\] \[\text{Geo}(\theta)\\p(x|\theta)=\theta^x\:(1-p)\] \[\text{Beta}(\theta,a+x,b+1)\\f(\theta|x)=c_3 \theta^{a+x-1}(1-\theta)^b\]

\[\text{where}\quad c_1=\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)},\quad c_2=\frac{\Gamma(a+b+x+1)}{\Gamma(a+x)\Gamma(b+1)}\]

Normal Prior Conjugate to Normal Likelihood:

PRIOR PDF likelihood POSTERIOR PDF
\[\text{N}(\theta,\mu_{prior},\sigma_{prior}^2)\\f(\theta)=c_1e^{-\frac{(\theta-\mu_{prior})^2}{2\sigma_{prior}^2}}\] \[\phi(x|\theta)\sim N(\theta,\sigma^2)\\\phi(x|\theta)=c_2e^{-\frac{(x-\theta)^2}{2\sigma^2}}\] \[f(\theta|x)\sim\text{N}(\theta,\mu_{post},\sigma_{post}^2)\\f(\theta|x)=c_3e^{-\frac{(\theta-\mu_{post})^2}{2\sigma_{post}^2}}\]
\[\text{where}\quad c_1=\frac{1}{\sigma_{prior}\sqrt{2\pi}},\quad c_2=\frac{1}{\sigma\sqrt{2\pi}},\quad c_3=\frac{1}{\sigma_{post}\sqrt{2\pi}}\]
\[\text{and}\quad a=\frac{1}{\sigma_{prior}^2},\quad b=\frac{1}{\sigma^2},\quad\mu_{post}=\frac{a\mu_{prior}+bx}{a+b},\quad\sigma_{post}^2=\frac{1}{a+b}\] very naughty - still to provide prove