Bayesian Updating with Discrete Priors & Discrete Data

Single Update

\[ P(\mathcal{H}|\mathcal{D})=\frac{P(\mathcal{D}|\mathcal{H})P(\mathcal{H})}{P(\mathcal{D})}\qquad\text{where}\:\begin{cases}\mathcal{D}\quad\text{information/data}\\\mathcal{H}\quad\text{hypothesis}\end{cases}\\ \quad\\ \boxed{\text{posterior}\propto\text{likelyhood}\cdot\text{prior}}\\\binom{\text{posterior is normalised by proportionality constant}\:P(\mathcal{D})}{\text{where}\:P(\mathcal{D})\:\text{is the probability of the information}} \]

Example 1. (a) randomly pick a coin and flip heads

Three types of coins; type \(A\) have probability \(0.5\) of heads, type \(B\) have probability \(0.6\) of heads, type \(C\) have probability \(0.9\) of heads. A drawer contains \(5\) coins: \(2\) of type \(A\), \(2\) of type \(B\) and \(1\) of type \(C\). A coin is selected at random and flip to get heads. What is the probability it is type \(A\)? Type \(B\)? Type \(C\)?

Hypothesis \(\mathcal{H}\) Prior \(P(\mathcal{H})\) Likelyhood \(P(D|\mathcal{H})\) Bayes’ Numerator \(P(D|\mathcal{H})P(\mathcal{H})\) Posterior \(P(\mathcal{H}|D)=\frac{P(D|\mathcal{H})P(\mathcal{H})}{P(D)}\) Likelyhood Ratio (Bayes Factor) \(\frac{P(\mathcal{H}|D)}{\mathcal{H}}\)
\(\mathcal{H}=A\) \(P(A)=0.4\) \(P(D|A)=0.5\) \(P(D|A)P(A)=0.2\) \(P(A|D)=\frac{P(D|A)P(A)}{P(D)}=0.322\) \(0.805\) less likely
\(\mathcal{H}=B\) \(P(B)=0.4\) \(P(D|B)=0.6\) \(P(D|B)P(B)=0.24\) \(P(B|D)=\frac{P(D|B)P(B)}{P(D)}=0.387\) \(0.9675\) less likely
\(\mathcal{H}=C\) \(P(C)=0.2\) \(P(D|C)=0.9\) \(P(D|C)P(C)=0.18\) \(P(C|D)=\frac{P(D|C)P(C)}{P(D)}=0.290\) \(1.45\) more likely
————————– ————————– ————————– ————————– ————————–
Hypothesis Prior PMF NO SUM Prior predictive probability \(\quad P(D)=0.62\quad\binom{\text{law of total probability}}{\text{normalisation factor}}\) Posterior PMF

Multi-Update

\[ \begin{align} P(\mathcal{H}|\mathcal{D}_1,\mathcal{D}_1)&=\frac{P(\mathcal{D}_1,\mathcal{D}_2|\mathcal{H})P(\mathcal{H})}{P(\mathcal{D}_1,\mathcal{D}_2)} \qquad\text{where}\:\begin{cases}\mathcal{D}_1\quad\text{information regarding first result}\\\mathcal{D}_2\quad\text{information regarding second result}\\\mathcal{H}\quad\text{hypothesis}\end{cases}\\ &=\frac{P(\mathcal{D}_1|\mathcal{H})P(\mathcal{D}_2|\mathcal{H})P(\mathcal{H})}{P(\mathcal{D}_1)P(\mathcal{D}_2)} \quad\text{for independent events}\:\mathcal{D}_1\:\text{and}\:\mathcal{D_2} \end{align} \]

sampling space for four hypothesis and two dependent events sampling space for four hypothesis and two independent events

Example 1. (b) …flip the coin a second time to get heads again

Hypothesis \(\mathcal{H}\) Prior \(P(\mathcal{H})\) \(1^{st}\) Likelihood \(P(D_1|\mathcal{H})\) \(1^{st}\) Bayes’ Numerator \(P(D_1|\mathcal{H})P(\mathcal{H})\) \(2^{nd}\) Likelihood \(P(D_2|\mathcal{H})\) \(2^{nd}\) Bayes’ Numerator \(P(D_1|\mathcal{H})P(D_2|\mathcal{H})P(\mathcal{H})\) Posterior \(P(\mathcal{H}|D_1,D_2) = \frac{P(D_1|\mathcal{H})P(D_2|\mathcal{H})P(\mathcal{H})}{P(D_1)}\)
\(\mathcal{H}=A\) \(P(A) \\= 0.4\) \(P(D_1|A)\\ = 0.5\) \(P(D_1|A)P(A) \\= 0.2\) \(P(D_2|A) \\= 0.5\) \(P(D_1|A)P(D_2|A)P(A) \\= 0.1\) \(P(A|D) = \frac{P(D_1|A)P(D_2|A)P(A)}{P(D_1)} \\= 0.246\)
\(\mathcal{H}=B\) \(P(B) \\= 0.4\) \(P(D_1|B)\\ = 0.6\) \(P(D_1|B)P(B) \\= 0.24\) \(P(D_2|B) \\= 0.6\) \(P(D_1|B)P(D_2|B)P(B) \\= 0.144\) \(P(B|D) = \frac{P(D_1|B)P(D_2|B)P(B)}{P(D_1)}\\ = 0.355\)
\(\mathcal{H}=C\) \(P(C) = 0.2\) \(P(D_1|C)\\ = 0.9\) \(P(D_1|C)P(C) \\= 0.18\) \(P(D_2|C) \\= 0.9\) \(P(D_1|C)P(D_2|C)P(C) \\= 0.162\) \(P(C|D) = \frac{P(D_1|C)P(D_2|C)P(C)}{P(D_1)}\\ = 0.399\)
Hypothesis Prior PMF NO SUM \(0.62\) NO SUM \(0.406\) Posterior PMF