Elements of Bayesian analysis (equations)

Bayes rule, applied to model parameters estimation: \[ prob(\theta|X) = \frac{prob(X|\theta)\cdot prob(\theta)}{prob(X)} \] In other words: posterior probability equals prior probability times likelihood divided by probability of evidence. The Bayesian philosophy is that we update our prior beliefs given new evidence. We do it by considering how was the new evidence likely according to our prior beliefs. \[ \text{posterior} = \frac{\text{likelihood}\cdot \text{prior}}{\text{evidence}} \] Partial evaluations:

• Maximum Likelihood Estimation (MLE). Maximize the likelihood component, i.e. \(prob(X|\theta)\). Find the \(\theta\) which makes the X most probable.
  • gives a point estimate
    
  • does not consider prior probabilities
    
• Maximum a Posteriori (MAP). Maximize the likelihood and priors, i.e. \(prob(X|\theta)\cdot prob(\theta)\)
  • still, only a point estimate
• Full Bayesian evaluation. Has to consider all components of the Bayes formula, including the denominator.
  • produces the full probability distribution

The denominator of the Bayes formula requires integration: \[ prob(X) = \int_{\Theta}prob(X|\theta)\cdot prob(\theta)\mathrm{d}\theta \] The denominator can also be approximated by summation: \[ prob(X) \approx \frac{1}{n}\sum_{i=1}^{n}prob(X|\theta_{i}) \] A random number generator from a statistical library can be used for distributions such as the normal or uniform. For more complex parameter spaces we must use an importance sampler that explores the space one step a time.

---

Note: Here is some intuition for the integral/summation appearing in the denominator. Suppose we only have one binary parameter \(\theta_1\) that can be either true or false. The denominator then can be expressed as \[prob(X) = prob(X|\theta_1)\cdot prob(\theta_1) + prob(X|\neg\theta_1)\cdot prob(\neg\theta_1)\] where \(prob(\theta)+prob(\neg\theta)=1\). The probabilities of the two states of the variable act as weights, because their sum is always 1. Extending this to more than two states we get the summation formula. Extending to continuous space we get the integration.