In traditional (Frequentist) statistics, we treat parameters (like a mean \(\mu\)) as fixed, unknown constants. In the Bayesian Framework, we treat everything unknown as a random variable.

The Paradigm Shift: In traditional stats, you ask: “If the truth were X, how likely is my data?” In Bayesian stats, you flip it and ask: “Given the data I just saw, what is the most likely truth?”

• The parameter \(\theta\) in the frequentist (traditional) is fixed while the data is random– meaning running the experiment again, we’ll give us different data.

• On the other hand, in the Bayesian paradigm, the parameter (\(\theta\)) is random, and the data we collected in the only objective truth we have.

In the old paradigm, probability is defined by long-run frequency. If you flip a coin, “50% heads” means if you flipped it a million times, half would be heads.

In the Bayesian Paradigm, probability is a measure of certainty. It allows us to assign probabilities to one-time events.

• Example: “There is a 70% chance of rain tomorrow.”

  • Frequentist: Struggles with this because “tomorrow” only happens once. You can’t flip “tomorrow” 1,000 times.

  • Bayesian: This makes perfect sense. It represents our current uncertainty based on satellite data (Likelihood) and historical patterns (Prior).

In the traditional paradigm, each study is an island. You start from zero, calculate a p-value, and reach a conclusion.

In the Bayesian Paradigm, science is a cumulative process.

• You start with what you know (Prior).

• You look at new evidence (Likelihood).

• You update your knowledge (Posterior).

• The kicker: Tomorrow, that Posterior becomes your new Prior.

This framework is more intuitive for human decision-making.

• It handles small data better: If you only have 3 data points, a Frequentist model might break or give a “non-significant” result. A Bayesian model says, “Well, based on these 3 points and what we knew before, here is our best (though still uncertain) guess.”

• It gives direct answers: Instead of confusing “confidence intervals,” it gives you a Credible Interval. You can actually say, “There is a 95% probability the drug works,” which is what most people actually want to know.

Imagine you are a researcher testing a new drug. You give it to 100 people and observe how much their blood pressure drops.

The Frequentist starts by assuming the drug does nothing (this is the “Null Hypothesis”). The Question: “If this drug actually does nothing, how likely am I to see a result as extreme as the one I just got?” The Tool: The p-value. The Result: “The p-value is 0.03. Since this is less than 0.05, I reject the idea that the drug does nothing. I conclude the drug works.”

• The Flaw: This doesn’t actually tell you the probability that the drug works; it only tells you how “weird” your data would be if the drug were useless.

The Bayesian starts with a Prior (perhaps based on previous chemical studies or similar drugs).The Question: “Given the data I just collected and what I already knew, what is the probability that this drug reduces blood pressure by at least 5 points?” The Tool: The Posterior Distribution. The Result: “There is a 94% probability that the drug reduces blood pressure by 5–10 points.”

• The Benefit: This is a direct answer to the doctor’s question. It provides a range of likely effects and the certainty of those effects.

Bayes’ Rule is the mathematical bridge that allows us to update our initial beliefs (the Prior) with new evidence (the Likelihood) to arrive at an updated belief (the Posterior).

✅ The Mathematical Formula.

For a parameter \(\theta\) and observed data \(y\):

\[P(\theta | y) = \frac{P(y | \theta) P(\theta)}{P(y)}\]

• \(P(\theta | y)\) (Posterior): Our belief about \(\theta\) after seeing the data.

• \(P(y | \theta)\) (Likelihood): How likely the data is, given a specific \(\theta\).

• \(P(\theta)\) (Prior): Our belief about \(\theta\) before seeing the data.

• \(P(y)\) (Evidence): The total probability of the data, acting as a normalizing constant.

Simulation Study to show the relationship of the prior, likelihood (data) and posterior distribution.

• The blue curve is the prior distribution. It represents what we believed about the parameter \(\theta\) before seeing any data. Notice that it is wide and flat, reflecting high uncertainty.

• The green curve is the likelihood. It summarizes what the data are telling us. Because the data are informative, the likelihood is sharply peaked around the observed value.

• The red curve is the posterior distribution. This is the result of applying Bayes’ Rule.

• Notice that the posterior lies between the prior and the likelihood. It is a compromise between what we believed before and what the data suggest.

• This red curve is the main output of Bayesian inference—it represents our updated belief about \(\theta\) after observing the data.

One of the biggest debates in statistics is how we choose our Prior.

✅ Subjective Probability (The Expert)

This view argues that probability represents a person’s personal degree of belief.

• Strength: It allows us to incorporate expert knowledge, previous studies, or scientific intuition into our model.

• Example: A doctor’s prior belief that a patient has a rare disease based on 20 years of clinical experience.

✅ Objective Probability (The Skeptic)

This view seeks to minimize the influence of the researcher’s personal opinions.

• Uninformative Priors: We use “flat” priors that give equal weight to all possible values so the data “speaks for itself.”

• Jeffreys’ Prior: A mathematical way to create a prior that remains the same even if we change the scale of our measurements.

Scenario: Predicting the probability of finding oil in different locations (A, B, C, D).

Subjective Prior (Expert Knowledge)

• The geologist thinks Location B is most promising based on experience.

• Prior probabilities reflect this belief:

  • A: 10%, B: 60%, C: 20%, D: 10%

Objective Prior (Flat / Non-informative)

• No prior knowledge; all locations are equally likely.

• Prior probabilities:

  • A: 25%, B: 25%, C: 25%, D: 25%

Note: A flat prior is the simplest approach—it assumes all possibilities are equally likely. But flat priors aren’t always appropriate, especially when the parameter isn’t naturally bounded or when transformations matter. That’s where Jeffreys prior comes in. It will be discussed in details in the succeeding lectures.

To illustrate how the framework works, let’s walk through a classic example:

1. Prior: A disease affects 0.1% of the population (\(P(\theta = \text{sick}) = 0.001\)).
2. Likelihood: A test is 99% accurate (\(P(y = + | \theta = \text{sick}) = 0.99\)).It also has a 5% false-positive rate (\(P(+ | \text{Healthy}) = 0.05\)).
3. Data: A patient tests positive (\(y = +\)).
4. Task: Calculate the Posterior. What is the probability you are actually sick given that your test result is positive?

In Bayes theorem,

\[P(\text{Sick} | +) = \frac{P(+|\text{Sick}) \times P(\text{Sick})}{P(+)}\] Now, compute the marginal probability of positive test: \[P(+)= P(+|\text{Sick}) \times P(\text{Sick})+P(+|\text{Healthy}) \times P(\text{Healthy})\] where:

• \(P(\text{Sick)}=0.001 \implies P(\text{Healthy}=1- P(\text{Sick)}=0.999\)

• \(P(\text{Sick})=0.99 \implies (\text{Healthy})=0.05\)

Hence, \[P(+)= (0.99)(0.001)+(0.05)(0.999)=0.05094\]

Therefore,

\[P(\text{Sick} | +) = \frac{P(+|\text{Sick}) \times P(\text{Sick})}{P(+)}\] \[P(\text{Sick} | +)=\frac{(0.99)(0.001)}{0.05094}=0.0194=1.94\%\] Even though the test is 99% accurate, the chance you are actually sick given a positive result is only about 1.94%.

• Why? Because the disease is so rare (the Prior) that even with a positive test, it is more likely you are a “false positive” from the 99.9% healthy population than a “true positive” from the 0.1% sick population.

• Most people intuitively think: “The test is 99% accurate → I must be 99% likely sick.”

• But because the disease is extremely rare, the false positives dominate.

• Out of 1000 people, ~1 is actually sick, but ~50 healthy people will also test positive → most positives are false alarms.