5 July 2015
What is probability?
Frequentism
- Probability is relative frequencies.
- Probability of event = Frequency of Event / Frequency of Total
- Examples: Coin flips, Dice rolls, Radioactive Decay…
Bayesianism
- Probability is subjective.
- Probability of an event = what odds would you bet at?
- Examples: Horse race betting, footy tipping, court cases…
What's the probability of Labor winning the next election?
Frequentist Answer
We can't measure it. It's a one-off event.
Bayesian Answer
What odds are you willing to bet that Labor will win?
There are other interpretations.
Interpretations of Probability
- Geometric
- Information Theoretic
- Thermodynamic
- Cryptographic
- Your Favourite Here
All Interpretations obey the Kolmogorov Axioms
Probability Axioms
- Probabilities are between 0 and 1
- The probability of an impossible event is 0.
- The probability of a certain event is 1.
- If A and B are mutually exclusive events, the probability of "A or B" occuring is equal to the probability of A + the probability of B.
Conditional Probability
\(P(A|B) = P(A&B)/P(B)\)
Problems!
Boy/Girl Problem - Take 1
You meet a proud mother of two children whose eldest is a girl. What's the probability that the youngest child is a girl?
- 50%
Boy/Girl Problem - Take 2
You meet another proud mother of two children. You ask her if at least one of her children is a girl, and she says yes. What's the probability that both children are girls?
- 33%
- Four possibilities: GG, GB, BG, BB
- Throw out the ones we learn are impossible
- Left with GG, GB, BG
Boy/Girl Problem - Take 3
You meet a yet another proud mother of two children. She brings one of them along at random to a party you throw. You greet the proud mother and her daughter warmly when they visit. What's the probability that both of her children are girls?
- 50%
- Eight Possibilities: GG, GG , GB , GB , BG, BG, BB, BB
- Throw out the ones we learn are impossible:
- Left with GG, GG , GB , BG
Monte Hall Problem.
Suppose a mother of two children is on a game show, and she's given the choice of three doors: Behind one door is a car; behind the others, goats. She picks a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to her, "Do you want to pick door No. 2?" Is it to her advantage to switch her choice?
- The host must always open a door that was not picked by the contestant.
- The host must always open a door to reveal a goat and never the car.
- The host must always offer the chance to switch between the originally chosen door and the remaining closed door.
Monte Hall Problem.
What's the probability of winning if she switches?
- 66%
Monte Hall Problem
Cancer Problem
- A mother of two children comes to you devastated. She's tested positively for breast cancer after taking a mammogram. What's the probability that she has breast cancer given the following facts?
- 1% of women have breast cancer (and therefore 99% do not).
- 80% of mammograms detect breast cancer when it is there (and therefore 20% miss it).
- 9.6% of mammograms detect breast cancer when it’s not there (and therefore 90.4% correctly return a negative result).
Cancer Problem Answers
- The probability that the mother has cancer is 7.76%.
Cancer Problem Explanation
\(P(\text{Cancer} | \text{Positive}) = \frac{P(\text{Positive} | \text{Cancer}) P(\text{Cancer})}{P(\text{Positive})}\)
\(\tiny{P(Positive) = P(Positive|Cancer) P(Cancer) + P(Positive | No Cancer)P(No Cancer)}\)
\(P(Cancer | Positive) = \frac{80\% \times 1\%}{80\% \times 1\% + 9.6\% \times 99\%} = 7.76\%\)
Bayes Theorem
\(P(\text{H} | \text{E}) = \frac{P(\text{E} | \text{H}) P(H) }{P(\text{E})}\)
\({P(E) = P(E | H1) P(H1) + P(E | H2) P(H2) + \ldots}\)
These equations are used as the basis for Bayesian Statistics.
Case Study (Murder and Battered Wives)
The Court Hearing:
- The Case: A woman has been found murdered in her home. Police arrested her husband.
- Prosecution: The victim's husband battered her. Wife batterers are more likely to murder their wives.
- Defense: P(Murdered by Husband | Battered) = 1 in 2500
- Prosecution: \(\tiny{P(\text{Murdered by Husband} | \text{Battered and Murdered} ) = \text{8 in 9}}\)
- In reality, the Prosecution failed to make the counter argument, and the husband walked free.
- Know your bayes, for justice.
A Brief History of Bayesian Methods
Bad old days
- Had to rely on Null Hypothesis Significance Testing
- Couldn't compute Bayes' Theorem for Real World Problems (Combinatorial Explosion)
Slightly Better old days
- Had computers.
- Could make "Monte Carlo Simulations" for Frequentist Reasoning
- Still couldn't compute Bayes' Theorem for Real World Problems
A Breakthrough!
Today
- Spam Filtering
- Robotics
- Bayesian Statistics
- Translation
- Causal Inference
Case Study (Spam Filtering)
The Naive Bayes Classification Algorithm
- Assumes independence of words (hence naive)
- What is P(Spam | Email Contains "Viagra")
- Probabilities obtained from emails marked either spam or not spam
- Very fast to compute bayes theorem for each word and multiply the results.
- No more spam!
One more problem
Who is John?
John is a man who wears gothic inspired clothing, has long black hair, and listens to death metal. How likely is it that he is a Christian and how likely is it that he is a Satanist?
- Remember that there are 2 billion Christians in the world, and only a few thousand Satanists.
- Moral: Good Bayesians pay less attention to the evidence at hand.
Bayesian Reasoning in the real world
Use reference class forecasting:
- Identify a reference class of past, similar projects
- Establish a probability distribution for the selected reference class for the parameter that is being forecast.
- Compare the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.
- Be vigilant for opportunities to change your mind.
Final Takeaway
In the real world, there are no certainties, only probabilities.