Psychological ‘Simulations’ & Bayes Theorem

Introduction

Part of the bread and butter of experimental psychology is the ability to accurately predict what a “normal” person would do. It turns out that this is often hard because once you start thinking about thinking, it is easy to think you would think something different from what you would actually think. It is even more difficult in a psychology class because you know, even if you don’t think about it consciously, that whatever you think a “normal” person would do is somehow the “wrong” thing to do, and people’s tendency is to not want to be like a “normal,” wrong person.

That is one of the things that is so annoying about Kahneman and Tversky’s, Ariely’s, and Gigerenzer’s work. You know that at some point, they are going to be saying, “You are wrong,” and no one wants to be wrong. (I will say that of the four, I think only Gigerenzer really seems to really enjoy it.)

One of the things you need to do as an experimental psychologist is realize that your System 1 is the same as everyone else’s. It’s actually a blessing as a psychologist because you can run simulations, and long as you allow your conscious System 2 to acknowledge your System 1. I’m probably mostly full of bologna here. My suspicion is that people like Kahneman and Tversky did not have much better access to System I than other people. I think that they were able to ask questions and consult past experiences from memory to think about what they did in their “natural habitat,” when they weren’t thinking about it.

After just having read Meehl (1973) in which he says everyone in psychological case conferences is an idiot (not really, but you get the idea), asking people things like I did in supervision:

If the base rate for a disorder is \(P(D) = .01\), the probability of a sign such as hearing voices given that someone has the disorder is \(P(S|D) = .45\), and the probability of the same sign given that someone does not have the disorder is \(P(S| \neg D) = .10\)

was dumb. It was dumb because you cannot access your System 1 response, and System 2 is saying, “Don’t be an idiot. Don’t be an idiot.” It takes a certain integrity, though, to make an honest guess about what your System 1 would say. I am guessing that you guys are not to different from physicians. Here’s what they do:

Try your hand at a typical problem, first presented to 100 physicians by Eddy (1982). The problem is interpreting results of screening mammograms.

Here is the background information (The terminology in parentheses was not provided):

1. In the absence of any special information, the probability that a woman (of the age and health status of this patient) has breast cancer is 1%. (This is the prevalence rate or base rate for the disease.)
2. If the patient has breast cancer, the probability that the radiologist will correctly diagnose it is 80%. (This is the sensitivity of the test.)
3. If the patient has a benign lesion (no breast cancer), the probability that the radiologist will incorrectly diagnose it as cancer is 10%. (This is the false-positive rate of the test.)

The question: What is the probability that a patient with a positive mammogram actually has breast cancer?

Ninety-five of the 100 physicians estimated the probability to be about 75%. Does that sound about right? (from http://instructional1.calstatela.edu/dweiss/psy302/confusion.htm)

The Controversy

Given the context, I had induced confusion, so my memory is that Alex and Lauren spoke up with what turned out to be pretty good estimates, and then, awesomely, Lauren wanted to keep going. Lauren helped me to figure out what her formula was. Here it is: \[p(D|S) = \frac{p(S|D)\,p(D)}{p(S|\neg D)}.\] Here is actual Bayes: \[p(D|S) = \frac{p(S|D)\,p(D)}{p(S|D)\,p(D) + p(S|\neg D)p(\neg D)}.\]

The Answer

Once you see the formulas together, you can see why Lauren’s formula worked in class. In both examples we worked out, \(p(D)\) was very small. If you look at Bayes’ denominator, when \(p(D)\) is very small, the left-hand term is very close to 0, and the right-hand term is very close to \(p(S|\neg D)\).

This figure shows “posterior probabilities” according to “Bell’s Rule” vs. actual posteriors according to Bayes’ Rule for a range of priors extending from 0 to 1 under the conditions I described in class. So, yup, as the priors approach 0, the two rules converge. Unfortunately, you need Bayes’ whole formula, probably, once the priors exceed .05 or so.

I threw in the value for a base rate of .5. If someone in the population was just as likely to have the disorder as not have the disorder, having the symptom or showing the sign would yield about an 80% chance of having the disorder. Personally, most base rate problems I see are in the context of low base rates, so that is pretty surprising for me. But if you think about it in the context of a 50% prior, it has to be higher than that, right?

Here is the same figure for the physician case describe above. I think it is really interesting how important the base rate is relative to the sensitivity and specificity of the test. The difference between the two examples is that the sensitivity went from 45% to 80%, but the equal-base-rate answer only went from about 82% to about 89%.

So, to follow up with Meehl, it is worthwhile to know what is really going on. He points out a number of times that base rates near 1 can be just as problematic as base rates near 0. If something is common, “diagnostic” information barely changes the posterior.