Part 5: Making Statistical Decisions Through Simulation

On Tuesday, June 18, we simulated random selection of airline pilots for promotion. We wanted to see whether there was evidence of gender bias in pilot selection.

Data simulated in R using 1,000 replications (trials) and 10,000 replications:

Example: Compare and contrast the two simulated histograms with our in-class simulation. Does the pattern change much? Did we ever observe 7 females chosen for promotion?

I used the tally function to estimate some probabilities based on the 10,000 simulated data points.

     0      1      2      3      4      5      6      7  Total 
0.0057 0.0605 0.2100 0.3295 0.2630 0.1084 0.0213 0.0016 1.0000 

Example: Use the estimated probability distribution to find the probability of choosing at least 5 females for management.

0.1084 + 0.0213 + 0.0016 = 0.1313

The probability of choosing at least 5 females for management is what we would refer to in statistics as a p-value. A p-value is the probability of observing a sample result as or more extreme than the one we observed.

Example: Is this p-value, 0.1304, large enough for us to believe that the pilots weren't selected at random? Why or why not?

It's likely that the pilots were actually selected at once. There's a roughly 13% chance of observing an outcome at least as high as the one we observed. While it's not the most common outcome, it's definitely possible!

Simulations are an incredibly powerful tool for us in statistics! They let us take advantage of computing power to estimate sampling distributions.

Sampling distribution: the probability distribution of a sample statistic. The sampling distribution describes how likely a certain sample statistic is for a given sample!

That's what we've done with our airline pilot example. So far, we have:

• Determined the statistic we're interested in (number of female pilots)
• Replicated the random lottery process a large (10,000) number of times
• Made a conclusion based on the estimated p-value from our randomization

Simulations are a natural and intutitive way to make statistical decisions. With a simulation study we aren't just relying on our imaginations or knowledge of probability theory – we can see exactly what happens with repeated random samples.

Example: Can People Distinguish Dog Food from Duck Pate?

On Homework 4, you read a study which investigated whether dog food was distinguishable from a variety of other products. The authors of the study used traditional statistical methods to analyze their data. Before we look at how to do traditional statistical tests, let's try using a simulation study.

The research objective we're interested in is:

• Can humans correctly identify dog food from a random sample, or are they just guessing?

In the study we read, 3 out of 18 participants correctly identified the dog food. To do this simulation, we need a few more pieces of information:

• What's the probability of randomly choosing the dog food out of 5 different foods?

There are 5 foods, so if this is random chance the probability is 0.20.

• How many times should we try to repeat this study to get a good idea of the sampling distribution?

As many times as we can!

• How should we simulate this data?

Suggestions?

We can think of this experiment as a weighted coin. 20% of the time, a subject will correctly guess which is the dog food – this is like a coin with 20% probability of heads. Since we have a “weighted coin scenario”, we can use R to quickly simulate the data. With the rflip function, we can simulate flipping 18 coins (n=18) that each have a 20% chance of coming up heads (prob=0.2).

rflip(n=18, prob=0.20)

Flipping 18 coins [ Prob(Heads) = 0.2 ] ...

T H T H T T T H H T T T T T T H T T

Result: 5 heads.

rflip(n=18, prob=0.20)

Flipping 18 coins [ Prob(Heads) = 0.2 ] ...

T H T T T T T T T T H H T T H T T H

Result: 5 heads.

We need a way to speed this up! Using the do function, we can tell R to repeat this calculation many times.

• Simulate 50 random samples using the code below, and plot the “number of heads” (number of correct dog food identifications) in a histogram. Use the tally function to estimate the probabilities from your sampling distribution. Compare your histogram and estimated probabilities to a neighbors'. Are they exactly the same? Are they close?

This simulated sampling distribution represents the probabilities for each number of subjects correctly identifying the dog food (out of 18 total subjects), if they are all truly guessing.

• How likely is the value that we observed, 3 out of 18 correct guesses? Is this unusual? Based on this probability (p-value), do you think that study participants could really tell a difference and correctly identify the dog food?

It looks like 3 is actually the most likely outcome in a scenario where subjects were guessing completely at random! Based on this simulation, there's not much evidence to support the idea that subjects could identify the dog food.