Multiple Binomial Tests

How many times will 1 or more of 12 binomial tests on data from 40 participants come out significant by chance? (In other words, is the false positive rate inflated by running a large number of binomial tests.) We will run a large number of simulated experiments with random binomial data (prob=.5), and measure 1) in what proportion of experiments is there one or more significant binomial test? and 2) the expected number of significant binomial tests per experiment.

simulate_experiment <- function(tests, Nsubj, prob) {
  sig = 0 # times a question's binomial test is significant 
  for(t in 1:tests) {
    tab = table(rbinom(Nsubj, 1, prob))
    bt = binom.test(tab)
    if(bt$p.value < .05) sig = sig + 1 
  }
  return(sig)
}

sig_tests = rep(NA, 10000)
for(i in 1:10000) {
  sig_tests[i] = simulate_experiment(tests=12, Nsubj=40, prob=.5)
}

pct_sig = sum(sig_tests>0) / length(sig_tests)

In 38% of the 10,000 simulated experiments there was at least one significant binomial test. On average, there were 0.47 significant binomial tests (false positives) per experiment.