Why care about mulitplicity

You might have heard about a problem called multiplicity. Basically the problem is this; you do a test with an error rate of 5% (0.05).
This means that every time you do the test you have 5% chance of wrongly concluding something is happening while in fact nothing is. Also called a type I error. In science this is a big issue.

Some people would show you this picture:

But for me it was easier to understand (ie. remember) this little situation.
You are commiting a type I error if you fail to ask that girl out because you think she might not like you while in fact she does like you.

The type II error is taking that girl out concluding she likes you afterwards but that is actually not true (girl doesn’t like you back).

There is also the Type III error, and that is mistaking the type I for the type II.

Anyhow, back to multiplicity.
Imagine you do this test more then once. Naively (like i did once upon a time) you think this error stays the same.

Turns out not so!

The code below shows you what the probability becomes of getting at least one type I error if you would repeat the test 10 times.

It turns out the answer is not 5% like we would hope for. But rather 40%! That is like a lot more likely right?

ntests <- (1:10) # how many times we do the test
  multicomp.issue <- function(a,k) { # a is my alpha level, k is how many different tests i am doing
    b <- rep(NA,length(k))
    for (i in k) { 
      b[i] <-1-(1-a)^k[i]
    } 
    
    plot(b, ylim = 0:1, 
         ylab = "Probability",
         main ="Multiplicity error", type = "p", pch = 15)
    abline(h=0.5, col = "red")
    abline(h=0.05, col = "green")
    
    print("The family wise error ratre is equal to")
    print(b)
    
  }

multicomp.issue(0.05, ntests)

[1] "The family wise error ratre is equal to"
 [1] 0.0500000 0.0975000 0.1426250 0.1854938 0.2262191 0.2649081 0.3016627 0.3365796
 [9] 0.3697506 0.4012631

What if you do this 50 times?

Just for fun.

multicomp.issue(0.05, ntests.2)

[1] "The family wise error ratre is equal to"
 [1] 0.0500000 0.0975000 0.1426250 0.1854938 0.2262191 0.2649081 0.3016627 0.3365796
 [9] 0.3697506 0.4012631 0.4311999 0.4596399 0.4866579 0.5123250 0.5367088 0.5598733
[17] 0.5818797 0.6027857 0.6226464 0.6415141 0.6594384 0.6764665 0.6926431 0.7080110
[25] 0.7226104 0.7364799 0.7496559 0.7621731 0.7740645 0.7853612 0.7960932 0.8062885
[33] 0.8159741 0.8251754 0.8339166 0.8422208 0.8501097 0.8576043 0.8647240 0.8714878
[41] 0.8779135 0.8840178 0.8898169 0.8953260 0.9005597 0.9055318 0.9102552 0.9147424
[49] 0.9190053 0.9230550

Well it turns out that after 15 tries you get to chances slightly better then a coinflip of making that type I error.