Error type I simulation

Well having decided a threshold of decision of alpha 5% that is one out of 20th.time that an extreme values of Test statistics is taken and might occur Well having your sample in hand with a pval<0.05 you hit an possible effect on your study sensu.lato (concept not developed for simplicity), Bu some phenomena significantly occurs just by chance ! But it might be also by rejecting Ho in this condition ,although really true, you make a Mistake:I.e a drug waas found significant by testing a sample of patients but the effect size was so small that the test cannot detect according to reseracher They conclude on that sample a type I error occured and the p.val <0.05 was just random chance variation.(Source Roche,2002)

Lets try to demonstrate that with a SIMULATION

head(rnorm(100),5)#make a Z (N;0,1) with 100 idd
## [1] -0.98351842 -0.85216133  0.35719105 -0.01749374 -0.99607016

This is running one Rnorm function. Now imagine that rnorm it is derived from a sample of people of (0,1) and we know that this rnorm function is a RANDOM vector on Normally distributed iid’s.

We try to run 100 time this rnorm function generating 100ix 100j random numbers.

AA=(replicate(100,rnorm(100)))
AA[1:5,5:10]
##            [,1]        [,2]      [,3]        [,4]        [,5]       [,6]
## [1,] -1.1019901 -0.24537992  0.950902  0.92169288 -2.86268674 -0.9513930
## [2,]  0.2115887 -0.99692254 -1.101162 -2.13738689  1.13147338 -0.1462044
## [3,]  1.1033634 -1.66794570  1.559717  0.02192985 -0.06644439 -0.1000934
## [4,] -1.3206597  0.96818325 -1.279899  2.08967428 -0.47493751 -1.0602002
## [5,]  0.9390729 -0.07845429 -1.180059 -2.61180421  0.28342233  0.9466996
hist(AA)

Now we will run a t.test expecting the true mean to be mu=0 (population) and the difference of course to be normally distributed around 0 meaning in this particular case of Ho is true

Note: one.sided or two sided test should be decided by researcher before data acquisition stage!.

My question!

How often you expect a T test to be significant when in this case when Ho is by simulation True???

set.seed(123)#set a seed to have the same random kernel generator
A=replicate(100,t.test(rnorm(100))$p.value)#the distribution of p value seems to be a UNIFOM!!
#you must run 1000 of replicate but takes time ....
A
##   [1] 0.324389813 0.268752102 0.207695337 0.728050657 0.287262226 0.653256534
##   [7] 0.148736826 0.297079257 0.375730610 0.851180967 0.252930856 0.847976794
##  [13] 0.978225746 0.553933887 0.347997153 0.690719264 0.418742379 0.762418958
##  [19] 0.533644754 0.504107909 0.184759776 0.128631295 0.263368695 0.472120159
##  [25] 0.702781788 0.279260998 0.269674805 0.872524346 0.641237437 0.388971941
##  [31] 0.618857351 0.079361008 0.719264429 0.546307496 0.960348074 0.530020831
##  [37] 0.856573204 0.005315825 0.430221682 0.178260473 0.453253072 0.591490665
##  [43] 0.904232550 0.704391394 0.581169005 0.541944973 0.537507101 0.496108375
##  [49] 0.572776774 0.427060550 0.446148901 0.681457222 0.419206341 0.726559242
##  [55] 0.354333279 0.427223622 0.821041692 0.889181830 0.721994124 0.174483413
##  [61] 0.582841895 0.041855416 0.438225408 0.108421606 0.228183889 0.113582139
##  [67] 0.295092087 0.113930313 0.628126568 0.242490714 0.619848611 0.575794363
##  [73] 0.071446567 0.958700951 0.283164044 0.176742172 0.325283663 0.699315416
##  [79] 0.604416854 0.100861031 0.651841408 0.959975361 0.200319545 0.113292846
##  [85] 0.886863127 0.446236116 0.609685042 0.818119228 0.299672449 0.064153490
##  [91] 0.944212723 0.534538924 0.573077726 0.275250678 0.602845147 0.526679362
##  [97] 0.916927986 0.321578750 0.208857025 0.730151690
sum(A<0.05)###
## [1] 2
summary(A)
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
## 0.005316 0.273857 0.500108 0.485249 0.683773 0.978226
plot(A,main="100 pvalue of t-test on rnorm(100)")
abline(h=0.05,col=2)
legend("topright",legend=c("5 % alpha"),col=2,lty=1)

You’ll see that after 100 experiment just by chance [2-7 run 10 x replicate ] is founded SIGNIFICANT at the 5% level alpha. That prove I will commit a type one error I in this case by rejecting the null if my test fall in these 2-7 random t.test case.

What conclusion can be drawn?

The decision of 5% is arbitrary but usually it s a a common belief and good practice of random phenomena seems to occurs in that range (see historical discussion between famous statistician never ends…). Anyhow you might decide for good reasons (JUSTIFICATIONS needed!) to change that level (effect size, other study…). But STATISTICS TESTING WILL NOT GARANTEE YOU to erroneously concluded to have found an effect if p.value is below 5% with your sample in hand. Of course we nor all agree about the interpretation of strength of evidence given by the magnitude of the p-value: If I found a p.vale <0.0001 instead of 0.04 in example it simply advised me to go deeper in the data and look for a possible effect and which (bio-phys-chem) phenomena is acting!

TYPE I,II is a little bit like:

“CORRELATION IS NOT CAUSATION”of course… but some guy claimed that “causation is all about correlation” (by apply the Bayes theorem..): Well I tell them what about if the relationship is not linear and we know that a H1 is true…

Long talk short enough: Stay logic and factual!

Multiple testing

This simulation hope shows you that multiple testing on repeating same experiments with same subjects might lead to erroneous conclusions.

Multiple testing rules are strict (Bonferonni,Holmes) and must be applied if multiple testing is envisioned in your DOE.