Adjustment for multiple testing

References

• EPI288 material on multiple testing
• A Tutorial on False Discovery Control: http://www.stat.cmu.edu/~genovese/talks/hannover1-04.pdf
• Wikipedia FDR: http://en.wikipedia.org/wiki/False_discovery_rate

p-value adjustment methods

Bonferroni method

• \( p_{i Bon.} = p_i * k \) where \( p_i \) is the \( i \) -th smallest naive p-value among \( k \) tests.
• It is set to 1.0 if it is greater than 1.0.

Holm step-down method

• \( p_{i Holm} =p_i * (k + 1 - i) \) where \( p_i \) is the \( i \) -th smallest naive p-value among \( k \) tests.
• It proceeds from the smallest to the largest naive p-values.
• If the adjusted p-value is smaller than that of the previous step, the previous value is used again.
• After the first adjusted p-value greater than the cutoff, all the following tests are considered non-significant.

Hochberg step-up method

• \( p_{i Hoch.} =p_i * (k + 1 - i) \) where \( p_i \) is the \( i \) -th smallest naive p-value among \( k \) tests.
• It proceeds from the largest to the smallest naive p-values.
• If the adjusted p-value is larger than that of the previous step, the previous value is used again.
• After the first adjusted p-value less than the cutoff, all the following tests are considered significant.

False discovery rate (FDR) q-value method

• It controls the expected proportion of false positives among significant results, not the usual alpha error rate.
• \( q_{i FDR} = p_i * k / i \) where \( p_i \) is the \( i \) -th smallest naive p-value among \( k \) tests.
• It proceeds from the largest to the smallest naive p-values.
• If the adjusted q-value is larger than that of the previous step, the previous value is used again.
• After the first adjusted q-value less than the cutoff, all the following tests are considered significant.
• Among the significant results, (cutoff * 100)% are expected to be false positives.

EPI288 example

## Demo data
k10 <- c(0.001, 0.005, 0.01, 0.04, 0.10, 0.34, 0.40, 0.55, 0.62, 0.80)

## Sort by p-value (small to large)
k10 <- sort(k10)

## Bonferroni adjustment
bonferroni_p <- p.adjust(k10, method = "bonferroni")

## Holm adjustment
holm_p <- p.adjust(k10, method = "holm")

## Hochberg adjustment
hochberg_p <- p.adjust(k10, method = "hochberg")

## False discovery rate method q-value
fdr_q <- p.adjust(k10, method = "fdr")

## Show 
res.k10 <- data.frame(naive_p      = k10,
                      bonferroni_p = bonferroni_p,
                      crude_holm   = k10 * rev(seq_along(k10)),
                      holm_p       = holm_p,
                      hochberg_p   = hochberg_p,
                      crude_fdr    = k10 * length(k10) / seq_along(k10),
                      fdr_q        = fdr_q)
res.k10

   naive_p bonferroni_p crude_holm holm_p hochberg_p crude_fdr   fdr_q
1    0.001         0.01      0.010  0.010      0.010   0.01000 0.01000
2    0.005         0.05      0.045  0.045      0.045   0.02500 0.02500
3    0.010         0.10      0.080  0.080      0.080   0.03333 0.03333
4    0.040         0.40      0.280  0.280      0.280   0.10000 0.10000
5    0.100         1.00      0.600  0.600      0.600   0.20000 0.20000
6    0.340         1.00      1.700  1.000      0.800   0.56667 0.56667
7    0.400         1.00      1.600  1.000      0.800   0.57143 0.57143
8    0.550         1.00      1.650  1.000      0.800   0.68750 0.68750
9    0.620         1.00      1.240  1.000      0.800   0.68889 0.68889
10   0.800         1.00      0.800  1.000      0.800   0.80000 0.80000

Interpretation

• Naive p: 4 statistically significant results
• Bonferroni p: 1 statistically significant result
• Holm procedure: 2 statistically significant result. (Stop at the first adjusted p > 0.05 and consider all following non-significant regardless of their p-values).
• False discovery rate q-value: If q < 0.20 is used as the cutoff, 4 singnificant results exist. Among these “positive” results, 20% are expected to be “false positive” (4 * 0.2 = 0.8 false positives).
• The crude_holm and crude_fdr are intermediate results before further sequential adjustment.

Another example

## Enter demo data
p_demo <- c(0.02718, 0.06665, 0.02961, 0.3738, 0.006848, 0.6714, 0.851, 0.1006, 0.4537, 0.5765, 0.2245, 0.2521, 0.09896, 0.5975, 0.02204)

## Name these values
names(p_demo) <- paste0("test", seq(length(p_demo)))

## Sort by p-value (small to large)
p_demo <- sort(p_demo)

## Bonferroni adjustment
bonferroni_p <- p.adjust(p_demo, method = "bonferroni")

## Holm adjustment
holm_p <- p.adjust(p_demo, method = "holm")

## Hochberg adjustment
hochberg_p <- p.adjust(p_demo, method = "hochberg")

## False discovery rate method q-value
fdr_q <- p.adjust(p_demo, method = "fdr")

## Show 
res.p_demo <- data.frame(naive_p      = p_demo,
                      bonferroni_p = bonferroni_p,
                      crude_holm   = p_demo * rev(seq_along(p_demo)),
                      holm_p       = holm_p,
                      hochberg_p   = hochberg_p,
                      crude_fdr    = p_demo * length(p_demo) / seq_along(p_demo),
                      fdr_q        = fdr_q)
res.p_demo

        naive_p bonferroni_p crude_holm holm_p hochberg_p crude_fdr  fdr_q
test5  0.006848       0.1027     0.1027 0.1027     0.1027    0.1027 0.1027
test15 0.022040       0.3306     0.3086 0.3086     0.3086    0.1653 0.1110
test1  0.027180       0.4077     0.3533 0.3533     0.3533    0.1359 0.1110
test3  0.029610       0.4441     0.3553 0.3553     0.3553    0.1110 0.1110
test2  0.066650       0.9998     0.7331 0.7331     0.7331    0.2000 0.2000
test13 0.098960       1.0000     0.9896 0.9896     0.8510    0.2474 0.2156
test8  0.100600       1.0000     0.9054 0.9896     0.8510    0.2156 0.2156
test11 0.224500       1.0000     1.7960 1.0000     0.8510    0.4209 0.4202
test12 0.252100       1.0000     1.7647 1.0000     0.8510    0.4202 0.4202
test4  0.373800       1.0000     2.2428 1.0000     0.8510    0.5607 0.5607
test9  0.453700       1.0000     2.2685 1.0000     0.8510    0.6187 0.6187
test10 0.576500       1.0000     2.3060 1.0000     0.8510    0.7206 0.6894
test14 0.597500       1.0000     1.7925 1.0000     0.8510    0.6894 0.6894
test6  0.671400       1.0000     1.3428 1.0000     0.8510    0.7194 0.7194
test7  0.851000       1.0000     0.8510 1.0000     0.8510    0.8510 0.8510

• Naive p: 4 statistically significant results (tests 5, 15, 1, 3)
• Bonferroni p: No statistically significant result.
• Holm procedure: The very first adjusted p-value is non-significant (p > 0.05), thus, the following results are all non-significant regardless of their values (non-significant even if p < 0.05).
• False discovery rate q-value: If q < 0.20 is used as the cutoff, 4 singnificant results exist. Among these “positive” results, 20% are expected to be “false positive” (4 * 0.2 = 0.8).
• The crude_holm and crude_fdr are intermediate results before further sequential adjustment.