Which effect will replicate?

E. J. Wagenmakers recently posted a Twitter poll asking:

You read two papers, each reporting three preregistered analyses. Which effect is more likely to replicate?

1. p=.024, p=.034, p=.031

2. p=.25, p=.032, p=.0001

Plenty of researchers struggle with the interpretation of p-values. There are two arguments for option (1):

1. It has more “significant” p-values (this is a bad argument).
2. Tighter p-values mean more consistent test statistics, which imply a more precise estimate of the effect size (a little better).

However, there are two reasons that option (2) is the better choice.

Distribution of p-values

For this first approach, I’ll generate fake data with a “small” to “medium” effect size, use Welch’s t-test to evaluate the effect, and examine the distribution of t statistics and p-values.

library(dplyr)
library(broom)
library(ggplot2)
set.seed(20812)
N <- 119
es <- 0.4
tTests <- data_frame(n = 1:1000) %>%
  group_by(n) %>%
  do(tidy(t.test(rnorm(N, 0, 1), rnorm(N, 0+es, 1))))

Here are the t statistics:

ggplot(tTests, aes(statistic, ..density..)) +
  geom_histogram(binwidth = 0.5)

Here are the resulting p-values:

ggplot(tTests, aes(p.value, y = ..density..)) +
  geom_histogram(binwidth = 0.01)

When the null hypothesis is false, the distrubition of p-values has positive skewness. The values from option (2) look more like what we’d expect from this distribution.

For example, the mean of this p-value distrubution is 0.028, which is close to the mean p-value for option (1). However, the probability of drawing three p-values from this distribution and seeing none below 0.01 is approximately:

mean(replicate(10000, all(sample(tTests$p.value, 3) > 0.01)))

[1] 0.0302

Effect size

Using the degrees of freedom in this example data, I can convert the p-values from the poll to t statistics and estimate Cohen’s d for effect size:

pVals <- matrix(c(0.024, 0.034, 0.031,
                  0.25, 0.032, 0.0001),
                nrow = 2, byrow = TRUE)
dof <- N*2 - 2
tVals <- qt(pVals/2, dof)
dVals <- abs(2*tVals) / sqrt(dof)
apply(dVals, 1, mean)

[1] 0.2853020 0.3154265

The p-values for option (2) imply a larger average effect size than those for option (1). If we assume that a successful “replication” means finding an effect where one exists, the second set of p-values the best choice.

Conclusion

Wagenmakers’s question is underspecified. However, the p-values in option (2) look more like what we’d expect from an actual set of replications, and also imply a larger effect.