• A p-value is the probability of obtaining test results at least as extreme as the observed results, assuming the null hypothesis is true
• Values range from 0 to 1
• Smaller p-values suggest stronger evidence against the null hypothesis
• Common significance levels: 0.05, 0.01

Let’s define p-value mathematically:

\[ p = P(|T| \geq |t| | H_0) \]

Where: - \(T\) is the test statistic - \(t\) is the observed value - \(H_0\) is the null hypothesis

The relationship between p-value and statistical power:

\[ \text{Power} = 1 - \beta = P(\text{reject } H_0 | H_1 \text{ is true}) \]

Where \(\beta\) is the probability of Type II error.

• P-value is NOT:
  • The probability that the null hypothesis is true
  • The probability of making a mistake
  • An effect size measure
  • A measure of practical significance

# Function to generate p-values from t-test
simulate_p_values <- function(n = 100, effect_size = 0) {
  p_values <- numeric(1000)
  for(i in 1:1000) {
    group1 <- rnorm(n)
    group2 <- rnorm(n, mean = effect_size)
    p_values[i] <- t.test(group1, group2)$p.value
  }
  return(p_values)
}

# Generate p-values with no effect
p_vals <- simulate_p_values()
mean(p_vals < 0.05) # Expected false positive rate

## [1] 0.034

• Cohen, J. (1988). Statistical power analysis for the behavioral sciences
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values
• Fisher, R. A. (1925). Statistical methods for research workers