2026-02-06

WHY HYPOTHESIS TESTING MATTERS?

A key question answered by statistics is whether an observed difference is simply due to random variation or is actually meaningful. Hypothesis testing provides the framework to evaluate data against a default assumption to decide whether results are “real” or just by chance.These methods are widely used in healthcare, science, engineering, and public policy.

This cartoon shows what can go wrong when conclusions are drawn before data is objectively analyzed. Hypothesis testing helps prevent issues by requiring clear and formal assumptions and decision rules.

THE NULL AND ALTERNATIVE HYPOTHESES

All statistical testing begins with making two competing statements:

the first is: \[ H_0: \mu = \mu_0 \]

and the second being: \[ H_a: \mu \neq \mu_0 \]

** The null hypothesis \(H_0\) represents the default assumption **

** The alternative hypothesis \(H_a\) represents the competing claim **

TEST STATISTIC AND P-VALUE

To determine whether the null hypothesis is likely to be false, we calculate a test statistic.

\[ t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \]

The p-value represents the probability, assuming \(H_0\) is true, of observing a result as/or more extreme than the one obtained.

DECISION RULE

A significance level \(\alpha\) is chosen in advance, most commonly 0.05 but can be other values such as 0.01.

** If the p-value \(\le \alpha\), we will reject the null hypothesis.

** If the p-value \(> \alpha\), we fail to reject the null hypothesis.

KEY NOTE: Failing to reject the null hypothesis does not mean it is true, but there is insufficient evidence against it.

SAMPLING DISTRIBUTION AND REGIONS OF REJECTION

WHAT THIS GRAPH SHOWS

– The curve shows the t-values we would expect to see just by chance if the null hypothesis were true.

– The shaded areas are the most extreme values, which are unlikely to occur from random variation alone.

– If our calculated t-value falls in one of these shaded areas, we reject the null hypothesis.

EXAMPLE: WHEN A TEST REJECTS AND WHEN IT DOESN’T

We can use the same sample data, but test two different null hypotheses to see how conclusions can change.

# SAMPLE DATA
x <- rnorm(25, mean = 126, sd =12)

# TWO HYPOTHESIS TESTS
test_100 <- t.test(x, mu = 100)
test_126 <- t.test(x, mu = 126)
## [1] 4.556511e-11
## [1] 0.8617487

When testing against 100: p-value is smaller than 0.05, so we reject the null.

When testing against 126: p-value is larger than 0.05, so we fail to reject the null.

HOW P-VALUES BEHAVE UNDER THE NULL HYPOTHESIS

The R code below can create many samples that all assume the null hypothesis to be true and calculate a corresponding p-value for each sample.

p_values <- replicate(2000, {
  sample_data <- rnorm(20, mean = 0, sd = 1)
  t.test(sample_data, mu = 0)$p.value
  })

df_p <- data.frame(p = p_values)

DISTRIBUTION OF P-VALUES UNDER THE NULL HYPOTHESIS

WHAT THIS SHOWS – When the null hypothesis is true, p-values are spread between 0 and 1. – Very small p-values occur sparingly just by chance. – Explains why a small p-value is evidence against the null hypothesis.

HOW P-VALUES CHANGE WITH SAMPLE SIZE AND EFFECT SIZE

INTERPRETING THE 3D PLOT - Larger effect sizes tend to produce smaller p-values. - Increasing the sample size reduces p-values. - Statistical significance depends on both effect and sample size, not simply p-values.

CONCLUSION

  • Hypothesis testing provides a structured approach in making decisions under uncertainty.

  • The p-value is quantifiable evidence against the null hypothesis, but it is not probability that it is true.

  • Sample and effect size can influence statistical significance.

  • A clear hypothesis and initial decision rules help reduce bias and data misinterpretation.