The p-value is the probability, computed under the assumption that the null hypothesis \(H_0\) is true, of observing a test statistic at least as extreme as the one actually observed.

• It is not the probability that \(H_0\) is true.
• It is not the probability that the alternative hypothesis is true.
• It quantifies how surprising the observed data would be if \(H_0\) were true.

In a typical two-sided test for a population mean \(\mu\):

\[H_0: \mu = \mu_0 \qquad \text{vs.} \qquad H_a: \mu \neq \mu_0\]

Given a sample of size \(n\) with sample mean \(\bar{x}\) and sample standard deviation \(s\), the test statistic is

\[t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}\]

which follows a \(t\)-distribution with \(n-1\) degrees of freedom under \(H_0\).

For a two-sided test, the p-value is

\[p\text{-value} = 2 \, P\big(T_{n-1} \geq |t_{obs}|\big)\]

where \(T_{n-1}\) denotes a random variable with a \(t\)-distribution on \(n-1\) degrees of freedom, and \(t_{obs}\) is the observed test statistic.

Small p-values indicate the observed data would be unusual under \(H_0\), providing evidence against it.

We compare the p-value to a pre-chosen significance level \(\alpha\) (commonly 0.05):

\[ \text{Decision} = \begin{cases} \text{Reject } H_0 & \text{if } p\text{-value} \leq \alpha \\ \text{Fail to reject } H_0 & \text{if } p\text{-value} > \alpha \end{cases} \]

This threshold controls the long-run rate of Type I errors (false rejections) at \(\alpha\).

Suppose we measure the reaction time (in milliseconds) of 30 subjects and want to test whether the mean reaction time differs from 250 ms.

set.seed(42)
reaction_time <- rnorm(30, mean = 258, sd = 20)
head(reaction_time)

## [1] 285.4192 246.7060 265.2626 270.6573 266.0854 255.8775

We will test \(H_0: \mu = 250\) against \(H_a: \mu \neq 250\).

test_result <- t.test(reaction_time, mu = 250)
test_result

## 
##  One Sample t-test
## 
## data:  reaction_time
## t = 2.045, df = 29, p-value = 0.05002
## alternative hypothesis: true mean is not equal to 250
## 95 percent confidence interval:
##  249.9990 268.7444
## sample estimates:
## mean of x 
##  259.3717

The output above reports the test statistic, degrees of freedom, and the p-value used to make our decision.

• The p-value from our test was 0.05002.
• Since this is (much) less than \(\alpha = 0.05\), we reject \(H_0\).
• This provides strong evidence that the true mean reaction time differs from 250 ms.
• The p-value is a tool for quantifying evidence, not a measure of effect size or practical importance.

\[p\text{-value} = P(\text{data as extreme or more extreme} \mid H_0 \text{ true})\]

• A small p-value \(\Rightarrow\) data is unlikely under \(H_0\) \(\Rightarrow\) evidence against \(H_0\).
• The p-value depends on the chosen test statistic, sample size, and null hypothesis.
• Always report the p-value alongside effect size and confidence intervals for full context.