2025-06-08

What is a P-value?

A p-value is the probability of observing a test statistic as extreme as, or more extreme than, the value observed, assuming the null hypothesis is true.

\[ p = P(\text{observed or more extreme statistic} \mid H_0 \text{ true}) \]

The t-statistic formula is:

\[ t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} \]

Hypothesis Testing: The Basics

Hypothesis testing is a formal statistical method used to make decisions or inferences about population parameters using sample data.

We start with two competing claims:

  • Null hypothesis (\(H_0\)): No effect or no difference.
    Example: \(H_0: \mu = 120\)

  • Alternative hypothesis (\(H_A\)): There is an effect or a difference.
    Example: \(H_A: \mu < 120\)

The Goal

We use sample data to decide whether there’s enough evidence to reject \(H_0\).

We compute a test statistic (like a t-value), which tells us how far our sample result is from the null hypothesis, in standard error units.

The Role of the P-value

  • The p-value is the probability of getting a result as extreme as the one observed, assuming \(H_0\) is true.
  • A small p-value means the observed data is unlikely under \(H_0\), and we consider rejecting it.

Typical Decision Rule

If the p-value < α (commonly 0.05), we reject \(H_0\).
Otherwise, we fail to reject \(H_0\).

This does not mean we “accept” \(H_0\) — it just means there’s not enough evidence against it.

Example: Testing a Mean

Suppose we want to test whether a new drug lowers average blood pressure.

We believe the normal blood pressure is 120 mmHg, but we suspect the drug reduces it.

We take a random sample of 30 patients after using the drug, and record their average blood pressure.

We use the following hypotheses:

  • Null hypothesis: \(H_0: \mu = 120\)
  • Alternative hypothesis: \(H_A: \mu < 120\)

We perform a one-sample t-test. The p-value tells us the probability of observing a sample mean this low (or lower) if the true mean is still 120.

Interpreting the P-value

  • If the p-value < 0.05, we have strong evidence that the mean blood pressure is less than 120, and we reject \(H_0\).
  • If the p-value ≥ 0.05, the evidence is weak, and we fail to reject \(H_0\).

This helps us decide whether the drug likely has a real effect on lowering blood pressure.

Simulated Data and Code

## [1] 0.01848519

Simulate data

Conclusion

  • The p-value is the probability of obtaining results at least as extreme as those observed, assuming the null hypothesis is true.
  • A small p-value (typically < 0.05) suggests strong evidence against the null hypothesis.
  • A large p-value indicates weak evidence against the null — we fail to reject the null hypothesis.
  • P-values do not:
    • Tell us the probability that the null hypothesis is true.
    • Measure the size or importance of an effect.
  • Always interpret p-values in context, alongside confidence intervals, effect sizes, and domain knowledge.