The method starts from a deliberately boring assumption: nothing is happening. We then check whether the data are hard to square with that assumption.

• \(H_0\) (null): the new method is no better. Any gap is chance.
• \(H_a\) (alternative): the new method genuinely raises the average score.

We summarize the data in a single test statistic that measures its distance from \(H_0\), then ask how often chance alone would produce a result at least that extreme.

If \(H_0\) were true, how often would we see a gap this big?

That frequency is the p-value. When it’s small, the null assumption and the data sit together badly, and we start to doubt the null.

Compare the two group means, \(\bar{x}_{\text{new}}\) and \(\bar{x}_{\text{std}}\).

\[ H_0:\; \mu_{\text{new}} = \mu_{\text{std}} \qquad\qquad H_a:\; \mu_{\text{new}} > \mu_{\text{std}} \]

The two-sample \(t\) statistic (Welch form, for unequal variances) is

\[ t \;=\; \frac{\bar{x}_{\text{new}} - \bar{x}_{\text{std}}} {\sqrt{\dfrac{s_{\text{new}}^{2}}{n_{\text{new}}} +\dfrac{s_{\text{std}}^{2}}{n_{\text{std}}}}} \]

The top is the observed gap between means. The bottom is roughly how much that gap would bounce around from sample to sample. Dividing one by the other puts the gap on a scale where a value near 0 is unremarkable and a large \(|t|\) is hard to explain by chance.

The p-value is the probability, assuming \(H_0\), of a \(t\) at least as large as the one we observed:

\[ p \;=\; P\!\left(T_{\nu} \ge t_{\text{obs}} \mid H_0\right) \]

with \(\nu\) the Welch-Satterthwaite degrees of freedom.

A 95% confidence interval for the true difference \(\mu_{\text{new}}-\mu_{\text{std}}\) is

\[ \left(\bar{x}_{\text{new}}-\bar{x}_{\text{std}}\right) \;\pm\; t_{\,0.975,\,\nu}\, \sqrt{\dfrac{s_{\text{new}}^{2}}{n_{\text{new}}} +\dfrac{s_{\text{std}}^{2}}{n_{\text{std}}}} \]

We fix a threshold \(\alpha = 0.05\) beforehand and reject \(H_0\) when \(p < \alpha\).

The experiment is simulated so the whole deck is reproducible: 40 students per group, scores drawn from normal distributions. Here is the test itself:

tt <- t.test(new_meth, standard,
             alternative = "greater",   # is the new mean higher?
             var.equal = FALSE)         # Welch, unequal variances

c(t_statistic = unname(tt$statistic),
  df          = unname(tt$parameter),
  p_value     = tt$p.value,
  mean_new    = mean(new_meth),
  mean_std    = mean(standard))

## t_statistic          df     p_value    mean_new    mean_std 
##  2.30326186 77.12901317  0.01197974 78.00780445 73.71101122

With this much overlap, you can’t read off a verdict by eye. That is what the test is for.

Power is the chance of detecting a real effect when there is one. It depends on two things we partly control: how big the true effect is, and how many people we test. The surface below shows power climbing with both. Drag to rotate it.

Hover a point for its score. Individual students are scattered all over; it’s the group averages that separate.

A significant result is not automatically an important one. With a large enough sample, even a trivial gap clears \(p < 0.05\), which is why the effect size and interval matter more than the p-value alone.

The test also leans on assumptions: roughly independent observations and approximately normal data. The Welch version used here drops the requirement that both groups share the same variance, but not the rest.

And this was simulated data, chosen so the result is reproducible. A real study would still need randomization, blinding, and hypotheses fixed in advance.

Question, idea, mathematics, R, visualization, insight. The whole analysis lived in one reproducible document.