2026-10-17

What does a 95% confidence interval really mean?

Let’s consider a hypothetical population model.

Imagine estimating a Minecraft server’s average response time in milliseconds where the true mean is 200 ms

One Sample, One Estimate

Imagine estimating a Minecraft server’s average response time in milliseconds:

  • 30 independent requests to the server (n = 30)
  • The sample mean estimates the population mean

Our sample mean: 198.6 milliseconds.

Question: How precise is this estimate for the entire model?

Building the interval

For independent observations from a normal population, a 95% t-confidence interval for the mean is:

\[ \bar{x} \pm t_{0.975,\;n-1}\frac{s}{\sqrt{n}} \]

  • \(\bar{x}\): sample mean
  • \(s\): sample standard deviation
  • \(n\): sample size
  • \(t_{0.975,\;n-1}\): critical value from the t-distribution

Our interval is [187.6, 209.6] milliseconds.

What Does 95% Mean?

Now, repeat this process with a new set of numbers:

  • take a new random sample
  • calculate the confidence interval
  • check if the the true population mean is within its limits

After doing this experiment many, many times, about 95% of intervals calculated will contain the true mean.

The intervals may change but the population mean stays fixed.

Repeating the Experiment

As you can see, most of the lines are red, meaning the true mean lies in the estimated interval. I can see only 4 out of 100 trials where the mean is not included in the interval, which is 96% accuracy. Pretty close to 95%!

Comparing Sample Sizes and Width of Interval

If you start with a small sample size, the interval containing the true mean is quite broad.

As you do more experiments, the interval becomes more concentrated, narrowing your margins.

Takeaways

  • A confidence interval lets you estimate the range of an average when you cannot measure from a large population
  • “95% confidence” describes the accuracy in the long term
  • It does not mean that 95% of individual observations will fall inside the interval
  • In 100 trials, the number that contains the true mean can vary from trial to trial

Source: NIST 1.3.4.2 Confidence Limits for the Mean