I looked at a scam-call detection project that compares a bunch of models, and picked confidence intervals as the stats topic.

The question is simple: if one model scores higher than another, is it actually better, or is the difference just noise?

Point estimates alone cannot answer that. You need confidence intervals.

Six classifier models were tested on the same set of real phone calls (n = 1423). Here are the reported scores.

F1 is the detection score. Latency and size are the cost of running the model.

The score is F1, the harmonic mean of precision and recall:

\[ \text{Precision} = \frac{TP}{TP+FP}, \qquad \text{Recall} = \frac{TP}{TP+FN} \]

\[ F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \]

The model also smooths its per-sentence scores with an exponential moving average before deciding:

\[ s_t = \alpha\, p_t + (1-\alpha)\, s_{t-1} \]

To get a confidence interval for a messy metric like F1, resample the test set with replacement B times and recompute the metric each time:

\[ \hat\theta^{*(b)}, \quad b = 1, \dots, B, \quad B = 10{,}000 \]

The 95% interval is just the 2.5% and 97.5% quantiles of those values:

\[ \text{CI}_{95\%} = \left[\, \hat\theta^{*}_{(0.025)},\ \hat\theta^{*}_{(0.975)} \,\right] \]

Non-overlapping intervals are evidence of a difference. To actually compare two models you bootstrap the difference between their scores, since overlap by itself does not prove they are equal.

# percentile bootstrap CI (demo on a proportion, not F1)
boot_ci <- function(x, B = 10000) {
  vals <- replicate(B, mean(sample(x, length(x), replace = TRUE)))
  unname(quantile(vals, c(0.025, 0.975)))
}

set.seed(1)
# simulated correct/wrong outcomes for one model at about 0.79 accuracy
acc <- rbinom(1423, 1, 0.79)
boot_ci(acc)

## [1] 0.7765285 0.8186929

This only shows how the bootstrap gives a CI for a proportion. A real F1 comparison needs the paired per-call predictions (TP, FP, FN) from the test set, not a single proportion, so I am not using this to claim two models are equal.

DistilBERT has the highest F1. The other models spread out below it, so it is not a tie. The smallest model (LR hand, 0.001 MB) scores lower but is still okay.

These are the 95% bootstrap intervals reported in the project (resampled over per-call predictions). They do not overlap, which is evidence that retraining helped on this held-out channel.

Bottom corner is cheap and small. DistilBERT is off on its own, paying a lot for a bit more F1.

• I compared several scam detection models using F1, latency, and model size.
• DistilBERT had the highest F1, but it was much larger and slower.
• The smaller models had lower F1 but were far more efficient.
• Confidence intervals show the uncertainty in a result. To compare two models properly you bootstrap the difference between their scores, not just check whether their intervals overlap.