Feeding the benchmark data from https://konghq.quip.com/6r46AZ9Ebfee/gRPC-Notes into “Tukey’s Honestly Significant Difference Test” and looking at the 95% confidence interval suggestions that:

Is that range within our comfort level e.g. to say “close enough”? If so we are done.

If we would prefer to have a more specific range then we could rerun the test for more data (e.g. 30+ results per branch) and then rerun the analysis to narrow down the range of likely values.

Implementation

library(tibble)
data <- tribble(
  ~version, ~requests_per_second,
  "PR144", 7594.27,
  "PR144", 7639.59,
  "PR144", 7377.62,
  "PR145", 7894.04,
  "PR145", 7705.10,
  "PR145", 7741.54)

Compare branches using “Tukey’s Honestly Significant Difference” test.

kong.lm <- lm(requests_per_second ~ version, data)
kong.aov <- aov(kong.lm)
kong.tukey <- TukeyHSD(kong.aov)
kong.tukey
  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = kong.lm)

$version
                diff       lwr      upr     p adj
PR145-PR144 243.0667 -32.95613 519.0895 0.0708294

Same information visually:

plot(kong.tukey)

LS0tCnRpdGxlOiAiZ1JQQyBiZW5jaG1hcmsgcmVzdWx0IGNvbXBhcmlzb24iCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCkZlZWRpbmcgdGhlIGJlbmNobWFyayBkYXRhIGZyb20gaHR0cHM6Ly9rb25naHEucXVpcC5jb20vNnI0NkFaOUViZmVlL2dSUEMtTm90ZXMgaW50byAiW1R1a2V5J3MgSG9uZXN0bHkgU2lnbmlmaWNhbnQgRGlmZmVyZW5jZSBUZXN0XShodHRwczovL3NjaWVuY2luZy5jb20vd2hhdC1pcy10aGUtdHVrZXktaHNkLXRlc3QtMTI3NTE3NDguaHRtbCkiIGFuZCBsb29raW5nIGF0IHRoZSA5NSUgY29uZmlkZW5jZSBpbnRlcnZhbCBzdWdnZXN0aW9ucyB0aGF0OgoKLSBFaXRoZXIgYnJhbmNoIG1pZ2h0IGJlIGZhc3RlcjsKLSBQUjE0NSB3b3VsZCBiZSAzMyByZXEvcyBzbG93ZXIgaW4gdGhlIHdvcnN0IGNhc2U7Ci0gUFIxNDUgd291bGQgYmUgNTE5IHJlcS9zIGZhc3RlciBpbiB0aGUgYmVzdCBjYXNlLgoKSXMgdGhhdCByYW5nZSB3aXRoaW4gb3VyIGNvbWZvcnQgbGV2ZWwgZS5nLiB0byBzYXkgImNsb3NlIGVub3VnaCI/IElmIHNvIHdlIGFyZSBkb25lLgoKSWYgd2Ugd291bGQgcHJlZmVyIHRvIGhhdmUgYSBtb3JlIHNwZWNpZmljIHJhbmdlIHRoZW4gd2UgY291bGQgcmVydW4gdGhlIHRlc3QgZm9yIG1vcmUgZGF0YSAoZS5nLiAzMCsgcmVzdWx0cyBwZXIgYnJhbmNoKSBhbmQgdGhlbiByZXJ1biB0aGUgYW5hbHlzaXMgdG8gbmFycm93IGRvd24gdGhlIHJhbmdlIG9mIGxpa2VseSB2YWx1ZXMuCgojIyMgSW1wbGVtZW50YXRpb24KCmBgYHtyfQpsaWJyYXJ5KHRpYmJsZSkKCmRhdGEgPC0gdHJpYmJsZSgKICB+dmVyc2lvbiwgfnJlcXVlc3RzX3Blcl9zZWNvbmQsCiAgIlBSMTQ0IiwgNzU5NC4yNywKICAiUFIxNDQiLCA3NjM5LjU5LAogICJQUjE0NCIsIDczNzcuNjIsCiAgIlBSMTQ1IiwgNzg5NC4wNCwKICAiUFIxNDUiLCA3NzA1LjEwLAogICJQUjE0NSIsIDc3NDEuNTQpCmBgYAoKQ29tcGFyZSBicmFuY2hlcyB1c2luZyAiVHVrZXkncyBIb25lc3RseSBTaWduaWZpY2FudCBEaWZmZXJlbmNlIiB0ZXN0LgoKYGBge3J9CmtvbmcubG0gPC0gbG0ocmVxdWVzdHNfcGVyX3NlY29uZCB+IHZlcnNpb24sIGRhdGEpCmtvbmcuYW92IDwtIGFvdihrb25nLmxtKQprb25nLnR1a2V5IDwtIFR1a2V5SFNEKGtvbmcuYW92KQoKa29uZy50dWtleQpgYGAKClNhbWUgaW5mb3JtYXRpb24gdmlzdWFsbHk6CgpgYGB7cn0KcGxvdChrb25nLnR1a2V5KQpgYGAKCg==