Summary

Feeding the benchmark data from 100 runs of grpc-pass (PR144) vs. feat.grps-auto (PR145) and looking at the 95% confidence interval suggests that:

We can’t say if one branch is faster than the other, but they are definitely close;
PR145 would be 13 req/s slower in the worst case;
PR145 would be 160 req/s faster in the best case.

Since these numbers are quite small we can be more confident that there is little or no performance difference between these branches on this benchmark.

Visualization

We can also visualize the results with a boxplot and see that there are some fairly extreme results amongst the tests, everything from 4000 req/s to 7500 req/s, although most of the results are tightly clustered around just under 6000 req/s.

ggplot(data, aes(y=requests_per_second, x=version, color=version)) +
  geom_boxplot() +
  labs(y="Requests/second", x="Branch") +
  expand_limits(y=0) +
  theme(legend.position = "none")

Implementation

library(tibble)
library(ggplot2)
data <- read.csv("/Users/lukegorrie/Downloads/bench_results/grpc.csv")

Compare branches using “Tukey’s Honestly Significant Difference” test.

kong.lm <- lm(requests_per_second ~ version, data)
kong.aov <- aov(kong.lm)
kong.tukey <- TukeyHSD(kong.aov)
kong.tukey

  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = kong.lm)

$version
                                 diff       lwr      upr     p adj
feat.grps-auto-feat.grpc-pass 73.6295 -13.33025 160.5892 0.0965544

Same information visually:

plot(kong.tukey)

LS0tCnRpdGxlOiAiZ1JQQyBiZW5jaG1hcmsgcmVzdWx0IGNvbXBhcmlzb24iCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCiMgU3VtbWFyeQoKRmVlZGluZyB0aGUgYmVuY2htYXJrIGRhdGEgZnJvbSAxMDAgcnVucyBvZiBgZ3JwYy1wYXNzYCAoUFIxNDQpIHZzLiBgZmVhdC5ncnBzLWF1dG9gIChQUjE0NSkgYW5kIGxvb2tpbmcgYXQgdGhlIDk1JSBjb25maWRlbmNlIGludGVydmFsIHN1Z2dlc3RzIHRoYXQ6CgotIFdlIGNhbid0IHNheSBpZiBvbmUgYnJhbmNoIGlzIGZhc3RlciB0aGFuIHRoZSBvdGhlciwgYnV0IHRoZXkgYXJlIGRlZmluaXRlbHkgY2xvc2U7Ci0gUFIxNDUgd291bGQgYmUgMTMgcmVxL3Mgc2xvd2VyIGluIHRoZSB3b3JzdCBjYXNlOwotIFBSMTQ1IHdvdWxkIGJlIDE2MCByZXEvcyBmYXN0ZXIgaW4gdGhlIGJlc3QgY2FzZS4KClNpbmNlIHRoZXNlIG51bWJlcnMgYXJlIHF1aXRlIHNtYWxsIHdlIGNhbiBiZSBtb3JlIGNvbmZpZGVudCB0aGF0IHRoZXJlIGlzIGxpdHRsZSBvciBubyBwZXJmb3JtYW5jZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlc2UgYnJhbmNoZXMgb24gdGhpcyBiZW5jaG1hcmsuCgojIFZpc3VhbGl6YXRpb24KCldlIGNhbiBhbHNvIHZpc3VhbGl6ZSB0aGUgcmVzdWx0cyB3aXRoIGEgW2JveHBsb3RdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0JveF9wbG90KSBhbmQgc2VlIHRoYXQgdGhlcmUgYXJlIHNvbWUgZmFpcmx5IGV4dHJlbWUgcmVzdWx0cyBhbW9uZ3N0IHRoZSB0ZXN0cywgZXZlcnl0aGluZyBmcm9tIDQwMDAgcmVxL3MgdG8gNzUwMCByZXEvcywgYWx0aG91Z2ggbW9zdCBvZiB0aGUgcmVzdWx0cyBhcmUgdGlnaHRseSBjbHVzdGVyZWQgYXJvdW5kIGp1c3QgdW5kZXIgNjAwMCByZXEvcy4KCmBgYHtyfQpnZ3Bsb3QoZGF0YSwgYWVzKHk9cmVxdWVzdHNfcGVyX3NlY29uZCwgeD12ZXJzaW9uLCBjb2xvcj12ZXJzaW9uKSkgKwogIGdlb21fYm94cGxvdCgpICsKICBsYWJzKHk9IlJlcXVlc3RzL3NlY29uZCIsIHg9IkJyYW5jaCIpICsKICBleHBhbmRfbGltaXRzKHk9MCkgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikKYGBgCgoKIyBJbXBsZW1lbnRhdGlvbgoKYGBge3J9CmxpYnJhcnkodGliYmxlKQpsaWJyYXJ5KGdncGxvdDIpCgpkYXRhIDwtIHJlYWQuY3N2KCIvVXNlcnMvbHVrZWdvcnJpZS9Eb3dubG9hZHMvYmVuY2hfcmVzdWx0cy9ncnBjLmNzdiIpCmBgYAoKCkNvbXBhcmUgYnJhbmNoZXMgdXNpbmcgIlR1a2V5J3MgSG9uZXN0bHkgU2lnbmlmaWNhbnQgRGlmZmVyZW5jZSIgdGVzdC4KCmBgYHtyfQprb25nLmxtIDwtIGxtKHJlcXVlc3RzX3Blcl9zZWNvbmQgfiB2ZXJzaW9uLCBkYXRhKQprb25nLmFvdiA8LSBhb3Yoa29uZy5sbSkKa29uZy50dWtleSA8LSBUdWtleUhTRChrb25nLmFvdikKCmtvbmcudHVrZXkKYGBgCgpTYW1lIGluZm9ybWF0aW9uIHZpc3VhbGx5OgoKYGBge3J9CnBsb3Qoa29uZy50dWtleSkKYGBgCgo=

gRPC benchmark result comparison

Summary

Visualization

Implementation