This is a follow-up to the previous analysis comparing the performance of a couple of feature branches. We were already confident that the software has the same performance but we were concerned about the level of jitter and variation in the benchmark environment. This analysis is with a new dataset from a tweaked test env and taking the order of tests into account e.g. to see if they slow down or speed up over time.

TLDR; Software performance and test env jitter both looking stable now!

Scatter plot

First a simple scatter-plot to see what the data looks like:

library(readr)
library(dplyr)
library(ggplot2)
data <- read.csv("https://gist.githubusercontent.com/lukego/0e6f5c4da95edca75cef1d20f4cd99a7/raw/ea5711f4d74ace87c3c4c0af022f05861d6d003f/results.csv")
ggplot(data, aes(y=requests_per_second, x=id, color=branch)) +
  expand_limits(y=0) +
  geom_point() +
  labs(y = "requests/second", x = "test sequence number")

Overall very consistent. Just two of the first data points stick out as exceptional (“outliers.”)

Filtering

Let’s filter the data to exclude those two outliers, since they are probably due to a minor issue with the test env rather than the software versions.

data2 <- filter(data, requests_per_second < 7000)

Trend

Let’s zoom in closer on the data and see if there is a trend in the results over time:

ggplot(data2, aes(y=requests_per_second, x=id, color=branch)) +
  geom_point(alpha=0.25) +
  geom_smooth(method="loess") +
  labs(y = "requests/second", x = "test sequence number")

Unclear. There does seem to be some kind of pattern here but nothing so simple as “gets slower over time.” Again, probably some factor that is not covered by the tests i.e. not caused by which gRPC branch we choose. This variation is only around +/- 2% which is actually really low and nothing to be concerned about.

Boxplot

Here is the distribution of results as a boxplot:

ggplot(data2, aes(y=requests_per_second, x=branch, color=branch)) +
  expand_limits(y=0) +
  geom_boxplot() +
  labs(y = "requests/second", x = "test sequence number") +
  theme(legend.position = "none")

Looks like a really great and well debugged benchmarking environment !

Difference

Check if one branch is faster than the other using “Tukey’s Honestly Significant Difference test”:

plot(TukeyHSD(aov(lm(requests_per_second ~ branch, data2))))

Nope! Even with this large and clean dataset the 95% confidence still includes a difference of 0 between branches, meaning that no significant difference was found, and on other other extreme only allows for grps-auto to be up to 30 req/s faster than grpc-pass.

So we can be even more confident that the branches have the same performance :+1:

LS0tCnRpdGxlOiAiRmVhdHVyZSBicmFuY2ggY29tcGFyaXNvbiIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKVGhpcyBpcyBhIGZvbGxvdy11cCB0byB0aGUgW3ByZXZpb3VzIGFuYWx5c2lzXShodHRwczovL3JwdWJzLmNvbS9sdWtlZ28vNDk4Mzc4KSBjb21wYXJpbmcgdGhlIHBlcmZvcm1hbmNlIG9mIGEgY291cGxlIG9mIGZlYXR1cmUgYnJhbmNoZXMuIFdlIHdlcmUgYWxyZWFkeSBjb25maWRlbnQgdGhhdCB0aGUgc29mdHdhcmUgaGFzIHRoZSBzYW1lIHBlcmZvcm1hbmNlIGJ1dCB3ZSB3ZXJlIGNvbmNlcm5lZCBhYm91dCB0aGUgbGV2ZWwgb2Ygaml0dGVyIGFuZCB2YXJpYXRpb24gaW4gdGhlIGJlbmNobWFyayBlbnZpcm9ubWVudC4gVGhpcyBhbmFseXNpcyBpcyB3aXRoIGEgbmV3IGRhdGFzZXQgZnJvbSBhIHR3ZWFrZWQgdGVzdCBlbnYgYW5kIHRha2luZyB0aGUgb3JkZXIgb2YgdGVzdHMgaW50byBhY2NvdW50IGUuZy4gdG8gc2VlIGlmIHRoZXkgc2xvdyBkb3duIG9yIHNwZWVkIHVwIG92ZXIgdGltZS4KClRMRFI7IFNvZnR3YXJlIHBlcmZvcm1hbmNlICphbmQqIHRlc3QgZW52IGppdHRlciBib3RoIGxvb2tpbmcgc3RhYmxlIG5vdyEKCiMgU2NhdHRlciBwbG90CgpGaXJzdCBhIHNpbXBsZSBzY2F0dGVyLXBsb3QgdG8gc2VlIHdoYXQgdGhlIGRhdGEgbG9va3MgbGlrZToKCmBgYHtyfQpsaWJyYXJ5KHJlYWRyKQpsaWJyYXJ5KGRwbHlyKQpsaWJyYXJ5KGdncGxvdDIpCmRhdGEgPC0gcmVhZC5jc3YoImh0dHBzOi8vZ2lzdC5naXRodWJ1c2VyY29udGVudC5jb20vbHVrZWdvLzBlNmY1YzRkYTk1ZWRjYTc1Y2VmMWQyMGY0Y2Q5OWE3L3Jhdy9lYTU3MTFmNGQ3NGFjZTg3YzNjNGMwYWYwMjJmMDU4NjFkNmQwMDNmL3Jlc3VsdHMuY3N2IikKCmdncGxvdChkYXRhLCBhZXMoeT1yZXF1ZXN0c19wZXJfc2Vjb25kLCB4PWlkLCBjb2xvcj1icmFuY2gpKSArCiAgZXhwYW5kX2xpbWl0cyh5PTApICsKICBnZW9tX3BvaW50KCkgKwogIGxhYnMoeSA9ICJyZXF1ZXN0cy9zZWNvbmQiLCB4ID0gInRlc3Qgc2VxdWVuY2UgbnVtYmVyIikKYGBgCgpPdmVyYWxsIHZlcnkgY29uc2lzdGVudC4gSnVzdCB0d28gb2YgdGhlIGZpcnN0IGRhdGEgcG9pbnRzIHN0aWNrIG91dCBhcyBleGNlcHRpb25hbCAoIm91dGxpZXJzLiIpCgojIEZpbHRlcmluZwoKTGV0J3MgZmlsdGVyIHRoZSBkYXRhIHRvIGV4Y2x1ZGUgdGhvc2UgdHdvIG91dGxpZXJzLCBzaW5jZSB0aGV5IGFyZSBwcm9iYWJseSBkdWUgdG8gYSBtaW5vciBpc3N1ZSB3aXRoIHRoZSB0ZXN0IGVudiByYXRoZXIgdGhhbiB0aGUgc29mdHdhcmUgdmVyc2lvbnMuCgpgYGB7cn0KZGF0YTIgPC0gZmlsdGVyKGRhdGEsIHJlcXVlc3RzX3Blcl9zZWNvbmQgPCA3MDAwKQpgYGAKCiMgVHJlbmQKCkxldCdzIHpvb20gaW4gY2xvc2VyIG9uIHRoZSBkYXRhIGFuZCBzZWUgaWYgdGhlcmUgaXMgYSB0cmVuZCBpbiB0aGUgcmVzdWx0cyBvdmVyIHRpbWU6CgpgYGB7cn0KZ2dwbG90KGRhdGEyLCBhZXMoeT1yZXF1ZXN0c19wZXJfc2Vjb25kLCB4PWlkLCBjb2xvcj1icmFuY2gpKSArCiAgZ2VvbV9wb2ludChhbHBoYT0wLjI1KSArCiAgZ2VvbV9zbW9vdGgobWV0aG9kPSJsb2VzcyIpICsKICBsYWJzKHkgPSAicmVxdWVzdHMvc2Vjb25kIiwgeCA9ICJ0ZXN0IHNlcXVlbmNlIG51bWJlciIpCmBgYAoKVW5jbGVhci4gVGhlcmUgZG9lcyBzZWVtIHRvIGJlIHNvbWUga2luZCBvZiBwYXR0ZXJuIGhlcmUgYnV0IG5vdGhpbmcgc28gc2ltcGxlIGFzICJnZXRzIHNsb3dlciBvdmVyIHRpbWUuIiBBZ2FpbiwgcHJvYmFibHkgc29tZSBmYWN0b3IgdGhhdCBpcyBub3QgY292ZXJlZCBieSB0aGUgdGVzdHMgaS5lLiBub3QgY2F1c2VkIGJ5IHdoaWNoIGdSUEMgYnJhbmNoIHdlIGNob29zZS4gVGhpcyB2YXJpYXRpb24gaXMgb25seSBhcm91bmQgKy8tIDIlIHdoaWNoIGlzIGFjdHVhbGx5IHJlYWxseSBsb3cgYW5kIG5vdGhpbmcgdG8gYmUgY29uY2VybmVkIGFib3V0LgoKIyBCb3hwbG90CgpIZXJlIGlzIHRoZSBkaXN0cmlidXRpb24gb2YgcmVzdWx0cyBhcyBhIGJveHBsb3Q6CgpgYGB7cn0KZ2dwbG90KGRhdGEyLCBhZXMoeT1yZXF1ZXN0c19wZXJfc2Vjb25kLCB4PWJyYW5jaCwgY29sb3I9YnJhbmNoKSkgKwogIGV4cGFuZF9saW1pdHMoeT0wKSArCiAgZ2VvbV9ib3hwbG90KCkgKwogIGxhYnMoeSA9ICJyZXF1ZXN0cy9zZWNvbmQiLCB4ID0gInRlc3Qgc2VxdWVuY2UgbnVtYmVyIikgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikKYGBgCgpMb29rcyBsaWtlIGEgcmVhbGx5IGdyZWF0IGFuZCB3ZWxsIGRlYnVnZ2VkIGJlbmNobWFya2luZyBlbnZpcm9ubWVudCAhCgojIERpZmZlcmVuY2UKCkNoZWNrIGlmIG9uZSBicmFuY2ggaXMgZmFzdGVyIHRoYW4gdGhlIG90aGVyIHVzaW5nICJUdWtleSdzIEhvbmVzdGx5IFNpZ25pZmljYW50IERpZmZlcmVuY2UgdGVzdCI6CgpgYGB7cn0KcGxvdChUdWtleUhTRChhb3YobG0ocmVxdWVzdHNfcGVyX3NlY29uZCB+IGJyYW5jaCwgZGF0YTIpKSkpCmBgYAoKTm9wZSEgRXZlbiB3aXRoIHRoaXMgbGFyZ2UgYW5kIGNsZWFuIGRhdGFzZXQgdGhlIDk1JSBjb25maWRlbmNlIHN0aWxsIGluY2x1ZGVzIGEgZGlmZmVyZW5jZSBvZiAwIGJldHdlZW4gYnJhbmNoZXMsIG1lYW5pbmcgdGhhdCBubyBzaWduaWZpY2FudCBkaWZmZXJlbmNlIHdhcyBmb3VuZCwgYW5kIG9uIG90aGVyIG90aGVyIGV4dHJlbWUgb25seSBhbGxvd3MgZm9yIGdycHMtYXV0byB0byBiZSB1cCB0byAzMCByZXEvcyBmYXN0ZXIgdGhhbiBncnBjLXBhc3MuCgpTbyB3ZSBjYW4gYmUgKmV2ZW4gbW9yZSogY29uZmlkZW50IHRoYXQgdGhlIGJyYW5jaGVzIGhhdmUgdGhlIHNhbWUgcGVyZm9ybWFuY2UgOisxOg==