NFV test matrix

Performance comparison between Snabb versions. Compares scores with various benchmarks, workloads, configurations, and software dependency versions. Results are mostly reported as probability density graphs to show the distribution of benchmark scores (measured in Gbps for iperf and Mpps for l2fwd).

The overall purpose of this report is to highlight the relative strengths and weaknesses between Snabb software versions. This is particularly intended for evaluating how experimental changes affect performance and reliability.

This report starts with a very broad summary and then drills down on iperf and l2fwd results separately.

Significance test

This first graph is special: its purpose is to show whether there are “statistically significant” differences between the Snabb versions. It is plotted in an especially suitable way: a CDF (see Reading CDF Graphs) with a 95% confidence band. (Explanation follows below.)

Significant differences appear as gaps (blank space) between confidence bands. For each gap the band above seems to be significantly faster than the band below. If bands criss-cross this suggests that one version is faster on some benchmarks and slower on others.

If two bands have no gaps (i.e. they overlap at every point on the X-axis) then no significant differences between those bands was found. In this case any apparent differences between branches in the other plots below should be taken with a grain of salt.

Notes:

Higher is better.
The breadth of the confidence bands depends on the number of benchmarks. The more data you have, the narrower your bands, the easier it is to find small gaps. If you think there is a difference but the bands are too broad to be sure then consider re-running the benchmark with more iterations.
The confidence bands are calculated using a method based on the Kolmogorov-Smirnov (KS) test.

Overall summary

Split by benchmark

iperf

Configuration

Guest kernel

QEMU

Success and failure

Success rate:

Sucesses:

##   config                           qemu        kernel   
##  Mode:logical   2.2.1                :28   3.18.29:183  
##  NA's:183       2.4.1                :28                
##                 2.6.2                :28                
##                 2.1.3-with-snabbpatch:26                
##                 2.1.3                :25                
##                 2.5.1                :25                
##                 (Other)              :23

Failures:

##   config                           qemu        kernel  
##  Mode:logical   2.7.1                :30   3.18.29:87  
##  NA's:87        2.8.0                :30               
##                 2.3.1                : 7               
##                 2.1.3                : 5               
##                 2.5.1                : 5               
##                 2.1.3-with-snabbpatch: 4               
##                 (Other)              : 6

l2fwd

Packet size