Analysis for Deluge Benchmarks - Static Network Dissemination Experiment

This document contains the analysis for the Deluge benchmarks.

library(tidyverse)

devtools::load_all()

1 Parse/Load Data

This is data that’s been pre-parsed from an experiment log source.

deluge <- read_all_experiments('./data/deluge')

Reading experiment e1
Reading experiment e10
Reading experiment e11
Reading experiment e12
Reading experiment e13
Reading experiment e14
Reading experiment e15
Reading experiment e16
Reading experiment e17
Reading experiment e18
Reading experiment e19
Reading experiment e2
Reading experiment e20
Reading experiment e21
Reading experiment e22
Reading experiment e23
Reading experiment e24
Reading experiment e25
Reading experiment e26
Reading experiment e3
Reading experiment e4
Reading experiment e5
Reading experiment e6
Reading experiment e7
Reading experiment e8
Reading experiment e9

Computes the benchmark statistics from raw download logs.

benchmarks <- lapply(deluge, function(experiment) {
  print(glue::glue('Process {experiment$experiment_id}'))
  download_time_stats <- tryCatch({
    meta <- experiment$meta
    completion <- experiment |>
      download_times() |>
      completion_time_stats()
    
    if (is.null(completion)) {
      NULL
    } else {
      completion |> mutate(
        network_size = meta$nodes$network_size,
        seeders = meta$seeders,
        leechers = network_size - meta$seeders,
        file_size = meta$file_size
      )
    }
  }, error = function(e) { print(e); NULL })
}) |> 
  drop_nulls() |>
  bind_rows() |>
  arrange(file_size, network_size, seeders, leechers) |>
  mutate(
    file_size = as.character(rlang::parse_bytes(as.character(file_size))),
    seeder_ratio = seeders/network_size
  ) |>
  relocate(file_size, network_size, seeders, leechers)

Process e1
Process e10
Process e11
Process e12
Process e13
Process e14
Process e15
Process e16
Process e17
Process e18
Process e19
Process e2
Process e20
Process e21
Process e22
Process e23
Process e24
Process e25
Process e26
Process e3
Process e4
Process e5
Process e6
Process e7
Process e8
Process e9

2 Results

First, we present the raw data in tabular format:

benchmarks

We then plot the median by network size, and facet it by seeder ratio and file size to see if looks sane:

ggplot(benchmarks) +
  geom_line(aes(x = network_size, y = median)) +
  geom_point(aes(x = network_size, y = median)) +
  ylab('median download time (seconds)') +
  xlab('network size') +
  theme_minimal(base_size=15) +
  facet_grid(
    file_size ~ seeder_ratio, 
    scales = 'free_y',
    labeller = labeller(
      file_size = as_labeller(function(x) x),
      seeder_ratio = as_labeller(function(x) {
        paste0("seeder ratio: ", scales::percent(as.numeric(x)))
      }))
  ) +
  ylim(c(0,NA))

The data looks largely sane: a larger seeder ratio makes performance somewhat better; though not nearly as consistently as one would hope, at least in this data, and there is a linear performance degradation trend as the network grows larger. Also, the \(100\text{MB}\) file seems to generate much better-behaved data than the \(1\text{GB}\) case with those trends; i.e., larger seeder ratio improving performance, and network size linearly degrading it, being more pronounced.