ETL pipelines extract, transform, and load data before it is used for analysis, reporting, or machine learning.

As data volume grows, runtime can become an engineering constraint:

• dashboards may refresh late,
• batch jobs may miss deadlines,
• compute costs may increase,
• downstream work may be delayed.

This presentation treats ETL performance as a statistical measurement problem.

The benchmark focuses on in-memory transformation work. It is an ETL-style teaching example rather than a full production benchmark of extraction, networking, storage, and loading.

For the main regression analysis, I focus on the Heavy pipeline.

Working expectations

• Larger input tables should take longer to process.
• Runtime may be approximately linear over the tested range.
• More complex workloads should generally require more processing time at the same input size.

The source table is generated entirely in R with a fixed seed, so no private or external data are required.

Five input sizes are tested:

\[ 50{,}000,\;100{,}000,\;200{,}000,\;350{,}000,\;500{,}000 \]

Each workload is measured 5 times, and each timing trial averages 20 executions to reduce timer noise.

Three workloads are compared:

1. Light: filter + derived column
2. Medium: transformation + grouping + aggregation
3. Heavy: several transformations + multi-column grouping + multiple summaries + sorting

The repeated measurements show both the average runtime and the amount of run-to-run variation.

For the Heavy pipeline, let \(Y_i\) be elapsed runtime and \(X_i\) be the number of input rows.

\[ Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i \]

The fitted model is:

\[ \widehat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i \]

Least squares chooses the coefficients that minimize:

\[ \sum_{i=1}^{n}(Y_i-\widehat{Y}_i)^2 \]

For this run, an additional 100,000 rows is associated with an estimated runtime change of 0.0115 seconds.

The 95% confidence interval for that slope is approximately 0.0105 to 0.0125 seconds per 100,000 rows.

The model has \(R^2 = 0.96\), meaning row count explains about 96% of the observed runtime variation in these Heavy-pipeline trials.

A roughly straight pattern supports using a linear model as an approximation within the tested range.

Runtime is one view of performance. Throughput measures how much data is processed per unit time:

\[ Throughput = \frac{\text{Rows Processed}}{\text{Elapsed Time}} \]

For \(N\) rows processed in \(t\) seconds:

\[ T = \frac{N}{t} \quad \text{rows per second} \]

Throughput adds context because a larger job can take longer while still processing rows efficiently.

Hover over the points to inspect the measured runtime, variability, and throughput for each workload and data size.

The estimated slope describes how runtime changes with additional input rows for this benchmark and this computer.

A small p-value for the slope provides evidence that runtime and data volume are associated within the benchmark.

The confidence interval gives a range of plausible values for the runtime increase per 100,000 additional rows.

The \(R^2\) value describes how much of the trial-to-trial runtime variation is explained by row count alone.

This is a controlled teaching benchmark, not a universal ETL performance test.

• The source data are synthetic.
• Exact timings depend on hardware and current system load.
• Everything runs in memory on one machine.
• Real ETL systems may include disk I/O, databases, networks, cloud storage, or distributed computation.
• Spark-style systems introduce partitioning, scheduling, serialization, and shuffle costs not represented here.
• Scaling that appears linear from 50,000 to 500,000 rows may not remain linear at much larger sizes.

The results therefore illustrate how statistics can be used to analyze engineering performance, not how every ETL system will behave.

• ETL performance can be studied using repeated measurements and statistical models.
• Data volume is tested as a predictor of Heavy-pipeline runtime.
• Regression gives an interpretable estimate and confidence interval for the runtime increase associated with additional rows.
• Comparing multiple workloads shows that processing complexity matters in addition to data size.
• The benchmark is self-contained and repeatable, while the exact timing results remain machine-dependent.

The same statistical workflow could later be applied to database queries, Spark jobs, or cloud ETL pipelines.