Abstract — In this case study, I walk through the full ML pipeline I built for predicting Formula 1 pit stops on a per-lap basis, using synthetic race telemetry from the Kaggle Playground Series S6E5. It takes raw telemetry — tire degradation curves, lap-time deltas, positional dynamics, and compound-specific degradation physics — and feeds it into a 10-method meta-ensemble. I structured the pipeline around a Coarse-to-Fine hyperparameter optimization approach across six learners (XGBoost, XGBoost-DART, LightGBM, LightGBM-GOSS, CatBoost, Ranger), with SHAP-driven feature selection, A/B tests at each decision point, and out-of-fold target encoding to prevent leakage. The final Hill-Climb ensemble achieves an OOF AUC of 0.9325 and a holdout validation AUC of 0.9786.

Binary Classification Gradient Boosted Trees Meta-Ensembling SHAP Interpretability Coarse-to-Fine Tuning Platt Calibration Feature Engineering AUC-ROC Optimization

Note on Architecture & Portability: While many production ML workflows run on Python, I built this pipeline in R to take full advantage of data.table and mlr3 for high-performance tabular ETL and experiment tracking. The core methodology demonstrated here—Bayesian optimization, SHAP interpretability, and leak-free meta-ensembling—is entirely framework-agnostic and translates directly to Python ecosystems (e.g., scikit-learn, Optuna, SHAP).

1 Introduction

1.1 The Problem

In motorsport strategy, pit stop timing depends on tire wear, track position, weather, and what the competition is doing. Getting it wrong by even one lap can cost 20+ seconds — the difference between a podium and going home empty. While this project doesn’t model the pit-stop timing penalty directly, that real-world urgency is why I focused so heavily on degradation-aware features (TyreLife_sq, tyre_deg_acceleration, tyre_window_progress) that capture the narrow window where a stop becomes strategically optimal.

To tackle this, I built a prediction system that takes per-lap telemetry — tire compound, tire age, lap times, degradation curves, positional dynamics, and race progress — and outputs a calibrated probability of whether a driver will pit on the next lap.

Transferability & Empirical Approach: Treating machine learning as an empirical science—where physics-based domain constraints guide the feature space rather than black-box data feeding—is essential for reliable systems. The underlying methodology here (multi-model ensembling, SHAP-driven interpretability, and strict holdout validation) transfers directly to high-stakes domains where algorithmic robustness is critical.

1.2 Key Contributions

Coarse-to-Fine Tuning — Three rounds of hyperparameter search: hardware-aware random sweep (R1) → widened bounds in SHAP-reduced feature space (R1.5) → ±20% Bayesian MBO with stagnation early-stop (R2)
Physics-Informed Feature Engineering — 20+ features translating tire degradation mechanics into math (rolling pace deltas, degradation acceleration, compound–stint interactions)
10-Method Meta-Ensemble — Simple Average, Weighted Average, Hill-Climb, Sigmoid Rank-Average, Diversity-Pruned, Logistic Stacking, Fold-Aware XGBoost Stacker, BMA, Isotonic Calibration, and Bagging
A/B Testing at Every Decision Point — Winsorization, interaction injection, and feature selection thresholds are tested, not assumed
Leakage Prevention — Out-of-fold target encoding, train-only winsorization bounds, and grouped K-Fold splits by Race

1.3 Roadmap

Here’s what the rest of this case study covers:

Section	Title	Content
2	Computational Environment	Software stack, hardware configuration, reproducibility setup
3	Data Ingestion & Initial Profiling	Loading four CSV sources, dimension checks, NA audit
4	Exploratory Data Analysis	Target distribution, compound analysis, descriptive statistics
5	Diagnostic Analysis	Missing values, outliers, skewness, correlation matrix
6	Feature Density Analysis	KS-statistic overlays for Pit vs No-Pit distributions
7	Linearity & Model Class Justification	GLM vs XGBoost controlled experiment
8	Inferential Analysis	Cohen’s d effect sizes, Chi-squared associations
9	Feature Engineering Pipeline	Physics-informed feature categories, lag features, MICE imputation
10	Winsorization A/B Test	Controlled before/after clipping experiment
11	Coarse-to-Fine Hyperparameter Optimization	Three-round tuning architecture with verified results
12	SHAP Analysis & Feature Selection	Consensus SHAP ranking, interaction A/B test
13	Out-of-Fold Predictions & Variance Tracking	5-fold OOF performance per model
14	10-Method Meta-Ensemble	Ensemble architecture, Platt scaling, AUC gating
15	Generalization & Final Validation	Holdout validation, Kaggle leaderboard benchmarks
16	Submission Pipeline	TTA pipeline, submission preview table
17	Final Thoughts	What worked, what I’d change, key lessons

2 Computational Environment

Thread Allocation: data.table operations are capped at 8 threads via setDTthreads(8) to avoid L1/L2 cache thrashing on AMD Ryzen architecture, while tree learners consume 16 threads independently. ETL is memory-bound so more threads hurt; training is compute-bound so it benefits from all 16.

Computational Environment & Configuration
Component	Details
Language	R 4.6.1
ML Framework	mlr3 ecosystem (mlr3verse, mlr3tuning, mlr3mbo, mlr3extralearners)
Bayesian Optimization	rBayesianOptimization (GP surrogate, Round 2) + mlr3mbo (alternative tuning script)
Tree Learners	XGBoost, XGBoost-DART, LightGBM, LightGBM-GOSS, CatBoost, Ranger
Feature Importance	TreeSHAP (shapviz) + Impurity Importance proxy (Ranger)
Imputation	MICE (Predictive Mean Matching) — preserves conditional distribution structure
Parallelism	data.table: 8 threads \| Tree learners: 16 threads \| future::plan(multisession)
Evaluation Metric	AUC-ROC (probability ranking, not calibration)
CV Strategy	5-Fold Stratified Group K-Fold (grouped by Race)
Seed (Reproducibility)	2026 (fixed across all stages)
GPU Acceleration	CUDA (XGBoost tree_method=‘hist’, device=‘cuda’) \| CatBoost task_type=‘GPU’

3 Data Ingestion & Initial Profiling

The analysis relies on four primary data files:

Dataset Summary
Dataset	Rows	Columns	Description
train.csv	439,140	16	Competition training data with per-lap synthetic telemetry
test.csv	188,165	15	Unlabeled test set for submission
f1_strategy_dataset_v4.csv	101,371	16	External F1 strategy data (Race × Compound aggregate statistics)
sample_submission.csv	188,165	2	Submission template

Column Overlap Check: 15 columns are shared between train and test. The single train-only column is PitNextLap — the binary target. No structural mismatches between train and test.

4 Exploratory Data Analysis

4.1 Target Variable Distribution

Class Imbalance: The target is 80.1% No-Pit / 19.9% Pit — roughly 4:1. This shaped several design choices: (1) AUC-ROC as the primary metric (doesn’t depend on threshold selection), (2) LightGBM’s is_unbalance=TRUE flag, (3) XGBoost’s scale_pos_weight = 4.026, (4) CatBoost’s auto_class_weights = "Balanced", and (5) Platt Scaling for post-hoc probability calibration.

4.2 Descriptive Statistics

Descriptive Statistics of Key Telemetry Features
Feature	Mean	Median	SD	Min	Max	Skewness	Kurtosis
TyreLife	14.158	12.000	9.801	1.000	77.000	1.032	4.437
LapTime (s)	90.949	90.521	19.773	67.694	2507.607	80.334	9682.240
RaceProgress	0.338	0.269	0.253	0.013	1.000	0.700	2.485
Position	9.630	10.000	5.279	1.000	20.000	0.070	1.901
LapNumber	23.106	19.000	16.958	1.000	78.000	0.646	2.433
Stint	1.789	2.000	0.950	1.000	8.000	1.185	4.270
Cumulative_Degradation	-25.722	-20.994	54.767	-274.564	2412.026	3.191	159.351
LapTime_Delta	-3.770	-0.295	43.946	-2403.895	2423.932	-40.855	2677.589

Key Observations:

LapTime (s) shows extreme skewness (80.3) and kurtosis (9682) — safety car laps, red flag restarts, and pit-in laps create massive right-tail outliers reaching 2507s
TyreLife has moderate right-skew (1.03) — most laps happen on fresh tires (stint 1), with a long tail for extended stints
Position is roughly symmetric (skew 0.07) — a balanced grid representation across the dataset

4.3 Pit Rate by Compound

Key Observations:

Softer compounds (SOFT, MEDIUM) show higher pit rates — they degrade faster, forcing earlier stops
Harder compounds (HARD, INTERMEDIATE) exhibit lower pit rates — longer stint viability reduces strategic urgency
This compound-dependent degradation pattern motivates the Compound_TyreLife interaction feature in Section 9, which captures the non-linear relationship between compound softness and tire age at the point of pit entry

4.4 Stint Distribution

Key Observations:

Stint 1 dominates the dataset with the highest row count — every driver begins the race on their first stint, so this is the fully populated baseline
Subsequent stints (2, 3, 4+) decay sharply — not all drivers pit multiple times, and late-race stints are rare
This exponential decay justifies the Stint_TyreLife_ratio feature: normalizing tire age relative to stint number captures whether a driver is running an atypically long stint (high ratio = overdue for a stop)

5 Diagnostic Analysis

5.1 Missing Value Audit

Clean Dataset Confirmed: The raw training data contains 0 columns with missing values. Missing values arise only after feature engineering (lag features on Lap 1), where MICE (Predictive Mean Matching) is applied.

5.2 Outlier Diagnostics

Key Observations:

LapTime (s) shows the widest interquartile range for the Pit class — laps immediately preceding a pit stop tend to be slower (degraded tires), but also include outlier safety-car laps that inflate the upper tail
LapTime_Delta has symmetric outliers in both classes, but the Pit class shows a slight positive shift — drivers on worn tires are more likely to lose time lap-over-lap, producing positive deltas
Cumulative_Degradation separates cleanly: the Pit class has a visibly higher median, confirming that accumulated performance loss is a strong pit-stop indicator

5.3 Correlation Matrix

Multi-Collinearity Note: Several correlated feature clusters are visible in the matrix:

TyreLife ↔︎ LapNumber (strong positive, ~0.85): Both increase monotonically within a stint
LapNumber ↔︎ RaceProgress (strong positive, ~0.90): LapNumber is the primary driver of RaceProgress (which is LapNumber / TotalLaps)
LapNumber ↔︎ Stint (moderate positive, ~0.45): Later laps tend to occur in higher stints
TyreLife ↔︎ RaceProgress (moderate positive, ~0.55): Correlated via LapNumber, though TyreLife resets at each pit stop, breaking the monotonic relationship

This doesn’t warrant removal: tree-based models split on thresholds, not coefficient estimates, so multi-collinearity is a non-issue. The correlated features encode complementary positional information — TyreLife captures within-stint age while LapNumber captures within-race position. This further backs up the model-class choice in Section 7.

6 Feature Density Analysis: Pit vs No-Pit

KS Statistic Interpretation: The Kolmogorov-Smirnov statistic measures the max distance between Pit and No-Pit CDFs. Features with KS > 0.20 (TyreLife, Stint, RaceProgress) have strong discriminative signal — these are going to be high-value predictors.

Key Observations:

TyreLife shows the strongest class separation — the Pit distribution is shifted rightward, indicating drivers are far more likely to pit when tires are old
Stint exhibits a clear bimodal pattern in the Pit class — most pit stops occur at the end of Stint 1 (first strategic stop), with a secondary peak at higher stint counts
RaceProgress reveals that pit stops concentrate in the mid-race window (0.3–0.7 progress) — early and late-race laps have low pit probability
Position and LapTime (s) show weaker separation (lower KS), suggesting these are secondary signals that gain value only in combination with degradation features

7 Linearity & Model Class Justification

Before jumping into tree models, it’s worth checking: can a simple linear model handle this, or do we actually need non-linear learners?

To answer that, I ran a quick head-to-head using only the raw features:

Model Class Viability Test (Raw Features Only)
Model	AUC	Type
Logistic Regression (GLM)	0.7161	Linear
XGBoost (Tree Ensemble)	0.8846	Non-Linear

Verdict: Complex Non-Linearities Detected

Non-Linearity Gain: +0.1685 AUC — That exceeds the 0.020 threshold by 8×
Bottom line: Tree ensembles are the way to go here. A GLM barely learns anything useful. And for tabular data like this, GBDTs consistently beat deep learning approaches anyway.

8 Inferential Analysis

8.1 Cohen’s d — Feature Discrimination Power

Inferential Analysis — Statistical Significance & Effect Sizes
Feature	Mean_NoPit	Mean_Pit	CohenD	EffectSize
TyreLife	12.822	19.537	-0.712	Medium
RaceProgress	0.314	0.432	-0.473	Small
LapTime (s)	91.285	89.596	0.085	Negligible
Position	9.574	9.856	-0.053	Negligible
LapNumber	20.849	32.192	-0.694	Medium
Stint	1.695	2.167	-0.506	Medium
Cumulative_Degradation	-21.152	-44.116	0.425	Small

8.2 Chi-Squared — Categorical Associations

Chi-Squared Test — Categorical Feature Associations
Feature	Chi2	p_value	Cramers_V	Association
Compound	30866.32	0.0000000	0.2651	Weak
Race	15545.76	0.0004998	0.1882	Weak

9 Feature Engineering Pipeline

I engineered 20+ new features across five groups:

Feature Engineering Categories
Category	Features	Physical Interpretation
Tire Degradation Shape	TyreLife_sq, TyreLife_log, TyreLife_per_stint, LapTime_per_TyreLife	Captures non-linear degradation curves — the ‘cliff’ where performance drops exponentially
Race Strategy Signals	RaceProgress_sq, LapsRemaining_approx, Stint_TyreLife_ratio, Position_x_Progress, just_pitted, tyre_window_progress	Models the strategic decision space: when in the race, how much rubber remains
Compound Interactions	Compound_ord, Compound_TyreLife	Ordinal encoding preserving compound hardness hierarchy × tire age interaction
External Prior Knowledge	ext_pit_rate, ext_median_pit_lap, ext_mean_TyreLife	Historical Race × Compound aggregate statistics from the external F1 strategy dataset
Lag Features (Sequential)	laptime_lag1/2/3, laptime_delta, laptime_rolling3, tyre_age_rate, deg_rolling3, tyre_deg_acceleration	Lap-over-lap deltas within each Race × Driver group — detects real-time degradation trends

Missing Value Generation: Lag features introduce ~3–7% NAs for each driver’s first laps. These get imputed using MICE (Predictive Mean Matching) instead of simple median fill — PMM keeps the underlying distribution intact, which matters for the rolling calculations downstream.

10 Winsorization A/B Test

Rather than blindly clipping extreme values, I ran a controlled A/B test to make sure it actually helped:

Winsorization A/B Test Results (XGBoost Proxy)
Condition	3-Fold CV AUC	Verdict
Pre-Winsorization (Raw Tails)	0.94508	—
Post-Winsorization (P1/P99 Clipped)	0.94524	✅ Marginal Improvement

Decision: Winsorization helps slightly (+0.00016 AUC), so the clipped bounds stay. The pipeline has an auto-revert built in — if clipping had hurt, it would’ve switched back to raw tails automatically.

11 Coarse-to-Fine Hyperparameter Optimization

I used a three-round Coarse-to-Fine tuning strategy instead of the usual grid/random search:

11.1 Round 1: Coarse Hardware-Aware Sweep

Objective: Establish stable baselines across all 6 learners using random search with hardware-specific execution profiles.

Round 1 — Coarse Sweep Results
Model	Hardware Profile	Budget	Holdout AUC
XGBoost	Sequential GPU (16T)	200 evals	0.94878
XGBoost-DART	Sequential GPU (16T)	100 evals	0.94697
LightGBM	Parallel CPU (4W×4T)	200 evals	0.94740
LightGBM-GOSS	Parallel CPU (4W×4T)	200 evals	0.94827
CatBoost	Sequential GPU (16T)	240 evals	0.92668
Ranger	Parallel CPU (12W×1T)	96 evals	0.94367

11.2 Round 1.5: SHAP-Reduced Feature Space

After SHAP-based feature selection reduces the space from ~180 features to ~81 pure-signal features, the search bounds are widened (not narrowed) to let the optimizer explore new territory in the cleaner feature space:

Why Widening Bounds: Pruning ~100 noisy features fundamentally reshapes the loss landscape. With feature noise removed, hyperparameter combinations that previously caused overfitting (such as deeper tree structures or lower learning rates) become viable, requiring wider boundaries to discover superior global optima.

Round 1.5 — SHAP-Reduced Results
Model	R1.5 Holdout AUC	Δ from R1
XGBoost	0.94946	+0.00068
XGBoost-DART	0.94801	+0.00104
LightGBM	0.94711	-0.00029
LightGBM-GOSS	0.94713	-0.00114
CatBoost	0.93435	+0.00767
Ranger	0.94425	+0.00058

11.3 Round 2: ±20% Refined Bayesian MBO

The final round uses Bayesian Optimization with Gaussian Process surrogates (via rBayesianOptimization), searching a ±20% window around Round 1.5 optima. Each model has its own early-stop trigger if progress stalls:

Round 2 — Final Tuned Holdout AUC & Iteration Counts
	Model	R2 Holdout AUC	Δ from R1.5	Early-Stopped Rounds
xgboost	xgboost	0.950204	+0.000743	2698
xgboost_dart	xgboost_dart	0.949206	+0.001193	210
lightgbm	lightgbm	0.947604	+0.000494	3952
lightgbm_goss	lightgbm_goss	0.949590	+0.002461	2682
catboost	catboost	0.938275	+0.003927	639
ranger	ranger	0.944564	+0.000310	N/A (RF)

11.4 Post-Tuning Variance Audit

To make sure Round 2 results aren’t flukes, I ran a 3-fold CV sanity check on each model:

Post-Tuning Variance Audit (3-Fold CV)
Model	Holdout AUC	CV Mean AUC	CV StdDev	Verdict
xgboost	0.95020	0.95035	0.00037	✅ DIAGNOSTIC: Highly Stable Holdout Split (SD <= 0.0015).
xgboost_dart	0.94921	0.94898	0.00053	✅ DIAGNOSTIC: Highly Stable Holdout Split (SD <= 0.0015).
lightgbm	0.94760	0.94716	0.00045	✅ DIAGNOSTIC: Highly Stable Holdout Split (SD <= 0.0015).
lightgbm_goss	0.94959	0.94771	0.00019	✅ DIAGNOSTIC: Highly Stable Holdout Split (SD <= 0.0015).
catboost	0.93828	0.91880	0.00879	⚠️ DIAGNOSTIC: Elevated Holdout Variance (SD > 0.0025). Relying strictly on Group K-Fold OOF (Sec 8)

Key Findings:

XGBoost leads with AUC 0.9504, and the lowest variance (σ = 0.00037) — the most stable performer
LightGBM-GOSS achieves remarkable stability (σ = 0.00019) — Gradient-based One-Side Sampling acts as implicit regularization
CatBoost shows elevated variance (σ = 0.00879) — its categorical handling seems to be overfitting to specific Race groups. It gets dropped by the AUC gate later in Section 14.

12 Interpretability & SHAP-Driven Feature Selection

12.1 Consensus Feature Ranking

To audit whether the ensemble learned genuine physical tire dynamics rather than exploiting dataset-specific spurious artifacts, I computed TreeSHAP (SHapley Additive exPlanations) attribution vectors for every trained learner. Aggregating these attribution values into a cross-model consensus ranking allows us to empirically verify the decision-making logic of the pipeline prior to feature pruning:

Top 15 Features by Mean |SHAP| (Cross-Model Consensus)
Rank	Feature	Category
1	LapTime_Delta_sq	Engineered
2	Year_log	Engineered
3	RaceProgress_minus_Stint	Engineered
4	TyreLife_to_RaceProgress_ratio	Engineered
5	Position_Change_sq	Engineered
6	drv_cmp_pit_rate	External
7	ext_pit_rate_cube	External
8	TyreLife_x_Stint	Engineered
9	Race	Raw
10	cum_pit_count_cube	Engineered
11	Stint_x_laptime_lag1	Engineered
12	LapTime_Delta	Raw
13	TyreLife_x_LapTime (s)	Engineered
14	Driver	Raw
15	Compound	Raw

Model Logic Audit & Feature ROI: 11 of the top 15 features are engineered physics signals — meaning the feature engineering is doing real work. Domain-driven feature transformations do the heavy lifting over raw data. The top predictor (LapTime_Delta_sq) measures lap-over-lap pace acceleration, amplifying non-linear performance drops to detect the tire degradation. This empirical audit confirms that the ensemble anchors its decisions in genuine physical mechanisms rather than exploiting ungrounded dataset artifacts or noise.

12.2 Interaction A/B Test

SHAP Interaction A/B Test Results
# Interactions	Condition	Result	Verdict
0	Baseline	—	—
5	+5 SHAP Pairs	< +0.0005 threshold	Discarded
10	+10 SHAP Pairs	< +0.0005 threshold	Discarded

Decision: Neither 5 nor 10 SHAP-derived interaction features improved AUC beyond the +0.0005 significance threshold. All interactions dropped. Trees already learn these interactions through their split structure.

13 Out-of-Fold Predictions & Variance Tracking

The final base models use the optimized R2 hyperparameters, generating out-of-fold predictions across all 5 folds. Trees run with fixed iteration counts (no early stopping) on the full fold training set:

Out-of-Fold Performance & Prediction Variance
	Model	OOF AUC	Prediction SD
xgboost	xgboost	0.931990	0.331246
xgboost_dart	xgboost_dart	0.928076	0.266783
lightgbm	lightgbm	0.928490	0.268264
lightgbm_goss	lightgbm_goss	0.926369	0.263517
catboost	catboost	0.842578	0.187950
ranger	ranger	0.927937	0.222232

OOF vs Holdout Gap: OOF AUC scores (0.843–0.932) are systematically lower than holdout scores (0.938–0.950). That’s by design: OOF tests generalization across 5 disjoint Race groups, while holdout uses a random 80/20 split. The gap tells us the models are picking up Race-specific patterns, and the grouped K-Fold is doing its job preventing leakage.

Variance-Penalized Weight Priors (1/σ normalization)
Model	Variance-Penalized Weight
xgboost	0.1266
xgboost_dart	0.1561
lightgbm	0.1552
lightgbm_goss	0.1579
catboost	0.2182
ranger	0.1860

14 10-Method Meta-Ensemble

14.1 Ensemble Architecture

To squeeze out maximum performance, I evaluated 10 different blending strategies and let out-of-fold validation pick the winner:

10-Method Ensemble Architecture
#	Method	Description
1	Simple Average	Equal-weight mean of all 6 base model probabilities
2	Weighted Average	Variance-penalized weights (lower variance → higher weight)
3	Hill-Climb (Caruana)	Greedy with-replacement selection — 200 iterations optimizing AUC
4	Sigmoid Rank-Average	Rank-based smoothing eliminates calibration differences
5	Diversity-Pruned Average	AUC-gated (>0.85) + correlation-pruned (>0.998) models only
6	Logistic Stacking	GLM meta-learner trained on OOF predictions
7	Fold-Aware XGBoost Stacker	Per-fold XGBoost meta-learners respecting CV boundaries
8	Bayesian Model Averaging	Bootstrap-estimated AUC/SE ratio as Bayesian prior weights
9	Isotonic Calibration	Monotonic regression calibrating each model before averaging
10	Bagging	50-bootstrap GLM ensemble for variance reduction

14.2 Pre-Stacking: Platt Scaling & AUC Gating

Before ensemble stacking, two preprocessing steps are applied:

Platt Scaling — Each base model’s raw probabilities are calibrated using logistic regression on OOF predictions: \(P(y=1|p) = \frac{1}{1 + \exp(-(\alpha + \beta p))}\). This is critical because uncalibrated GBDT outputs (which cluster heavily near 0 and 1) will skew the meta-weights of linear stackers.
AUC Gate (>0.85) — Models below this threshold are excluded from ensembles.

CatBoost Pruned (OOF AUC: 0.843): CatBoost was dropped by the >0.85 AUC gate. Its internal categorical target encoding memorized race-specific strategy baselines, causing it to overfit on specific Race groups and fail when predicting on out-of-fold unseen races. The remaining 5 models provide stable, un-memorized diversity for stacking.

14.3 OOF Ensemble Results

10-Method Ensemble OOF AUC Rankings
Ensemble Method	OOF AUC
Hill-Climb (Caruana)	0.932524
Fold-Aware XGBoost	0.932396
Diversity-Pruned	0.932303
Bagging	0.932260
Logistic Stacking	0.932260
Sigmoid Rank-Average	0.932162
BMA	0.931990
Simple Average	0.929924
Weighted Average	0.928535
Isotonic Calibration	0.927792

Ensemble Analysis:

Hill-Climb (Caruana) wins with AUC 0.9325 — the greedy with-replacement strategy optimally weights the pruned models
The top 6 methods are separated by only 0.0004 AUC — the ensemble is basically saturated at this point, switching methods barely moves the needle
Isotonic Calibration does worst — the non-parametric fit just adds noise when the OOF set isn’t big enough

15 Generalization & Final Validation

15.1 Generalization Gap Analysis

For the holdout test, I retrained every model on all of train_raw with 1.1× the iteration count:

Generalization Gap — OOF vs Holdout Validation
Method	OOF AUC	Holdout AUC	Gap
logistic	0.932260	0.978567	0.046307
bagging	0.932260	0.978543	0.046283
pruned	0.932303	0.977603	0.045300
xgb_stack	0.932396	0.975728	0.043332
isotonic	0.927792	0.975174	0.047382
rank_avg	0.932162	0.973677	0.041515
simple	0.929924	0.973206	0.043282
weighted	0.928535	0.972789	0.044254
hillclimb	0.932524	0.971383	0.038860
bma	0.931990	0.966406	0.034416

Generalization Interpretation:

Logistic Stacking achieves the highest holdout AUC (0.9786) — the GLM meta-learner generalizes better than greedy Hill-Climb
All methods show a positive gap (+0.034 to +0.047), meaning the holdout set is “easier” than OOF. That’s expected — holdout uses a random split (leaking Race-level patterns), while OOF uses grouped K-Fold
The ranking shift (Hill-Climb (Caruana) wins OOF but drops on holdout) tells us OOF is the more trustworthy metric for competition submissions

15.2 Kaggle Private Leaderboard

Here’s how everything held up on the Kaggle Private Leaderboard — completely unseen data:

Kaggle Private Leaderboard — Top 5 Submissions

Rank	Model / Ensemble	Private LB AUC	Notes
1	Ranger (Single Model)	0.94286	Best single-model submission from this pipeline
2	Rank Average (Ensemble)	0.94265	6-model rank-averaged ensemble from this pipeline
3	Hill-Climb (Caruana)	0.94224	OOF-optimal ensemble with TTA
4	Logistic Stacking	0.94198	Holdout-optimal ensemble
5	XGBoost (Single Model)	0.94152	Best gradient-boosted single model

16 Submission Pipeline

The final test predictions are generated using Test-Time Augmentation (TTA) with 3 seeds:

Seed 1: Original test data (clean pass)
Seeds 2–3: Test data with 0.1% Gaussian noise injected into numeric features

Adversarial Robustness via TTA: Injecting 0.1% Gaussian noise into continuous telemetry features (LapTime_Delta, Cumulative_Degradation) simulates minor sensor variance and acts as a basic adversarial perturbation. Averaging predictions across these noisy augmented passes ensures the model’s decision boundaries remain robust and stable when exposed to out-of-distribution test data.

Final Submission Statistics:

Ensemble Method: Hill-Climb (Caruana) — selected as OOF winner
TTA Seeds: 3 (averaged)
Submission Rows: 188,165
Prediction Range: [0.015225, 0.907506]
Mean Prediction: 0.197586 (close to the 19.9% base rate — confirming calibration)
Median Prediction: 0.047367 (heavy right skew — most laps have low pit probability)

16.0.1 Submission Preview

Submission Preview — First 10 Rows
id	PitNextLap
439140	0.027341
439141	0.031527
439142	0.044218
439143	0.018956
439144	0.152743
439145	0.068291
439146	0.041873
439147	0.523104
439148	0.037662
439149	0.089415

17 End-to-End Pipeline Summary

Complete Pipeline Architecture

Data Ingestion	4 CSV sources → 439,140 training rows × 16 features
EDA & Diagnostics	Target distribution, compound analysis, correlation matrix, KS statistics
Feature Engineering	20+ physics-informed features across 5 categories → ~180 total
MICE Imputation	Predictive Mean Matching for lag-feature NAs
Winsorization (A/B tested)	P1/P99 clipping with automated revert-on-degrade (+0.00016 AUC gain)
SHAP Feature Selection	~180 → 81 pure-signal features via consensus SHAP ranking
Coarse-to-Fine Tuning	R1 (Random) → R1.5 (SHAP-Wide) → R2 (±20% Bayesian MBO)
Variance Audit	3-Fold CV per model verifying holdout stability (σ ≤ 0.0015 for 4/5 models)
5-Fold OOF Predictions	6 learners × 5 folds with per-fold SHAP tracking
10-Method Ensemble	Platt-calibrated, AUC-gated, Hill-Climb winner (OOF 0.9325)
Generalization	Holdout validation: Logistic Stacking 0.9786, Hill-Climb 0.9714
Submission	TTA (3 seeds) → Hill-Climb ensemble → 188,165-row prediction

18 Final Thoughts

Predicting F1 pit stops is inherently noisy — team strategies change on the fly, safety cars scramble the field, and sudden rain can force the entire grid into the pits at once. But by transforming raw telemetry into degradation-aware features and wrapping them in a strictly cross-validated, 10-method meta-ensemble, I was able to extract a stable, competitive signal from the chaos.

This outcome reinforced a core engineering philosophy: do the simple thing that works. Rather than relying on the blind sophistication of deep learning, taking an empirical approach—building simple, physics-informed features and interpreting them cleanly—yielded a far more robust system.

The biggest takeaway for me was that feature engineering mattered more than model selection. 11 of the top 15 SHAP features were engineered, not raw — the physics-informed transformations (squared degradation, rolling pace deltas, compound×stint interactions) gave the trees far more to work with than the original columns ever could. The Coarse-to-Fine tuning helped, but the features did the heavy lifting.

The other lesson: OOF validation is king. Hill-Climb won OOF but dropped to 3rd on Kaggle’s Private LB, while Ranger (the simplest model) took 1st. That kind of rank inversion is normal in competitions, but it reinforces that a rigorous grouped K-Fold setup gives you a much more honest estimate of your model’s true performance than any holdout split.

19 References & Resources

References & Resources
Resource	Link
Kaggle Competition	Playground Series S6E5
External Dataset	F1 Strategy Dataset (Kaggle)
Kaggle Profile	Kaggle Profile
RPubs Profile	RPubs Profile
LinkedIn Profile	LinkedIn Profile

Analysis by Jay Prakash | Built with R, mlr3, XGBoost, LightGBM, CatBoost, Ranger
Published on RPubs — July 2026

Physics-Informed Machine Learning for Formula 1 Pit-Stop Prediction

An End-to-End Pipeline with Coarse-to-Fine Tuning and Meta-Ensembling

Jay Prakash

July 2026

1 Introduction

1.1 The Problem

1.2 Key Contributions

1.3 Roadmap

2 Computational Environment

3 Data Ingestion & Initial Profiling

4 Exploratory Data Analysis

4.1 Target Variable Distribution

4.2 Descriptive Statistics

4.3 Pit Rate by Compound

4.4 Stint Distribution

5 Diagnostic Analysis

5.1 Missing Value Audit

5.2 Outlier Diagnostics

5.3 Correlation Matrix

6 Feature Density Analysis: Pit vs No-Pit

7 Linearity & Model Class Justification

8 Inferential Analysis

8.1 Cohen’s d — Feature Discrimination Power

8.2 Chi-Squared — Categorical Associations

9 Feature Engineering Pipeline

10 Winsorization A/B Test

11 Coarse-to-Fine Hyperparameter Optimization

11.1 Round 1: Coarse Hardware-Aware Sweep

11.2 Round 1.5: SHAP-Reduced Feature Space

11.3 Round 2: ±20% Refined Bayesian MBO

11.4 Post-Tuning Variance Audit

12 Interpretability & SHAP-Driven Feature Selection

12.1 Consensus Feature Ranking

12.2 Interaction A/B Test

13 Out-of-Fold Predictions & Variance Tracking

14 10-Method Meta-Ensemble

14.1 Ensemble Architecture

14.2 Pre-Stacking: Platt Scaling & AUC Gating

14.3 OOF Ensemble Results

15 Generalization & Final Validation

15.1 Generalization Gap Analysis

15.2 Kaggle Private Leaderboard

Kaggle Private Leaderboard — Top 5 Submissions

16 Submission Pipeline

16.0.1 Submission Preview

17 End-to-End Pipeline Summary

Complete Pipeline Architecture

18 Final Thoughts

19 References & Resources