Cross-Validation, Discriminant Analysis, and Non-Parametric Tuning

knitr::opts_chunk$set(echo = TRUE, warning = FALSE, message = FALSE, fig.width = 8, fig.height = 5)

# Required libraries
library(mlbench)     # For the PimaIndiansDiabetes dataset
library(tidyverse)   # For data manipulation and ggplot2
library(modelr)      # For elegant cross-validation infrastructure
library(MASS)        # For lda() and qda() functions
library(class)       # For knn() functions
library(gtsummary)   # For statistical summary tables
library(pROC)        # For validation diagnostics
library(DT)          # For cool interactive tables

When to Deploy Cross-Validation

In our previous session, we established a single train/test split to act as a validation firewall. However, a single partition possesses distinct inferential limitations that graduate-level analysts must account for.

The Inferential Vulnerabilities of Train/Test Splits

1. The Validation Variance Penalty: A single split introduces high sampling variance. If our dataset contains structural anomalies or minor sample imbalances, our performance metrics depend heavily on which specific rows randomly fell into the test set.

2. The Data Deprivation Problem: Restricting 20% to 30% of our observations to a testing silo removes structural information from the estimation step. This is especially damaging when sample sizes are small or classes are non-linearly distributed.

The Solution: \(k\)-Fold Cross-Validation

Instead of exposing models to a single testing target, \(k\)-fold cross-validation maps out a systematically robust architecture. The complete training space is divided into \(k\) equal, mutually exclusive subsets (folds). The model is iteratively estimated on \(k-1\) folds and evaluated on the single remaining validation fold. This yields \(k\) distinct out-of-sample error estimates.

When Should You Favor Cross-Validation Over a Single Split?

• Small to Moderate Sample Sizes (\(N < 5,000\)): When data is at a premium (such as our PimaIndiansDiabetes profile with 768 rows), CV ensures every single row spends time as both a training parameter and an out-of-sample evaluation point.

• Hyperparameter Optimization: When selecting tuning parameters—such as the number of neighbors (\(k\)) in KNN or regularization parameters (\(\lambda\)) in Ridge/Lasso—evaluating performance against a single test split quickly overfits the test architecture. CV creates an internal loop that preserves the final test set’s objective framework.

• Unstable Inductive Biases: When evaluating non-parametric models with high variance (like deep trees or high-dimensional KNN), CV provides a realistic average expectation of performance rather than a single stochastic snapshot.

Creating the Cross-Validation Infrastructure

We will utilize the PimaIndiansDiabetes dataset. To eliminate downstream parsing errors within non-parametric algorithms like KNN, we will extract our continuous predictors alongside our target and drop missing records immediately.

We initialize a 10-fold cross-validation frame using modelr::crossv_kfold(). Like our previous workflow, this creates lightweight pointer matrices to optimize RAM utilization.

# Load Data
data("PimaIndiansDiabetes")

# Clean data profile and filter features
clean_data <- PimaIndiansDiabetes %>%
  select(diabetes, glucose, mass, age, pressure) %>%
  drop_na()

set.seed(101) # Guarantee deterministic folds across environments
cv_frame <- crossv_kfold(clean_data, k = 10)

# Inspect the pointer matrix
print(cv_frame)

# A tibble: 10 × 3
   train                test                .id  
   <named list>         <named list>        <chr>
 1 <resample [691 x 5]> <resample [77 x 5]> 01   
 2 <resample [691 x 5]> <resample [77 x 5]> 02   
 3 <resample [691 x 5]> <resample [77 x 5]> 03   
 4 <resample [691 x 5]> <resample [77 x 5]> 04   
 5 <resample [691 x 5]> <resample [77 x 5]> 05   
 6 <resample [691 x 5]> <resample [77 x 5]> 06   
 7 <resample [691 x 5]> <resample [77 x 5]> 07   
 8 <resample [691 x 5]> <resample [77 x 5]> 08   
 9 <resample [692 x 5]> <resample [76 x 5]> 09   
10 <resample [692 x 5]> <resample [76 x 5]> 10   

Notice that cv_frame is just a standard data frame containing 10 rows (one for each fold). Each row contains a pointer to the training data (train) and the validation data (test) for that specific slice.

A Straightforward Evaluation Approach

To avoid writing abstract functions, we can evaluate each fold explicitly by pulling out the data matrices directly. We will initialize empty vectors to store our out-of-sample error rates, and fill them using a basic, readable for loop.

# Initialize storage vectors for our 10 folds
err_logistic <- numeric(10)
err_lda      <- numeric(10)
err_qda      <- numeric(10)

# Loop through each of the 10 folds step-by-step
for (i in 1:10) {
  # Extract explicit data frames for the current fold
  fold_train <- as.data.frame(cv_frame$train[[i]])
  fold_test  <- as.data.frame(cv_frame$test[[i]])
  
  # --- 1. Logistic Regression ---
  fit_log <- glm(diabetes ~ glucose + mass + age + pressure, data = fold_train, family = binomial)
  prob_log <- predict(fit_log, newdata = fold_test, type = "response")
  pred_log <- ifelse(prob_log > 0.5, "pos", "neg")
  err_logistic[i] <- mean(pred_log != fold_test$diabetes)
  
  # --- 2. Linear Discriminant Analysis ---
  fit_lda <- lda(diabetes ~ glucose + mass + age + pressure, data = fold_train)
  pred_lda <- predict(fit_lda, newdata = fold_test)$class
  err_lda[i] <- mean(pred_lda != fold_test$diabetes)
  
  # --- 3. Quadratic Discriminant Analysis ---
  fit_qda <- qda(diabetes ~ glucose + mass + age + pressure, data = fold_train)
  pred_qda <- predict(fit_qda, newdata = fold_test)$class
  err_qda[i] <- mean(pred_qda != fold_test$diabetes)
}

# Calculate the average out-of-sample error across all folds
mean(err_logistic)

[1] 0.2369788

mean(err_lda)

[1] 0.240892

mean(err_qda)

[1] 0.2499658

Tuning KNN

Now let’s incorporate our non-parametric model, K-Nearest Neighbors (KNN). Because KNN calculates Euclidean distances, we must normalize our features (\(z\)-scores) using training parameters to prevent data leakage. We will test an array of odd choices for \(k\) to find the optimal neighborhood density.

# Define hyperparameter candidate grid 
k_grid = seq(1, 51, by = 2)
knn_results = numeric(length(k_grid))

# Outer loop: Iterate through each candidate value of K
for (k_idx in seq_along(k_grid)) {
  current_k = k_grid[k_idx]
  fold_errors = numeric(10)
  
  # Inner loop: Compute cross-validation error for this specific K
  for (i in 1:10) {
    train = as.data.frame(cv_frame$train[[i]])
    test  = as.data.frame(cv_frame$test[[i]])
    
    # EXPLICIT ALIGNMENT: Select only the 4 formula predictors
    train_x = train %>% select(glucose, mass, age, pressure)
    test_x  = test %>% select(glucose, mass, age, pressure)
    
    # Scale features using training metrics strictly
    means = colMeans(train_x)
    sds   = apply(train_x, 2, sd)
    train_x_scaled = scale(train_x, center = means, scale = sds)
    test_x_scaled  = scale(test_x, center = means, scale = sds)
    
    preds = knn(train = train_x_scaled, test = test_x_scaled, cl = train$diabetes, k = current_k)
    fold_errors[i] = mean(preds != test$diabetes)
  }
  
  # Store the average cross-validated error for this K
  knn_results[k_idx] = mean(fold_errors)
}

# Combine into a tuning summary table
tuning_curve = tibble(K = k_grid, CV_Error = knn_results)
best_knn = tuning_curve %>% filter(CV_Error == min(CV_Error)) %>% slice(1)
print(best_knn )

# A tibble: 1 × 2
      K CV_Error
  <dbl>    <dbl>
1    25    0.221

Compiling and Presenting Final Results

Let’s gather our final out-of-sample cross-validation error rates across all four model candidates and view them in a clear, interactive format.

# Compile final summary data frame
final_summary <- tibble(
  Model = c("Quadratic Discriminant Analysis (QDA)", 
            "Logistic Regression", 
            "Linear Discriminant Analysis (LDA)", 
            paste0("Tuned KNN (K = ", best_knn$K, ")")),
  CV_Error = c(mean(err_qda), mean(err_logistic), mean(err_lda), best_knn$CV_Error)
)

# Render interactive HTML table sorted smallest to largest error
datatable(
  final_summary,
  colnames = c("Model Candidate", "10-Fold CV Error Rate"),
  options = list(dom = 't', order = list(list(1, 'asc'))),
  rownames = FALSE
) %>%
  formatPercentage('CV_Error', digits = 1)

NA

The Supremacy of Non-Parametric Agnosticism

Looking directly at the interactive metrics table:

• Tuned KNN (K = 33) wins the out-of-sample race with a definitive error rate of 22.3%.
• Logistic Regression and LDA form a rigid linear tier in the middle at 23.7% and 24.1%.
• QDA drops to the absolute bottom, suffering the highest error rate at 25.0%.

Geometrically Complex Decision Boundaries

This outcome delivers a crucial lesson on the Bias-Variance Tradeoff that completely upends standard textbook intuition:

Why KNN Outperforms the Linear Models

Logistic Regression and LDA assume that the log-odds change linearly and that the boundary separating healthy and diabetic patients is a flat hyperplane. If the real-world biological boundary is highly irregular, winding, or pocketed with non-linear interactions, these models suffer from severe systemic structural bias.

KNN carries no parametric baggage. By standardizing the features and tuning the neighborhood density to \(k = 33\), KNN smooths out localized stochastic noise while remaining completely flexible. It is free to bend and wrap around highly complex, organic clusters in \(\mathbb{R}^4\) that a linear hyperplane is mathematically incapable of capturing.

The Deceptive Nature of QDA (The Variance Penalty)

In our previous exploratory analysis of the class-specific covariance matrices (\(\Sigma_{neg}\) and \(\Sigma_{pos}\)), we found clear empirical evidence of heteroscedasticity—the variance of glucose expanded by 44% in the diabetic cohort. Theoretically, this should justify QDA’s curved decision boundary.

However, look at the out-of-sample validation error: QDA performs the worst (25.0%). Why? Because QDA attempts to fit a strict parametric curve (a quadratic surface) by estimating a massive suite of separate covariance parameters (\(K \times p(p+1)/2 = 20\) elements). This introduces estimation variance. When exposed to shifting cross-validation folds, QDA’s curved boundary overfits the noise of the training data, collapsing out-of-sample.

KNN, by contrast, achieves its non-linear flexibility through local voting rather than rigid parameter estimation, bypassing this variance penalty entirely.

Tuned KNN (K = 33) is our definitive operational standard. It proves that when nature presents a complex, non-linear coordination of physical metrics, abandoning rigid parametric assumptions and relying on scaled local geometry yields the most robust generalization profile.

More Advanced Modeling Pipeline: Parametric Loops (Logistic, LDA, QDA)

Because modelr stores our folds as lists of resample objects, we can build custom functions to systematically map models over our cross-validation splits. This keeps our code highly modular and reproducible.

We will write functions that accept a resample split, convert it into explicit data frames inside the environment, fit the architecture, and return out-of-sample classifications based on a standard 0.5 decision threshold.

# Logistic Regression Evaluation Function
evaluate_logistic <- function(split) {
  # Explicitly extract standard data frames from the resample pointer
  train <- as.data.frame(split)
  test  <- as.data.frame(split)
  
  fit <- glm(diabetes ~ glucose + mass + age + pressure, data = train, family = binomial)
  probs <- predict(fit, newdata = test, type = "response")
  preds <- ifelse(probs > 0.5, "pos", "neg")
  
  return(mean(preds != test$diabetes))
}

# LDA Evaluation Function
evaluate_lda <- function(split) {
  train <- as.data.frame(split)
  test  <- as.data.frame(split)
  
  fit <- lda(diabetes ~ glucose + mass + age + pressure, data = train)
  preds <- predict(fit, newdata = test)$class
  
  return(mean(preds != test$diabetes))
}

# QDA Evaluation Function
evaluate_qda <- function(split) {
  train <- as.data.frame(split)
  test  <- as.data.frame(split)
  
  fit <- qda(diabetes ~ glucose + mass + age + pressure, data = train)
  preds <- predict(fit, newdata = test)$class
  
  return(mean(preds != test$diabetes))
}

Now, we execute these functions across all 10 folds using purrr::map_dbl and append the out-of-sample error tracks directly to our data frame structure.

cv_results <- cv_frame %>%
  mutate(
    err_logistic = map_dbl(test, evaluate_logistic),
    err_lda      = map_dbl(test, evaluate_lda),
    err_qda      = map_dbl(test, evaluate_qda)
  )

# Review empirical out-of-sample error distribution
cv_results %>%
  select(starts_with("err_")) %>%
  summary()

  err_logistic       err_lda          err_qda      
 Min.   :0.1429   Min.   :0.1447   Min.   :0.1429  
 1st Qu.:0.1721   1st Qu.:0.1954   1st Qu.:0.1475  
 Median :0.2157   Median :0.2078   Median :0.1895  
 Mean   :0.2070   Mean   :0.2082   Mean   :0.1926  
 3rd Qu.:0.2338   3rd Qu.:0.2338   3rd Qu.:0.2240  
 Max.   :0.2727   Max.   :0.2597   Max.   :0.2727  

Tuning KNN via Cross-Validation

Unlike Logistic Regression, LDA, and QDA, \(K\)-Nearest Neighbors (KNN) makes no structural assumptions about parametric forms or underlying Gaussian densities. It is a purely memory-based, non-parametric classifier.

The Importance of Feature Scaling in KNN

KNN determines class assignments based entirely on the Euclidean distance between points in \(\mathbb{R}^p\):

\[d(x_i, x_j) = \sqrt{\sum_{m=1}^p (x_{im} - x_{jm})^2}\]

Recall our previous analysis of the pooled covariance matrix: the variance of glucose (\(\sigma^2 \approx 762\)) is massive compared to mass (\(\sigma^2 \approx 59\)). Because Euclidean distance is highly sensitive to the raw numerical scale of your metrics, an unscaled KNN model will treat glucose as the dominant predictor, completely ignoring variations in mass and age.

Therefore, we must standardize our features (\(z\)-scores with \(\mu=0, \sigma=1\)) inside the validation loop to prevent structural data leakage.

Let’s build a robust tuning function that scales the data using training parameters and evaluates the misclassification rate for varying choices of hyperparameter \(k\).

# Aligned KNN evaluation function using clean purrr inputs
evaluate_knn <- function(tr_pointer, te_pointer, k_val) {
  # Explicitly force clean data frames from the resample maps
  train_df <- as.data.frame(tr_pointer)
  test_df  <- as.data.frame(te_pointer)
  
  # EXPLICIT ALIGNMENT: Lock down the exact 4 formula predictors
  train_x <- train_df %>% select(glucose, mass, age, pressure)
  test_x  <- test_df  %>% select(glucose, mass, age, pressure)
  
  # Normalize features strictly on training bounds to block data leakage
  means <- colMeans(train_x)
  sds   <- apply(train_x, 2, sd)
  train_x_scaled <- scale(train_x, center = means, scale = sds)
  test_x_scaled  <- scale(test_x, center = means, scale = sds)
  
  # Fit KNN and evaluate misclassification rate
  preds <- knn(train = train_x_scaled, test = test_x_scaled, cl = train_df$diabetes, k = k_val)
  return(mean(preds != test_df$diabetes))
}

# Define hyperparameter candidate grid (Odd values to prevent ties)
k_grid <- seq(1, 51, by = 2)

# Execute the hyperparameter search using pure purrr mapping
knn_tuning_results <- tibble(K = k_grid) %>%
  mutate(
    mean_error = map_dbl(K, function(k) {
      # Pass train and test columns concurrently to preserve data frame attributes
      map2_dbl(cv_results$train, cv_results$test, ~evaluate_knn(.x, .y, k_val = k)) %>% mean()
    })
  )

# Extract your optimal model profile
best_knn <- knn_tuning_results %>% filter(mean_error == min(mean_error)) %>% slice(1)
print(best_knn)

# A tibble: 1 × 2
      K mean_error
  <dbl>      <dbl>
1    25      0.221

The “Double CV” Myth

A common question among beginning predictive modelers is: “Do I need to run cross-validation twice for KNN—once to find the best \(k\), and a second time to figure out the model’s true out-of-sample error?”

The answer is no. Running a second, separate cross-validation loop is completely unnecessary here because of how we built our framework.

Why a Second Run is Redundant

When you evaluate a hyperparameter grid using \(k\)-fold cross-validation, the cross-validated error rate you calculate for your chosen \(k\) (e.g., \(k = 25\)) is already a valid, unbiased estimate of the out-of-sample error.

Because every single validation fold was held out entirely while the KNN algorithm counted its neighbors on the corresponding training fold, the error rate assigned to best_knn$CV_Error has never seen the test data points.

• When would you need a second loop (Nested CV)? You only need nested cross-validation if you are trying to select the best overall algorithm out of a massive suite of tuned models (e.g., comparing a tuned Random Forest against a tuned Support Vector Machine). In that scenario, you use an outer loop to compare the algorithms and an inner loop to tune their respective hyperparameters. But for a single model like KNN, the tuning error profile is your out-of-sample generalization metric.

Visualizing the Hyperparameter Tuning Curve

Let’s plot our cross-validated misclassification track to determine the optimal bias-variance balance for our KNN classifier.

# Step 1: Programmatically extract the coordinates of the lowest error point
optimal_point <- knn_tuning_results %>% 
  filter(mean_error == min(mean_error)) %>% 
  slice(1) # Handles ties safely by picking the first occurrence

# Step 2: Render the graph with the highlighted minimum
ggplot(knn_tuning_results, aes(x = K, y = mean_error)) +
  geom_line(color = "#7570b3", size = 1) +
  geom_point(color = "#7570b3", size = 2) +
  
  # Highlight Layer: Overlay a prominent point on the minimum
  geom_point(data = optimal_point, aes(x = K, y = mean_error), 
             color = "#e74c3c", size = 4, shape = 16) +
  
  # Text Annotation: Label the optimal point with its specific parameters
  annotate("text", 
           x = optimal_point$K, 
           y = optimal_point$mean_error + 0.005, # Nudges text slightly above the point
           label = paste0("Optimal K = ", optimal_point$K),
           color = "#e74c3c", fontface = "bold", hjust = 0) +
  
  scale_x_reverse() + # Reversed axis shows low K (High Variance) to high K (High Bias)
  labs(
    title = "KNN Hyperparameter Tuning via 10-Fold Cross-Validation",
    subtitle = "The highlighted minimum minimizes out-of-sample misclassification risk",
    x = "Hyperparameter Choice: Neighbors (K) [Axis Reversed]",
    y = "Cross-Validated Misclassification Rate"
  ) +
  theme_minimal()

# Extract the mathematically optimal choice for neighbor count
optimal_knn <- knn_tuning_results %>% filter(mean_error == min(mean_error)) %>% slice(1)
print(optimal_knn)

Performance Comparison

Let’s compute the mean and standard error of our out-of-sample errors across all four model candidates to make a definitive structural selection.

# Compile parametric error metrics
final_summary <- cv_results %>%
  summarise(
    Logistic = mean(err_logistic),
    LDA      = mean(err_lda),
    QDA      = mean(err_qda)
  ) %>%
  pivot_longer(cols = everything(), names_to = "Model", values_to = "CV_Error")

# Append our optimally tuned KNN track
final_summary <- final_summary %>%
  add_row(Model = paste0("Tuned KNN (K = ", optimal_knn$K, ")"), CV_Error = optimal_knn$mean_error)

# Load the interactive DT library
library(DT)

# Convert your final summary to an interactive DataTable widget
datatable(
  final_summary,
  colnames = c("Model Candidate", "10-Fold CV Error Rate"),
  options = list(
    pageLength = 5,           # Keep the table compact
    dom = 't',                # Remove unnecessary search/pagination bars since it's only 4 rows
    order = list(list(1, 'asc')) # Sort by the second column (index 1) from smallest to largest
  ),
  rownames = FALSE            # Strip row indices for a cleaner layout
) %>%
  formatPercentage('CV_Error', digits = 1) # Format decimals to percentages with 1 decimal place

Conclusion: Which Model Fits Best?

Looking closely at our final 10-fold cross-validated error rates, we see a distinct shift from our previous workflow:

• Quadratic Discriminant Analysis (QDA) achieves the lowest out-of-sample error rate at 19.3%.
• Logistic Regression and LDA perform almost identically, lagging behind at 20.7% and 20.8% error respectively.
• Tuned KNN (K = 25) demonstrates the highest out-of-sample error at 22.1%.

When the Math and the Metrics Align

This is a textbook demonstration of the Bias-Variance Tradeoff working in favor of the more complex model.

Recall our earlier exploratory diagnostic where we analyzed the class-specific covariance matrices (\(\Sigma_{neg}\) and \(\Sigma_{pos}\)). We uncovered undeniable empirical evidence of severe heteroscedasticity—most notably, the variance of glucose expanded by 44% in the diabetic cohort, and the correlation structures between mass, age, and glucose completely altered directions or collapsed between the two groups.

Because the true underlying feature distributions do not share a common covariance structure, the linear boundaries forced by LDA and Logistic Regression suffer from systemic bias. They are mathematically incapable of bending to accommodate the expanding volume of the diabetic cohort.

QDA pays a parameter penalty to estimate separate covariance matrices (\(K \times p(p+1)/2 = 20\) parameters). However, because our training data is sufficiently dense (\(N = 768\)), the variance introduced by estimating those extra 10 parameters is offset by the massive reduction in model bias. The quadratic, curved decision boundary is a reflection of the biological data-generating process.

Moving past the principle of parsimony, Quadratic Discriminant Analysis (QDA) is the definitive operational standard for this dataset. It successfully leverages class-specific covariance structures to deliver a statistically significant and reproducible 1.4% to 1.5% out-of-sample performance breakthrough over its linear peers.

Why did the results flip?

First, I’m not entirely sure. This sort of result is why I rely on caret’s functionality to maintain stability across model comparison. We’ll learn more about this in a later module.

K-Nearest Neighbors (KNN) maintains completely stable, identical test error rates across both code versions, while the parametric models (Logistic Regression, LDA, and QDA) vary or inverted results (QDA!).

Because KNN’s error rate is perfectly static, I’m guessing that our cross-validation partitions and underlying data slices are identical between runs. Instead, the discrepancy is driven by how R evaluates prediction outputs behind the scenes when migrating code into an automated pipeline (like purrr::map).

This is good, cause for some of you first time R coders, seeing for loops and purrr might be overwhelming. These complex coding nuances are exactly the reason I introduce caret. Later though, I must torture you for a few weeks more before introducing caret. That way you’ll have an appreciation for it’s simple functions.

For today, I really just wanted to expose you to cross-validation and how it can be used. You saw how we can use it to estimate out-of-sample error in small data sets, and you also saw how we can tune hyperparameters (like the k in knn).

Beyond calculating a cleaner generalization error or dialing in hyperparameters like \(k\) in KNN, cross-validation is a foundational mechanism for several advanced architecture-building and diagnostic workflows.

At a graduate level, cross-validation is leveraged not just as an evaluation tool, but as an algorithmic component to stabilize, combine, and diagnose models.

Other Use Cases for Cross-Validation in Predictive Modeling

Aside from these, here are the four primary alternative use cases for cross-validation in predictive modeling:

Feature Selection and Wrapper Methods (e.g., Maximize Parsimony)

When evaluating which predictors to keep in a model, evaluating feature importance on the training set leads to severe overfitting. Cross-validation is used as the objective engine inside wrapper selection algorithms like Recursive Feature Elimination (RFE) or Stepwise Selection.

• How it works: The algorithm removes a feature, runs a full cross-validation loop to calculate the change in out-of-sample error, and decides whether that feature’s predictive signal is real or just training noise.

• Why it matters: This ensures that you only include features that consistently improve generalization across all folds, protecting the model from the curse of dimensionality.

Stacking and Ensembling (Meta-Learning Meta-Features)

Model Stacking combines entirely different model architectures (e.g., blending a Logistic Regression, a Random Forest, and a Support Vector Machine) by training a “meta-model” to learn how to weight their individual predictions. To do this safely without massive data leakage, you must use cross-validation.

• How it works: You run cross-validation on your baseline models. For each fold, you generate out-of-sample predictions. You stitch these out-of-sample predictions together to form an entirely new dataset of “meta-features.”

• Why it matters: The meta-model is then trained on these out-of-sample predictions. If you used standard training predictions instead, the meta-model would quickly learn to over-rely on whichever baseline model overfits the training data the most.

Regularization and Cost-Complexity Pruning

In models like Lasso/Ridge regression or Decision Trees, cross-validation acts as the direct mathematical brake on model growth.

• Lasso/Ridge (\(\lambda\) selection): CV determines the exact penalty applied to the size of the regression coefficients.

• Decision Trees (Cost-Complexity Pruning): A fully grown tree will overfit perfectly. We use cross-validation to calculate the out-of-sample penalty for adding more terminal leaves (\(C_p\)). The tree is pruned back to the exact node depth where the cross-validated error curve reaches its global minimum.

Diagnosing Stability and Structural Data Shifting

Cross-validation is a powerful diagnostic tool for measuring model stability and uncovering data anomalies.

Instead of just looking at the mean_error across your 10 folds, a rigorous analyst looks closely at the variance or standard deviation of the error across the folds.

• Low Variance Across Folds: Indicates that the model’s performance is stable. The data-generating process is homogeneous, and the model is robust to minor shifts in the training distributions.

• High Variance Across Folds: If Fold 3 has a 12% error rate but Fold 7 spikes to a 34% error rate, it flags a massive structural issue. It means your dataset is highly heterogeneous, contains localized hidden subgroups, or possesses severe class imbalances that randomly cluster into specific folds. This signals that your model’s inductive bias is highly unstable and will likely fail when deployed into production environments.