Exercise 7.2

The Friedman simulation uses X1–X5 to generate the outcome. X6–X10 are noise predictors. I tune four nonlinear models on 200 observations using the same 10 cross-validation (CV) folds, then compare them on a separate sample of 5,000 observations.

set.seed(200)
sim_train <- mlbench::mlbench.friedman1(200, sd = 1)
sim_test <- mlbench::mlbench.friedman1(5000, sd = 1)
predictor_names <- paste0("X", 1:10)
x72 <- setNames(as.data.frame(sim_train$x), predictor_names)
test_x72 <- setNames(as.data.frame(sim_test$x), predictor_names)

set.seed(200)
folds72 <- createFolds(sim_train$y, k = 10, returnTrain = TRUE)
cv72 <- trainControl(method = "cv", index = folds72)
scaled <- c("center", "scale")
models72 <- list(
  KNN = list(method = "knn", preProcess = scaled, tuneLength = 10),
  "Averaged neural network" = list(
    method = "avNNet", preProcess = scaled,
    tuneGrid = expand.grid(decay = c(0, 0.01, 0.1), size = 1:10, bag = FALSE),
    linout = TRUE, trace = FALSE, maxit = 500
  ),
  MARS = list(method = "earth", tuneGrid = mars_grid),
  "Radial SVM" = list(method = "svmRadial", preProcess = scaled, tuneLength = 10)
)
fits72 <- lapply(models72, fit_model, x = x72, y = sim_train$y,
                 control = cv72, seed = 200)
scores72 <- score_models(fits72, test_x72, sim_test$y)
show_scores(scores72)
Model CV RMSE Test RMSE Test R²
MARS 1.202 1.280 0.934
Radial SVM 1.853 2.074 0.826
Averaged neural network 2.028 2.089 0.827
KNN 3.044 3.122 0.669
mars_fit <- fits72$MARS
mars_used <- rownames(earth::evimp(mars_fit$finalModel, trim = TRUE))
informative <- predictor_names[1:5]
mars_informative <- intersect(informative, mars_used)
mars_noise <- intersect(predictor_names[6:10], mars_used)

Findings. MARS had the lowest CV RMSE (1.202) and test RMSE (1.280). An RMSE of 1 is the benchmark set by the simulation’s noise. MARS selected 5 of the five informative predictors (X1, X2, X3, X4, X5) and 0 noise predictors. MARS approximates the smooth signal with piecewise-linear terms and can use products of those terms to capture interactions.

Exercise 7.5

I use the same 80/20 split, 10 CV folds, and preprocessing as in Exercise 6.3. Within each fold, nzv removes near-constant predictors and knnImpute standardizes predictors and fills missing values using five nearest neighbors from the fitting rows. The final fits apply the preprocessing learned from the full training set when predicting the test runs.

data("ChemicalManufacturingProcess", package = "AppliedPredictiveModeling")
chem <- ChemicalManufacturingProcess
x <- chem[, names(chem) != "Yield"]
y <- chem$Yield

set.seed(624)
train_index <- drop(createDataPartition(y, p = 0.8, list = FALSE))
train_x <- x[train_index, , drop = FALSE]
train_y <- y[train_index]
test_x <- x[-train_index, , drop = FALSE]
test_y <- y[-train_index]

folds75 <- createFolds(train_y, k = 10, returnTrain = TRUE)
cv75 <- trainControl(method = "cv", index = folds75)
prep75 <- c("nzv", "knnImpute")

(a) Model performance

I trained four nonlinear models and refitted the PLS and elastic net models from Exercise 6.3. I use the lowest CV RMSE to select the nonlinear and linear models for the predictor comparison. RMSE is in percentage points of yield.

linear75 <- list(
  PLS = list(method = "pls", tuneLength = 20),
  "Elastic net" = list(
    method = "enet",
    tuneGrid = expand.grid(lambda = c(0.001, 0.01, 0.1),
                           fraction = seq(0.05, 1, length.out = 20))
  )
)

nonlinear75 <- list(
  KNN = list(method = "knn", tuneLength = 10),
  "Averaged neural network" = list(
    method = "avNNet",
    tuneGrid = expand.grid(decay = c(0, 0.01, 0.1), size = c(1, 3, 5, 7, 9),
                           bag = FALSE),
    linout = TRUE, trace = FALSE, repeats = 3, maxit = 500
  ),
  MARS = list(method = "earth", tuneGrid = mars_grid),
  "Radial SVM" = list(method = "svmRadial", tuneLength = 10)
)

models75 <- lapply(c(linear75, nonlinear75), function(settings) {
  c(settings, list(preProcess = prep75))
})
fits75 <- lapply(models75, fit_model, x = train_x, y = train_y,
                 control = cv75, seed = 624)
scores75 <- score_models(fits75, test_x, test_y)
linear_scores <- scores75[scores75$Model %in% names(linear75), ]
nonlinear_scores <- scores75[scores75$Model %in% names(nonlinear75), ]
best_linear <- linear_scores$Model[1]
best_nonlinear <- nonlinear_scores$Model[1]
show_scores(scores75)
Model CV RMSE Test RMSE Test R²
MARS 1.099 1.172 0.704
Radial SVM 1.135 1.093 0.778
Elastic net 1.201 1.281 0.721
KNN 1.272 1.402 0.613
Averaged neural network 1.353 1.393 0.568
PLS 1.395 1.202 0.676

Finding. MARS had the lowest CV RMSE (1.099). Radial SVM had the lowest test RMSE (1.093). The best linear model by CV RMSE was Elastic net (CV RMSE 1.201).

linear_fit <- fits75[[best_linear]]
nonlinear_fit <- fits75[[best_nonlinear]]
linear_error <- test_y - predict(linear_fit, newdata = test_x)
nonlinear_error <- test_y - predict(nonlinear_fit, newdata = test_x)

process_variable <- "ManufacturingProcess16"
training_range <- range(train_x[[process_variable]], na.rm = TRUE)
unusual_row <- which.min(test_x[[process_variable]])
unusual_value <- test_x[[process_variable]][unusual_row]

rmse <- function(errors) sqrt(mean(errors^2))
full_rmse <- c(rmse(linear_error), rmse(nonlinear_error))
reduced_rmse <- c(rmse(linear_error[-unusual_row]), rmse(nonlinear_error[-unusual_row]))

One test run had ManufacturingProcess16 equal to 0 (training range 4441–4775). Excluding that same run from both sets of predictions moves the Elastic net RMSE from 1.28 to 1.30 and the MARS RMSE from 1.17 to 1.18. The table reports scores on all 32 test runs.

(b) Most important predictors

I refit the selected nonlinear model at its chosen settings in each CV fold. On the validation rows, I shuffle one predictor’s observed values at a time and measure the increase in RMSE. Each score uses five shuffles per fold. Larger increases indicate that predictions depend more on that predictor. The linear ranking uses the importance method from Exercise 6.3.

selected <- models75[[best_nonlinear]]
selected$tuneLength <- NULL
selected$tuneGrid <- nonlinear_fit$bestTune
one_fit <- trainControl(method = "none")

fold_errors <- lapply(seq_along(folds75), function(fold) {
  rows <- folds75[[fold]]
  valid_x <- train_x[-rows, , drop = FALSE]
  valid_y <- train_y[-rows]
  fit <- fit_model(selected, train_x[rows, , drop = FALSE], train_y[rows],
                   control = one_fit, seed = 624 + fold)
  squared_error <- function(new_x) {
    sum((valid_y - predict(fit, newdata = new_x))^2)
  }
  original <- squared_error(valid_x)

  set.seed(1624 + fold)
  shuffled <- vapply(names(valid_x), function(predictor) {
    if (predictor %in% fit$preProcess$method$remove) return(original)
    values <- valid_x[[predictor]]
    measured <- which(!is.na(values))
    if (length(measured) < 2) return(original)

    altered_x <- valid_x
    errors <- replicate(5, {
      altered_x[[predictor]][measured] <- sample(values[measured])
      squared_error(altered_x)
    })
    mean(errors)
  }, numeric(1))
  c(original = original, shuffled)
})

pooled_rmse <- sqrt(colSums(do.call(rbind, fold_errors)) / length(train_y))
kept <- setdiff(names(train_x), nonlinear_fit$preProcess$method$remove)
rmse_increase <- sort(pooled_rmse[kept] - pooled_rmse[["original"]],
                      decreasing = TRUE)
top_nonlinear <- head(rmse_increase, 10)
if (linear_fit$method == "enet") {
  coefficients <- elasticnet::predict.enet(
    linear_fit$finalModel, s = linear_fit$bestTune$fraction,
    mode = "fraction", type = "coefficients"
  )$coefficients
  linear_importance <- abs(coefficients[coefficients != 0])
} else {
  importance_table <- varImp(linear_fit)$importance
  linear_importance <- setNames(importance_table$Overall, rownames(importance_table))
}

top_linear <- head(sort(linear_importance, decreasing = TRUE), 10)
ranks <- seq_len(max(length(top_nonlinear), length(top_linear)))
comparison <- data.frame(
  Rank = ranks,
  Nonlinear = names(top_nonlinear)[ranks],
  Increase = unname(top_nonlinear[ranks]),
  Linear = names(top_linear)[ranks]
)
knitr::kable(comparison, digits = 3, row.names = FALSE,
             col.names = c("Rank", best_nonlinear, "RMSE increase", best_linear))
Rank MARS RMSE increase Elastic net
1 ManufacturingProcess32 0.943 ManufacturingProcess32
2 ManufacturingProcess09 0.447 ManufacturingProcess09
3 ManufacturingProcess28 0.219 ManufacturingProcess34
4 BiologicalMaterial06 0.084 ManufacturingProcess17
5 ManufacturingProcess39 0.084 ManufacturingProcess06
6 ManufacturingProcess17 0.070 BiologicalMaterial06
7 ManufacturingProcess24 0.062 ManufacturingProcess44
8 ManufacturingProcess05 0.040 ManufacturingProcess36
9 BiologicalMaterial09 0.040 ManufacturingProcess37
10 BiologicalMaterial12 0.031 ManufacturingProcess11
count_biological <- function(predictors) sum(startsWith(predictors, "Biological"))
bio_nonlinear <- count_biological(names(top_nonlinear))
bio_linear <- count_biological(names(top_linear))
shared <- intersect(names(top_nonlinear), names(top_linear))
unique_nonlinear <- setdiff(names(top_nonlinear), names(top_linear))

Finding. Shuffling ManufacturingProcess32 and ManufacturingProcess09 produced the two largest increases in prediction error. The MARS list includes 7 process and 3 biological predictors. The Elastic net list includes 9 process and 1 biological predictors. The lists share 4 (ManufacturingProcess32, ManufacturingProcess09, BiologicalMaterial06, ManufacturingProcess17). Process measurements account for 45 of the 57 original predictors.

(c) Relationships with yield

Each plot shows observed training runs for a predictor unique to the nonlinear top ten. The dashed line is a straight-line fit. Blue diamonds mark the median predictor value and median yield in the low, middle, and high groups, where those groups contain observations. The cutoffs are the 33rd and 67th percentiles of the observed predictor values.

if (length(unique_nonlinear)) {
  plot_data <- do.call(rbind, lapply(unique_nonlinear, function(predictor) {
    measured <- !is.na(train_x[[predictor]])
    data.frame(Predictor = predictor, Value = train_x[[predictor]][measured],
               Yield = train_y[measured])
  }))
  plot_data$Predictor <- factor(plot_data$Predictor, levels = unique_nonlinear)
  panels <- split(plot_data, plot_data$Predictor)
  grouped <- lapply(panels, function(panel) {
    cuts <- quantile(panel$Value, c(1 / 3, 2 / 3))
    panel$Band <- 1L + (panel$Value > cuts[1]) + (panel$Value > cuts[2])
    panel
  })
  band_medians <- aggregate(cbind(Value, Yield) ~ Predictor + Band,
                            data = do.call(rbind, grouped), FUN = median)

  ggplot(plot_data, aes(Value, Yield)) +
    geom_point(alpha = 0.45) +
    geom_smooth(method = "lm", formula = y ~ x, se = FALSE,
                linetype = "dashed", colour = "grey40") +
    geom_point(data = band_medians, shape = 18, size = 3, colour = "blue") +
    facet_wrap(~ Predictor, scales = "free_x") +
    labs(x = "Observed predictor value", y = "Yield (%)") +
    theme_minimal()
}

# Explain what the plots reveal about yield for Exercise 7.5(c).
# results = "asis" lets knitr render the text from cat() as Markdown.
if (length(unique_nonlinear)) {
  # Spearman compares the ranks of the observed predictor values and yields.
  correlations <- vapply(panels, function(panel) {
    cor(panel$Value, panel$Yield, method = "spearman")
  }, numeric(1))
  types <- ifelse(startsWith(names(correlations), "Biological"),
                  "biological", "process")
  cat("**Findings.**\n\n")
  for (group in sort(unique(types))) {
    group_cor <- correlations[types == group]

    # Pick the strongest association in either direction for this predictor type.
    predictor <- names(group_cor)[which.max(abs(group_cor))]

    # Reuse the yield medians marked by the blue diamonds in the plots.
    medians <- band_medians[band_medians$Predictor == predictor, ]
    names_of_bands <- c("low", "middle", "high")[medians$Band]
    values <- sprintf("%.2f%% (%s)", medians$Yield, names_of_bands)
    finding <- paste("- **%s** has the strongest rank association among the %s",
                     "predictors shown (Spearman %.2f). Median yield by group was %s.\n")
    cat(sprintf(finding, predictor, group, correlations[[predictor]],
                paste(values, collapse = ", ")))
  }
  if ("ManufacturingProcess28" %in% unique_nonlinear) {
    cat("\nManufacturingProcess28 has clusters near 0 and 10.",
        "Median yield rises most in the high group.",
        "The gap between clusters leaves part of the predictor's range unobserved.\n")
  }
} else {
  cat("**Finding.** The nonlinear top ten contained no predictors unique to that list.\n")
}

Findings.

  • BiologicalMaterial12 has the strongest rank association among the biological predictors shown (Spearman 0.42). Median yield by group was 38.67% (low), 40.11% (middle), 40.89% (high).
  • ManufacturingProcess28 has the strongest rank association among the process predictors shown (Spearman 0.38). Median yield by group was 39.58% (low), 39.79% (middle), 41.50% (high).

ManufacturingProcess28 has clusters near 0 and 10. Median yield rises most in the high group. The gap between clusters leaves part of the predictor’s range unobserved.