The Friedman simulation uses X1–X5 to generate the outcome. X6–X10 are noise predictors. I tune four nonlinear models on 200 observations using the same 10 cross-validation (CV) folds, then compare them on a separate sample of 5,000 observations.
set.seed(200)
sim_train <- mlbench::mlbench.friedman1(200, sd = 1)
sim_test <- mlbench::mlbench.friedman1(5000, sd = 1)
predictor_names <- paste0("X", 1:10)
x72 <- setNames(as.data.frame(sim_train$x), predictor_names)
test_x72 <- setNames(as.data.frame(sim_test$x), predictor_names)
set.seed(200)
folds72 <- createFolds(sim_train$y, k = 10, returnTrain = TRUE)
cv72 <- trainControl(method = "cv", index = folds72)
scaled <- c("center", "scale")
models72 <- list(
KNN = list(method = "knn", preProcess = scaled, tuneLength = 10),
"Averaged neural network" = list(
method = "avNNet", preProcess = scaled,
tuneGrid = expand.grid(decay = c(0, 0.01, 0.1), size = 1:10, bag = FALSE),
linout = TRUE, trace = FALSE, maxit = 500
),
MARS = list(method = "earth", tuneGrid = mars_grid),
"Radial SVM" = list(method = "svmRadial", preProcess = scaled, tuneLength = 10)
)
fits72 <- lapply(models72, fit_model, x = x72, y = sim_train$y,
control = cv72, seed = 200)
scores72 <- score_models(fits72, test_x72, sim_test$y)
show_scores(scores72)
| Model | CV RMSE | Test RMSE | Test R² |
|---|---|---|---|
| MARS | 1.202 | 1.280 | 0.934 |
| Radial SVM | 1.853 | 2.074 | 0.826 |
| Averaged neural network | 2.028 | 2.089 | 0.827 |
| KNN | 3.044 | 3.122 | 0.669 |
mars_fit <- fits72$MARS
mars_used <- rownames(earth::evimp(mars_fit$finalModel, trim = TRUE))
informative <- predictor_names[1:5]
mars_informative <- intersect(informative, mars_used)
mars_noise <- intersect(predictor_names[6:10], mars_used)
Findings. MARS had the lowest CV RMSE (1.202) and test RMSE (1.280). An RMSE of 1 is the benchmark set by the simulation’s noise. MARS selected 5 of the five informative predictors (X1, X2, X3, X4, X5) and 0 noise predictors. MARS approximates the smooth signal with piecewise-linear terms and can use products of those terms to capture interactions.
I use the same 80/20 split, 10 CV folds, and preprocessing as in
Exercise 6.3. Within each fold, nzv removes near-constant
predictors and knnImpute standardizes predictors and fills
missing values using five nearest neighbors from the fitting rows. The
final fits apply the preprocessing learned from the full training set
when predicting the test runs.
data("ChemicalManufacturingProcess", package = "AppliedPredictiveModeling")
chem <- ChemicalManufacturingProcess
x <- chem[, names(chem) != "Yield"]
y <- chem$Yield
set.seed(624)
train_index <- drop(createDataPartition(y, p = 0.8, list = FALSE))
train_x <- x[train_index, , drop = FALSE]
train_y <- y[train_index]
test_x <- x[-train_index, , drop = FALSE]
test_y <- y[-train_index]
folds75 <- createFolds(train_y, k = 10, returnTrain = TRUE)
cv75 <- trainControl(method = "cv", index = folds75)
prep75 <- c("nzv", "knnImpute")
I trained four nonlinear models and refitted the PLS and elastic net models from Exercise 6.3. I use the lowest CV RMSE to select the nonlinear and linear models for the predictor comparison. RMSE is in percentage points of yield.
linear75 <- list(
PLS = list(method = "pls", tuneLength = 20),
"Elastic net" = list(
method = "enet",
tuneGrid = expand.grid(lambda = c(0.001, 0.01, 0.1),
fraction = seq(0.05, 1, length.out = 20))
)
)
nonlinear75 <- list(
KNN = list(method = "knn", tuneLength = 10),
"Averaged neural network" = list(
method = "avNNet",
tuneGrid = expand.grid(decay = c(0, 0.01, 0.1), size = c(1, 3, 5, 7, 9),
bag = FALSE),
linout = TRUE, trace = FALSE, repeats = 3, maxit = 500
),
MARS = list(method = "earth", tuneGrid = mars_grid),
"Radial SVM" = list(method = "svmRadial", tuneLength = 10)
)
models75 <- lapply(c(linear75, nonlinear75), function(settings) {
c(settings, list(preProcess = prep75))
})
fits75 <- lapply(models75, fit_model, x = train_x, y = train_y,
control = cv75, seed = 624)
scores75 <- score_models(fits75, test_x, test_y)
linear_scores <- scores75[scores75$Model %in% names(linear75), ]
nonlinear_scores <- scores75[scores75$Model %in% names(nonlinear75), ]
best_linear <- linear_scores$Model[1]
best_nonlinear <- nonlinear_scores$Model[1]
show_scores(scores75)
| Model | CV RMSE | Test RMSE | Test R² |
|---|---|---|---|
| MARS | 1.099 | 1.172 | 0.704 |
| Radial SVM | 1.135 | 1.093 | 0.778 |
| Elastic net | 1.201 | 1.281 | 0.721 |
| KNN | 1.272 | 1.402 | 0.613 |
| Averaged neural network | 1.353 | 1.393 | 0.568 |
| PLS | 1.395 | 1.202 | 0.676 |
Finding. MARS had the lowest CV RMSE (1.099). Radial SVM had the lowest test RMSE (1.093). The best linear model by CV RMSE was Elastic net (CV RMSE 1.201).
linear_fit <- fits75[[best_linear]]
nonlinear_fit <- fits75[[best_nonlinear]]
linear_error <- test_y - predict(linear_fit, newdata = test_x)
nonlinear_error <- test_y - predict(nonlinear_fit, newdata = test_x)
process_variable <- "ManufacturingProcess16"
training_range <- range(train_x[[process_variable]], na.rm = TRUE)
unusual_row <- which.min(test_x[[process_variable]])
unusual_value <- test_x[[process_variable]][unusual_row]
rmse <- function(errors) sqrt(mean(errors^2))
full_rmse <- c(rmse(linear_error), rmse(nonlinear_error))
reduced_rmse <- c(rmse(linear_error[-unusual_row]), rmse(nonlinear_error[-unusual_row]))
One test run had ManufacturingProcess16 equal to 0
(training range 4441–4775). Excluding that same run from both sets of
predictions moves the Elastic net RMSE from 1.28 to 1.30 and the MARS
RMSE from 1.17 to 1.18. The table reports scores on all 32 test
runs.
I refit the selected nonlinear model at its chosen settings in each CV fold. On the validation rows, I shuffle one predictor’s observed values at a time and measure the increase in RMSE. Each score uses five shuffles per fold. Larger increases indicate that predictions depend more on that predictor. The linear ranking uses the importance method from Exercise 6.3.
selected <- models75[[best_nonlinear]]
selected$tuneLength <- NULL
selected$tuneGrid <- nonlinear_fit$bestTune
one_fit <- trainControl(method = "none")
fold_errors <- lapply(seq_along(folds75), function(fold) {
rows <- folds75[[fold]]
valid_x <- train_x[-rows, , drop = FALSE]
valid_y <- train_y[-rows]
fit <- fit_model(selected, train_x[rows, , drop = FALSE], train_y[rows],
control = one_fit, seed = 624 + fold)
squared_error <- function(new_x) {
sum((valid_y - predict(fit, newdata = new_x))^2)
}
original <- squared_error(valid_x)
set.seed(1624 + fold)
shuffled <- vapply(names(valid_x), function(predictor) {
if (predictor %in% fit$preProcess$method$remove) return(original)
values <- valid_x[[predictor]]
measured <- which(!is.na(values))
if (length(measured) < 2) return(original)
altered_x <- valid_x
errors <- replicate(5, {
altered_x[[predictor]][measured] <- sample(values[measured])
squared_error(altered_x)
})
mean(errors)
}, numeric(1))
c(original = original, shuffled)
})
pooled_rmse <- sqrt(colSums(do.call(rbind, fold_errors)) / length(train_y))
kept <- setdiff(names(train_x), nonlinear_fit$preProcess$method$remove)
rmse_increase <- sort(pooled_rmse[kept] - pooled_rmse[["original"]],
decreasing = TRUE)
top_nonlinear <- head(rmse_increase, 10)
if (linear_fit$method == "enet") {
coefficients <- elasticnet::predict.enet(
linear_fit$finalModel, s = linear_fit$bestTune$fraction,
mode = "fraction", type = "coefficients"
)$coefficients
linear_importance <- abs(coefficients[coefficients != 0])
} else {
importance_table <- varImp(linear_fit)$importance
linear_importance <- setNames(importance_table$Overall, rownames(importance_table))
}
top_linear <- head(sort(linear_importance, decreasing = TRUE), 10)
ranks <- seq_len(max(length(top_nonlinear), length(top_linear)))
comparison <- data.frame(
Rank = ranks,
Nonlinear = names(top_nonlinear)[ranks],
Increase = unname(top_nonlinear[ranks]),
Linear = names(top_linear)[ranks]
)
knitr::kable(comparison, digits = 3, row.names = FALSE,
col.names = c("Rank", best_nonlinear, "RMSE increase", best_linear))
| Rank | MARS | RMSE increase | Elastic net |
|---|---|---|---|
| 1 | ManufacturingProcess32 | 0.943 | ManufacturingProcess32 |
| 2 | ManufacturingProcess09 | 0.447 | ManufacturingProcess09 |
| 3 | ManufacturingProcess28 | 0.219 | ManufacturingProcess34 |
| 4 | BiologicalMaterial06 | 0.084 | ManufacturingProcess17 |
| 5 | ManufacturingProcess39 | 0.084 | ManufacturingProcess06 |
| 6 | ManufacturingProcess17 | 0.070 | BiologicalMaterial06 |
| 7 | ManufacturingProcess24 | 0.062 | ManufacturingProcess44 |
| 8 | ManufacturingProcess05 | 0.040 | ManufacturingProcess36 |
| 9 | BiologicalMaterial09 | 0.040 | ManufacturingProcess37 |
| 10 | BiologicalMaterial12 | 0.031 | ManufacturingProcess11 |
count_biological <- function(predictors) sum(startsWith(predictors, "Biological"))
bio_nonlinear <- count_biological(names(top_nonlinear))
bio_linear <- count_biological(names(top_linear))
shared <- intersect(names(top_nonlinear), names(top_linear))
unique_nonlinear <- setdiff(names(top_nonlinear), names(top_linear))
Finding. Shuffling ManufacturingProcess32 and ManufacturingProcess09 produced the two largest increases in prediction error. The MARS list includes 7 process and 3 biological predictors. The Elastic net list includes 9 process and 1 biological predictors. The lists share 4 (ManufacturingProcess32, ManufacturingProcess09, BiologicalMaterial06, ManufacturingProcess17). Process measurements account for 45 of the 57 original predictors.
Each plot shows observed training runs for a predictor unique to the nonlinear top ten. The dashed line is a straight-line fit. Blue diamonds mark the median predictor value and median yield in the low, middle, and high groups, where those groups contain observations. The cutoffs are the 33rd and 67th percentiles of the observed predictor values.
if (length(unique_nonlinear)) {
plot_data <- do.call(rbind, lapply(unique_nonlinear, function(predictor) {
measured <- !is.na(train_x[[predictor]])
data.frame(Predictor = predictor, Value = train_x[[predictor]][measured],
Yield = train_y[measured])
}))
plot_data$Predictor <- factor(plot_data$Predictor, levels = unique_nonlinear)
panels <- split(plot_data, plot_data$Predictor)
grouped <- lapply(panels, function(panel) {
cuts <- quantile(panel$Value, c(1 / 3, 2 / 3))
panel$Band <- 1L + (panel$Value > cuts[1]) + (panel$Value > cuts[2])
panel
})
band_medians <- aggregate(cbind(Value, Yield) ~ Predictor + Band,
data = do.call(rbind, grouped), FUN = median)
ggplot(plot_data, aes(Value, Yield)) +
geom_point(alpha = 0.45) +
geom_smooth(method = "lm", formula = y ~ x, se = FALSE,
linetype = "dashed", colour = "grey40") +
geom_point(data = band_medians, shape = 18, size = 3, colour = "blue") +
facet_wrap(~ Predictor, scales = "free_x") +
labs(x = "Observed predictor value", y = "Yield (%)") +
theme_minimal()
}
# Explain what the plots reveal about yield for Exercise 7.5(c).
# results = "asis" lets knitr render the text from cat() as Markdown.
if (length(unique_nonlinear)) {
# Spearman compares the ranks of the observed predictor values and yields.
correlations <- vapply(panels, function(panel) {
cor(panel$Value, panel$Yield, method = "spearman")
}, numeric(1))
types <- ifelse(startsWith(names(correlations), "Biological"),
"biological", "process")
cat("**Findings.**\n\n")
for (group in sort(unique(types))) {
group_cor <- correlations[types == group]
# Pick the strongest association in either direction for this predictor type.
predictor <- names(group_cor)[which.max(abs(group_cor))]
# Reuse the yield medians marked by the blue diamonds in the plots.
medians <- band_medians[band_medians$Predictor == predictor, ]
names_of_bands <- c("low", "middle", "high")[medians$Band]
values <- sprintf("%.2f%% (%s)", medians$Yield, names_of_bands)
finding <- paste("- **%s** has the strongest rank association among the %s",
"predictors shown (Spearman %.2f). Median yield by group was %s.\n")
cat(sprintf(finding, predictor, group, correlations[[predictor]],
paste(values, collapse = ", ")))
}
if ("ManufacturingProcess28" %in% unique_nonlinear) {
cat("\nManufacturingProcess28 has clusters near 0 and 10.",
"Median yield rises most in the high group.",
"The gap between clusters leaves part of the predictor's range unobserved.\n")
}
} else {
cat("**Finding.** The nonlinear top ten contained no predictors unique to that list.\n")
}
Findings.
ManufacturingProcess28 has clusters near 0 and 10. Median yield rises most in the high group. The gap between clusters leaves part of the predictor’s range unobserved.