The goal of this problem is to compare the performance of several nonlinear predictive models including K-Nearest Neighbors (KNN), Multivariate Adaptive Regression Splines (MARS), Sopport vector machines (SVM) and Neural Networks (NN) using the Friedman (1991) simulated data set.
Using the Friedmann non-linear equation below,we are going to answer the 2 following questions:
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ kernlab::alpha() masks ggplot2::alpha()
✖ purrr::cross() masks kernlab::cross()
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
✖ purrr::lift() masks caret::lift()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Let’s generate the training Data
set.seed(200)trainingData <-mlbench.friedman1(200, sd=1)# Let's convert the 'x' data from a matrix to a data frame and we will call it trainXtrainX <-data.frame(trainingData$x)head(trainX)
# Let's assign the unique 'y' column from the trainingData set to trainY to simplify the parameter when training our models.trainY <- trainingData$yhead(trainY)
The exploratory visualization of our data presents the following results: - X4 and X5 show clear increasing trends - X3 shows a weak U shape. - X1 and X2 show modest increasing nonlinear trends (their effect is mostly through the interaction, which a one-variable plot cannot fully show) - X6 through X10 look like flat clouds of noise.
Let’s generate test Data
testData <-mlbench.friedman1(5000,sd =1)# Let's convert the 'x' data from a matrix to a data frame and we will call it testXtestX <-data.frame(testData$x)head(testX)
Tuning the K-Nearest Neighbors (KNN), Multivariate Adaptive Regression Splines (MARS), Support vector machines (SVM) and Neural Networks (nnet) models for comparison
In order to compare and find the best model out of the four models, we will conduct the following steps:
a) Resampling and Pre-processing:
we will define a 10-fold cross-validation and reused for every model so each model sees exactly the same folds. Then,centering and scaling of predictors for KNN, SVM and neural networks, because these methods depend on distances or weighted sums and are sensitive to predictor scale. MARS regression do not require it.
b) Selection criterion
The model with the smallest RMSE and largest R² is considered the best predictor of the Friedman data.
k-Nearest Neighbors
200 samples
10 predictor
Pre-processing: centered (10), scaled (10)
Resampling: Cross-Validated (10 fold, repeated 5 times)
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
Resampling results across tuning parameters:
k RMSE Rsquared MAE
5 3.199670 0.5910371 2.649336
7 3.108444 0.6299684 2.537062
9 3.090673 0.6553105 2.506983
11 3.070640 0.6798579 2.489940
13 3.064958 0.6966689 2.467551
15 3.082000 0.7111449 2.491659
17 3.102544 0.7142288 2.506028
19 3.115994 0.7231515 2.517837
21 3.148223 0.7237919 2.560250
23 3.177131 0.7236183 2.589908
RMSE was used to select the optimal model using the smallest value.
The final value used for the model was k = 13.
The KNN model results shows that the optimal number of latent variables values with the least RMSE is equal to 13. The corresponding resampled estimate of R^2 is 0.6966689.
b) Multivariate Adaptive Regression Splines (MARS) Model
The MARS model using the nominal forward pass and pruning step can be called simply. To set the interval for nprune , the lower bound should be 2 because it guarantees at least one linear term or one side of a hinge pair enters the model. For the upper bound, with 10 predictors in trainX, earth generates at most 2(10) + 1 = 21 to 25 forward terms by default. Setting nprune much higher than 25 without explicitly increasing the ceiling of earth would simply duplicate identical models at the top of the grid.
The MARS model results shows that the optimal values of nprune with the least RMSE is equal to 14. The corresponding resampled estimate of R^2 is 0.9370177.
Support Vector Machines with Radial Basis Function Kernel
200 samples
10 predictor
Pre-processing: centered (10), scaled (10)
Resampling: Cross-Validated (10 fold, repeated 5 times)
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
Resampling results across tuning parameters:
C RMSE Rsquared MAE
0.25 2.507435 0.7919773 1.998356
0.50 2.245299 0.8067910 1.779072
1.00 2.080386 0.8263704 1.642411
2.00 1.972909 0.8415154 1.536535
4.00 1.903618 0.8515272 1.487372
8.00 1.867645 0.8569921 1.476883
16.00 1.869578 0.8564776 1.482335
32.00 1.869486 0.8564854 1.482160
64.00 1.869486 0.8564854 1.482160
128.00 1.869486 0.8564854 1.482160
256.00 1.869486 0.8564854 1.482160
512.00 1.869486 0.8564854 1.482160
1024.00 1.869486 0.8564854 1.482160
2048.00 1.869486 0.8564854 1.482160
4096.00 1.869486 0.8564854 1.482160
Tuning parameter 'sigma' was held constant at a value of 0.06299324
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were sigma = 0.06299324 and C = 8.
The SVM model results shows that the final values used for the model were sigma = 0.06299324 and C = 8. The corresponding resampled estimate of R^2 is 0.8569921.
d) Neural Networks model
# Let's create a specific candidate set of models nnetGrid <-expand.grid(size =c(1, 3, 5, 7, 9), # number of hidden decay =c(0, 0.01, 0.1), # weight decay penalty range between 0 and 0.1bag =FALSE)# Let's use the model-averaged Neural Networks set.seed(200)nnetModel <-train(x = trainX,y = trainY,method ="avNNet",preProc =c("center", "scale"),tuneGrid = nnetGrid,trControl = ctrl,linout =TRUE,trace =FALSE,maxit =500)
Warning: executing %dopar% sequentially: no parallel backend registered
nnetModel
Model Averaged Neural Network
200 samples
10 predictor
Pre-processing: centered (10), scaled (10)
Resampling: Cross-Validated (10 fold, repeated 5 times)
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
Resampling results across tuning parameters:
size decay RMSE Rsquared MAE
1 0.00 2.429088 0.7619813 1.912979
1 0.01 2.405182 0.7656206 1.888224
1 0.10 2.414289 0.7634389 1.893089
3 0.00 2.117307 0.8161658 1.667941
3 0.01 2.115405 0.8160233 1.675606
3 0.10 2.130880 0.8139318 1.678982
5 0.00 2.531791 0.7702277 1.824101
5 0.01 2.085032 0.8226430 1.640935
5 0.10 2.044786 0.8301496 1.630167
7 0.00 4.124188 0.6079310 2.654323
7 0.01 2.383503 0.7718486 1.889833
7 0.10 2.212877 0.8036464 1.747985
9 0.00 6.029783 0.4685330 3.372273
9 0.01 2.391078 0.7763434 1.897638
9 0.10 2.322163 0.7842253 1.838205
Tuning parameter 'bag' was held constant at a value of FALSE
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were size = 5, decay = 0.1 and bag = FALSE.
Let’s compare the models performance
We will evaluate all four tuned models on the 5,000-sample test set and produce the result performance by hierarchy.
The model with the smallest RMSE and largest R² is considered the best predictor of the Friedman data. As we see in our result, the MARS model is the best predictor of the Friedman data. It was predictable because the Friedman function is a sum of simple pieces (linear, quadratic) plus a single two-way interaction, which is exactly the kind of structure that hinge functions with degree = 2 can represent.Also, MARS drops irrelevant predictors, so the five noise variables do not affect it.
Does MARS select the informative predictors (those named X1–X5)?
To answer this question, we will inspect MARS Selected Terms and Variable Importance.
summary(marsModel$finalModel)
Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
nprune=14)
coefficients
(Intercept) 21.905319
h(0.621722-X1) -15.726181
h(X1-0.621722) 9.234027
h(0.601063-X2) -18.253527
h(X2-0.601063) 10.448545
h(0.447442-X3) 9.700589
h(X3-0.606015) 12.674694
h(0.734892-X4) -9.863526
h(X4-0.734892) 10.297964
h(0.850094-X5) -5.324175
h(0.218266-X1) * h(X2-0.601063) -55.358726
h(X1-0.218266) * h(X2-0.601063) -29.100250
h(X1-0.621722) * h(X2-0.295997) -26.833129
h(0.649253-X1) * h(0.601063-X2) 27.120721
Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
Termination condition: Reached nk 21
Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
Number of terms at each degree of interaction: 1 9 4
GCV 1.62945 RSS 225.8601 GRSq 0.9338437 RSq 0.953688
The result confirms that MARS model selects exclusively the five informative predictors (X1, X2, X3, X4, X5) and rejects 100% of the noise variables (X6–X10). Indeed, The importance ranking is X1 > X4 > X2 > X5 > X3. The interaction pair X1&X2 and the strongest linear term X4 dominate, which matches the coefficient sizes in the true equation. Moreover, the fitted effect shapes also agree with the truth: linear in X4 and X5, U-shaped in X3, and an interaction surface for X1 and X2. MARS therefore found both which predictors matter and, approximately, how they matter.
Problem 7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
# Data PreProcessing : Let's Apply Box-Cox, Centering, Scaling, and KNN Imputation on training datapreProcValues <-preProcess( trainX_clean,method =c("BoxCox", "center", "scale", "bagImpute"))trainX_trans <-predict(preProcValues, trainX_clean)testX_trans <-predict(preProcValues, testX_clean)# Inspect where the non-finite values are in testX_transnon_finite_mask <-!is.finite(as.matrix(testX_trans))cat("Total non-finite cells in testX_trans:", sum(non_finite_mask), "\n")
Total non-finite cells in testX_trans: 1
print(which(non_finite_mask, arr.ind =TRUE))
row col
58 11 54
# Replace any NA, NaN, or Inf in BOTH trainX_trans and testX_trans# using the corresponding training column medianfor (col innames(trainX_trans)) { tr_bad <-!is.finite(trainX_trans[[col]])if (any(tr_bad)) { trainX_trans[[col]][tr_bad] <-median(trainX_trans[[col]][!tr_bad], na.rm =TRUE) } te_bad <-!is.finite(testX_trans[[col]])if (any(te_bad)) { testX_trans[[col]][te_bad] <-median(trainX_trans[[col]], na.rm =TRUE) }}# Create a collinearity-filtered subset (cutoff = 0.75) for Neural Networks. The cuttoff percent value is recommended in the book.tooHigh <-findCorrelation(cor(trainX_trans),cutoff =0.75)length(tooHigh) # Predictors removed for Neural Network (|r| > 0.75)
[1] 20
trainX_nnet <- trainX_trans[, -tooHigh,drop =FALSE]testX_nnet <- testX_trans[, -tooHigh, drop =FALSE]
Resampling and Benchmark the PLS for later
# Let's tune a PLS model using the 10-fold cross validationctrl <-trainControl(method ="repeatedcv", number =10, repeats =5)# Tuning Pls modelset.seed(100)plsTune <-train(x = trainX_trans,y = trainY,method ="pls",tuneGrid =expand.grid(ncomp =1:20),trControl = ctrl)plsTune
Partial Least Squares
144 samples
56 predictor
No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times)
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ...
Resampling results across tuning parameters:
ncomp RMSE Rsquared MAE
1 1.383853 0.4566568 1.1235914
2 1.194807 0.5770138 0.9666330
3 1.136956 0.6284003 0.9426221
4 1.121200 0.6448438 0.9281977
5 1.119484 0.6480625 0.9126347
6 1.124826 0.6474776 0.9153996
7 1.127778 0.6467972 0.9238947
8 1.141770 0.6453254 0.9424724
9 1.174216 0.6360110 0.9684752
10 1.223800 0.6204575 0.9961525
11 1.273874 0.6066813 1.0226059
12 1.327426 0.5924425 1.0473229
13 1.380056 0.5850754 1.0663420
14 1.437527 0.5772966 1.0848342
15 1.492608 0.5719447 1.1034697
16 1.521274 0.5690166 1.1112816
17 1.546510 0.5654116 1.1188745
18 1.574607 0.5658335 1.1246613
19 1.612056 0.5630648 1.1335052
20 1.690166 0.5534257 1.1579944
RMSE was used to select the optimal model using the smallest value.
The final value used for the model was ncomp = 5.
plot(marsTune, main ="MARS Resampling Profile across Degree and Retained Terms")
# Support Vector Machine model with Radial Basis Kernel (svmR)set.seed(100)svmRTune <-train(x = trainX_trans,y = trainY,method ="svmRadial",tuneLength =14,trControl = ctrl)svmRTune
Support Vector Machines with Radial Basis Function Kernel
144 samples
56 predictor
No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times)
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ...
Resampling results across tuning parameters:
C RMSE Rsquared MAE
0.25 1.378190 0.5223701 1.1110099
0.50 1.263961 0.5698465 1.0193896
1.00 1.167992 0.6199923 0.9396418
2.00 1.138538 0.6322834 0.9220901
4.00 1.109223 0.6462412 0.8996210
8.00 1.096357 0.6548280 0.8895809
16.00 1.096357 0.6548280 0.8895809
32.00 1.096357 0.6548280 0.8895809
64.00 1.096357 0.6548280 0.8895809
128.00 1.096357 0.6548280 0.8895809
256.00 1.096357 0.6548280 0.8895809
512.00 1.096357 0.6548280 0.8895809
1024.00 1.096357 0.6548280 0.8895809
2048.00 1.096357 0.6548280 0.8895809
Tuning parameter 'sigma' was held constant at a value of 0.01691486
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were sigma = 0.01691486 and C = 8.
# Test set predictions and evaluationpred_svm <-predict(svmRTune, newdata = testX_trans)pred_nnet <-predict(nnetTune, newdata = testX_nnet)pred_mars <-predict(marsTune, newdata = testX_trans)pred_knn <-predict(knnTune, newdata = testX_trans)pred_pls <-predict(plsTune, newdata = testX_trans)testResults <-rbind(SVM_Radial =postResample(pred = pred_svm, obs = testY),Neural_Net =postResample(pred = pred_nnet, obs = testY),MARS =postResample(pred = pred_mars, obs = testY),KNN =postResample(pred = pred_knn, obs = testY),PLS_Linear =postResample(pred = pred_pls, obs = testY))# Combine CV means with Test set metricscvMeans <-summary(resampList)$statistics$RMSE[, "Mean"]cvR2 <-summary(resampList)$statistics$Rsquared[, "Mean"]perfTable <-data.frame(Model =rownames(testResults),CV_RMSE =round(cvMeans[rownames(testResults)], 4),CV_R2 =round(cvR2[rownames(testResults)], 4),Test_RMSE =round(testResults[, "RMSE"], 4),Test_R2 =round(testResults[, "Rsquared"], 4),Test_MAE =round(testResults[, "MAE"], 4))rownames(perfTable) <-NULLknitr::kable(perfTable, caption ="Resampling and Held-Out Test Set Performance")
Resampling and Held-Out Test Set Performance
Model
CV_RMSE
CV_R2
Test_RMSE
Test_R2
Test_MAE
SVM_Radial
1.0964
0.6548
1.2086
0.5666
1.0029
Neural_Net
1.3746
0.4925
1.7433
0.2049
1.4235
MARS
1.2394
0.5638
1.2232
0.5734
0.9757
KNN
1.2339
0.5613
1.4520
0.3700
1.1376
PLS_Linear
1.1195
0.6481
1.3858
0.4780
1.1987
b) Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
Variable Importance analysis & Comparison with PLS Linear Model
We are going to compute variable importance scores for the optimal nonlinear model (svmRadial which uses a LOESS smoother R^2 measure for model-independent regression importance when internal weights are nonlinear) and compare its top 10 predictors against the top 10 predictors of the optimal linear model (plsTune, which uses weighted sums of absolute regression coefficients / VIP scores).
# Let's check the variable importance for Optimal Nonlinear Model (SVM)svmImp <-varImp(svmRTune, scale =TRUE)plsImp <-varImp(plsTune, scale =TRUE)plot(svmImp, top =15, main ="Top 15 Predictors: Optimal Nonlinear Model (SVM Radial)")
plot(plsImp, top =15, main ="Top 15 Predictors: Optimal Linear Model (PLS)")
# Extract Top 10 predictor names and importance scoresgetTopN <-function(impObj, n =10) { df <- impObj$importance df$Predictor <-rownames(df) df <- df[order(-df$Overall), ]head(df[, c("Predictor", "Overall")], n)}top10_svm <-getTopN(svmImp, 10)top10_pls <-getTopN(plsImp, 10)compTop10 <-data.frame(Rank =1:10,SVM_Predictor = top10_svm$Predictor,SVM_Score =round(top10_svm$Overall, 2),PLS_Predictor = top10_pls$Predictor,PLS_Score =round(top10_pls$Overall, 2))knitr::kable(compTop10, caption ="Top 10 Most Important Predictors: SVM Radial vs. PLS")
Top 10 Most Important Predictors: SVM Radial vs. PLS
Rank
SVM_Predictor
SVM_Score
PLS_Predictor
PLS_Score
1
ManufacturingProcess32
100.00
ManufacturingProcess32
100.00
2
BiologicalMaterial06
95.00
ManufacturingProcess17
90.70
3
BiologicalMaterial03
83.55
ManufacturingProcess13
89.73
4
ManufacturingProcess36
83.52
ManufacturingProcess09
88.86
5
ManufacturingProcess13
83.25
ManufacturingProcess36
86.16
6
BiologicalMaterial02
78.48
ManufacturingProcess06
74.68
7
ManufacturingProcess17
78.25
ManufacturingProcess33
63.68
8
ManufacturingProcess31
77.21
BiologicalMaterial03
62.08
9
ManufacturingProcess09
75.13
BiologicalMaterial06
61.84
10
BiologicalMaterial12
71.85
BiologicalMaterial08
60.35
# Let's categorize Biological vs. Process predictors in Top 10svm_bio <-sum(grepl("^BiologicalMaterial", top10_svm$Predictor))svm_proc <-sum(grepl("^ManufacturingProcess", top10_svm$Predictor))pls_bio <-sum(grepl("^BiologicalMaterial", top10_pls$Predictor))pls_proc <-sum(grepl("^ManufacturingProcess", top10_pls$Predictor))cat(sprintf("SVM Top 10 Composition: %d Process, %d Biological\\n", svm_proc, svm_bio))
SVM Top 10 Composition: 6 Process, 4 Biological\n
cat(sprintf("PLS Top 10 Composition: %d Process, %d Biological\\n", pls_proc, pls_bio))
cat("Unique to SVM Top-10: ", paste(uniqueSVM, collapse =", "), "\\n")
Unique to SVM Top-10: BiologicalMaterial02, ManufacturingProcess31, BiologicalMaterial12 \n
cat("Unique to PLS Top-10: ", paste(uniquePLS, collapse =", "), "\\n")
Unique to PLS Top-10: \n
Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
To answer this question, we examine scatter plots with LOESS smoothers relating yield to the predictors that appear in the top 10 of svmRadial but not in the top 10 of plsTune. We plot these both on the pre-processed scale and on the original imputed scale so physical process interpretations remain transparent.
# Plot relationships for predictors unique to SVM Top 10if (length(uniqueSVM) >0) { plotData <-data.frame(Yield = trainY, trainX_trans[, uniqueSVM, drop =FALSE] ) plotDataLong <- reshape2::melt(plotData, id.vars ="Yield",variable.name ="Predictor",value.name ="Standardized_Value")ggplot(plotDataLong, aes(x = Standardized_Value, y = Yield)) +geom_point(alpha =0.7, color ="#2c3e50") +geom_smooth(method ="loess", se =TRUE, color ="#e74c3c", fill ="#f5b7b1") +geom_smooth(method ="lm", se =FALSE, linetype ="dashed", color ="#2980b9") +facet_wrap(~ Predictor, scales ="free_x", ncol =2) +labs(title ="Nonlinear vs. Linear Trends for Predictors Unique to Optimal SVM Top 10",subtitle ="Solid red curve: LOESS smoother | Dashed blue line: Ordinary linear fit",x ="Pre-processed Predictor Value (Standardized)",y ="Manufacturing Yield (%)" ) +theme_bw(base_size =12)}
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
We also inspect partial dependence / model response profiles from the fitted svmR model across these unique predictors to confirm how the SVM exploits non-monotonic, threshold, or saturation effects.
Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.
Conclusion
Bivariate LOESS smoothers and SVM partial response profiles demonstrate that predictors unique to the SVM top 10 exhibit non-monotonic curvature, diminishing returns (plateaus), or threshold regimes with respect to yield—patterns that linear projections in PLS average out, explaining why kernel-based nonlinear regression captures substantial additional predictive signal.