DATA 624 - Homework 8

Author

Pascal Hermann Kouogang Tafo

Problem 7.2

The goal of this problem is to compare the performance of several nonlinear predictive models including K-Nearest Neighbors (KNN), Multivariate Adaptive Regression Splines (MARS), Sopport vector machines (SVM) and Neural Networks (NN) using the Friedman (1991) simulated data set.

Using the Friedmann non-linear equation below,we are going to answer the 2 following questions:

y = 10 sin(πx1x2) + 20(x3 − 0.5)2 + 10x4 + 5x5 + N(0, σ2)

where the x values are random variables uniformly distributed between [0, 1]

  1. Which models appear to give the best performance?
  2. Does MARS select the informative predictors (those named X1–X5)?

Let’s install and load the necessary library

We will install the package in the R console so that it is not being installed every time that we run our code.

library(mlbench)
Warning: package 'mlbench' was built under R version 4.5.3
library(caret)
Warning: package 'caret' was built under R version 4.5.3
Loading required package: ggplot2
Warning: package 'ggplot2' was built under R version 4.5.3
Loading required package: lattice
Warning: package 'lattice' was built under R version 4.5.2
library(earth)
Warning: package 'earth' was built under R version 4.5.3
Loading required package: Formula
Warning: package 'Formula' was built under R version 4.5.3
Loading required package: plotmo
Warning: package 'plotmo' was built under R version 4.5.3
Loading required package: plotrix
Warning: package 'plotrix' was built under R version 4.5.2
library(kernlab)
Warning: package 'kernlab' was built under R version 4.5.2

Attaching package: 'kernlab'
The following object is masked from 'package:ggplot2':

    alpha
library(nnet)
library(tidyverse)
Warning: package 'tidyverse' was built under R version 4.5.3
Warning: package 'tibble' was built under R version 4.5.2
Warning: package 'tidyr' was built under R version 4.5.2
Warning: package 'readr' was built under R version 4.5.2
Warning: package 'purrr' was built under R version 4.5.3
Warning: package 'dplyr' was built under R version 4.5.3
Warning: package 'stringr' was built under R version 4.5.2
Warning: package 'forcats' was built under R version 4.5.2
Warning: package 'lubridate' was built under R version 4.5.2
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ lubridate 1.9.5     ✔ tibble    3.3.1
✔ purrr     1.2.2     ✔ tidyr     1.3.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ kernlab::alpha() masks ggplot2::alpha()
✖ purrr::cross()   masks kernlab::cross()
✖ dplyr::filter()  masks stats::filter()
✖ dplyr::lag()     masks stats::lag()
✖ purrr::lift()    masks caret::lift()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

Let’s generate the training Data

set.seed(200)

trainingData <- mlbench.friedman1(200, sd=1)

# Let's convert the 'x' data from a matrix to a data frame and we will call it trainX

trainX <- data.frame(trainingData$x)
head(trainX)
         X1        X2         X3         X4         X5         X6        X7
1 0.5337724 0.6478064 0.85078526 0.18159957 0.92903976 0.36179060 0.8266609
2 0.5837650 0.4381528 0.67272659 0.66924914 0.16379784 0.45305931 0.6489601
3 0.5895783 0.5879065 0.40967108 0.33812728 0.89409334 0.02681911 0.1785614
4 0.6910399 0.2259548 0.03335447 0.06691274 0.63744519 0.52500637 0.5133614
5 0.6673315 0.8188985 0.71676079 0.80324287 0.08306864 0.22344157 0.6644906
6 0.8392937 0.3862983 0.64618857 0.86105431 0.63038947 0.43703891 0.3360117
         X8         X9       X10
1 0.4214081 0.59111440 0.5886216
2 0.8446239 0.92819306 0.7584008
3 0.3495908 0.01759542 0.4441185
4 0.7970260 0.68986918 0.4450716
5 0.9038919 0.39696995 0.5500808
6 0.6489177 0.53116033 0.9066182
# Let's assign the unique 'y' column from the trainingData set to trainY to simplify the parameter when training our models.

trainY <- trainingData$y
head(trainY)
[1] 18.46398 16.09836 17.76165 13.78730 18.42984 20.85817

The One reason to convert the ‘x’ data from a matrix to a data frame is that it will give the columns names which are called X1, X2, …, X10.

Exploratory look at the data

We will look at the data using the featurePlot function.

featurePlot(
  x = trainingData$x,
  y = trainingData$y,
  between = list(x = 1, y = 1),
  type = c("g", "p", "smooth"),
  labels = c("Predictor Value", "Response (y)"),
  par.settings = list(
    plot.line = list(col = "red", lwd = 2)
  )
)

Results

The exploratory visualization of our data presents the following results: - X4 and X5 show clear increasing trends - X3 shows a weak U shape. - X1 and X2 show modest increasing nonlinear trends (their effect is mostly through the interaction, which a one-variable plot cannot fully show) - X6 through X10 look like flat clouds of noise.

Let’s generate test Data

testData <- mlbench.friedman1(
  5000,
  sd = 1
)

# Let's convert the 'x' data from a matrix to a data frame and we will call it testX

testX <- data.frame(testData$x)
head(testX)
          X1        X2        X3          X4        X5         X6        X7
1 0.49578735 0.2611653 0.8098616 0.823182309 0.8217010 0.32193361 0.0544078
2 0.40777740 0.7163803 0.9635523 0.505653024 0.8802850 0.57453040 0.4551802
3 0.49910226 0.7146126 0.6810968 0.003843726 0.4983548 0.06028523 0.8925606
4 0.19563672 0.3689804 0.3779541 0.385691399 0.2787616 0.55466124 0.3972387
5 0.02280348 0.7460249 0.3907981 0.873982576 0.1967682 0.17619075 0.5066592
6 0.16124385 0.9289150 0.1927960 0.498280346 0.3071502 0.70104601 0.7272552
         X8         X9          X10
1 0.5185030 0.39143353 0.7389365041
2 0.9806052 0.66632676 0.0005895968
3 0.9750831 0.08564886 0.5922129985
4 0.8403877 0.09039455 0.1622669764
5 0.5561729 0.37898130 0.6500855859
6 0.4935052 0.18074734 0.1574340949
testY <- testData$y
head(testY)
[1] 17.518866 20.868802 12.817534  5.094464 10.793634 12.209017

Tuning the K-Nearest Neighbors (KNN), Multivariate Adaptive Regression Splines (MARS), Support vector machines (SVM) and Neural Networks (nnet) models for comparison

In order to compare and find the best model out of the four models, we will conduct the following steps:

a) Resampling and Pre-processing:

we will define a 10-fold cross-validation and reused for every model so each model sees exactly the same folds. Then,centering and scaling of predictors for KNN, SVM and neural networks, because these methods depend on distances or weighted sums and are sensitive to predictor scale. MARS regression do not require it.

b) Selection criterion

The model with the smallest RMSE and largest R² is considered the best predictor of the Friedman data.

# train control

ctrl <- trainControl(
  method = "repeatedcv",
  number = 10,
  repeats = 5
)

a) K-Nearest Neighbors model

set.seed(200)

knn_modelfit <- train(
  x = trainX,
  y = trainY,
  method = "knn",
  tuneLength = 10,
  preProcess = c("center","scale"),
  trControl = ctrl
)

knn_modelfit
k-Nearest Neighbors 

200 samples
 10 predictor

Pre-processing: centered (10), scaled (10) 
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
Resampling results across tuning parameters:

  k   RMSE      Rsquared   MAE     
   5  3.199670  0.5910371  2.649336
   7  3.108444  0.6299684  2.537062
   9  3.090673  0.6553105  2.506983
  11  3.070640  0.6798579  2.489940
  13  3.064958  0.6966689  2.467551
  15  3.082000  0.7111449  2.491659
  17  3.102544  0.7142288  2.506028
  19  3.115994  0.7231515  2.517837
  21  3.148223  0.7237919  2.560250
  23  3.177131  0.7236183  2.589908

RMSE was used to select the optimal model using the smallest value.
The final value used for the model was k = 13.

The KNN model results shows that the optimal number of latent variables values with the least RMSE is equal to 13. The corresponding resampled estimate of R^2 is 0.6966689.

b) Multivariate Adaptive Regression Splines (MARS) Model

The MARS model using the nominal forward pass and pruning step can be called simply. To set the interval for nprune , the lower bound should be 2 because it guarantees at least one linear term or one side of a hinge pair enters the model. For the upper bound, with 10 predictors in trainX, earth generates at most 2(10) + 1 = 21 to 25 forward terms by default. Setting nprune much higher than 25 without explicitly increasing the ceiling of earth would simply duplicate identical models at the top of the grid.

marsGrid <- expand.grid(degree = 1:2, nprune = 2:25)

set.seed(200)

marsModel <- train(
  x = trainX,
  y = trainY,
  method = "earth",
  tuneGrid = marsGrid,
  trControl = ctrl
)

marsModel
Multivariate Adaptive Regression Spline 

200 samples
 10 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
Resampling results across tuning parameters:

  degree  nprune  RMSE      Rsquared   MAE      
  1        2      4.386168  0.2412407  3.6346127
  1        3      3.621483  0.4742386  2.9231886
  1        4      2.595717  0.7297442  2.0796496
  1        5      2.307259  0.7837985  1.8608089
  1        6      2.266285  0.7932822  1.8080255
  1        7      1.754388  0.8732428  1.3866966
  1        8      1.704233  0.8796386  1.3397570
  1        9      1.675474  0.8839704  1.3197799
  1       10      1.676810  0.8839600  1.3176441
  1       11      1.632768  0.8899677  1.2806226
  1       12      1.608233  0.8933035  1.2612971
  1       13      1.613222  0.8918455  1.2611245
  1       14      1.617342  0.8914827  1.2661040
  1       15      1.617399  0.8915040  1.2662071
  1       16      1.617399  0.8915040  1.2662071
  1       17      1.617399  0.8915040  1.2662071
  1       18      1.617399  0.8915040  1.2662071
  1       19      1.617399  0.8915040  1.2662071
  1       20      1.617399  0.8915040  1.2662071
  1       21      1.617399  0.8915040  1.2662071
  1       22      1.617399  0.8915040  1.2662071
  1       23      1.617399  0.8915040  1.2662071
  1       24      1.617399  0.8915040  1.2662071
  1       25      1.617399  0.8915040  1.2662071
  2        2      4.385499  0.2410298  3.6351721
  2        3      3.619764  0.4744779  2.9232867
  2        4      2.593957  0.7294001  2.0782930
  2        5      2.281286  0.7877162  1.8391161
  2        6      2.238092  0.7969543  1.7750315
  2        7      1.787248  0.8669925  1.4190884
  2        8      1.660475  0.8875846  1.2831281
  2        9      1.448667  0.9125300  1.1341093
  2       10      1.392833  0.9201482  1.1145233
  2       11      1.322100  0.9282665  1.0499498
  2       12      1.285224  0.9315178  1.0187759
  2       13      1.268420  0.9339584  1.0102693
  2       14      1.240975  0.9370177  0.9917534
  2       15      1.253084  0.9362805  0.9972075
  2       16      1.257347  0.9362414  1.0004956
  2       17      1.253084  0.9369115  0.9996041
  2       18      1.256789  0.9366587  1.0021983
  2       19      1.256789  0.9366587  1.0021983
  2       20      1.256789  0.9366587  1.0021983
  2       21      1.256789  0.9366587  1.0021983
  2       22      1.256789  0.9366587  1.0021983
  2       23      1.256789  0.9366587  1.0021983
  2       24      1.256789  0.9366587  1.0021983
  2       25      1.256789  0.9366587  1.0021983

RMSE was used to select the optimal model using the smallest value.
The final values used for the model were nprune = 14 and degree = 2.

The MARS model results shows that the optimal values of nprune with the least RMSE is equal to 14. The corresponding resampled estimate of R^2 is 0.9370177.

c) Support Vector Machines model

set.seed(200)

svmModel_fit <- train(
  x = trainX,
  y = trainY,
  method = "svmRadial",
  preProc = c("center", "scale"),
  tuneLength = 15,
  trControl = ctrl
)

svmModel_fit
Support Vector Machines with Radial Basis Function Kernel 

200 samples
 10 predictor

Pre-processing: centered (10), scaled (10) 
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
Resampling results across tuning parameters:

  C        RMSE      Rsquared   MAE     
     0.25  2.507435  0.7919773  1.998356
     0.50  2.245299  0.8067910  1.779072
     1.00  2.080386  0.8263704  1.642411
     2.00  1.972909  0.8415154  1.536535
     4.00  1.903618  0.8515272  1.487372
     8.00  1.867645  0.8569921  1.476883
    16.00  1.869578  0.8564776  1.482335
    32.00  1.869486  0.8564854  1.482160
    64.00  1.869486  0.8564854  1.482160
   128.00  1.869486  0.8564854  1.482160
   256.00  1.869486  0.8564854  1.482160
   512.00  1.869486  0.8564854  1.482160
  1024.00  1.869486  0.8564854  1.482160
  2048.00  1.869486  0.8564854  1.482160
  4096.00  1.869486  0.8564854  1.482160

Tuning parameter 'sigma' was held constant at a value of 0.06299324
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were sigma = 0.06299324 and C = 8.

The SVM model results shows that the final values used for the model were sigma = 0.06299324 and C = 8. The corresponding resampled estimate of R^2 is 0.8569921.

d) Neural Networks model

# Let's create a specific candidate set of models 

nnetGrid <- expand.grid(
  size = c(1, 3, 5, 7, 9), # number of hidden 
  decay = c(0, 0.01, 0.1), # weight decay penalty range between 0 and 0.1
  bag = FALSE
)


# Let's use the model-averaged Neural Networks 

set.seed(200)

nnetModel <- train(
  x = trainX,
  y = trainY,
  method = "avNNet",
  preProc = c("center", "scale"),
  tuneGrid = nnetGrid,
  trControl = ctrl,
  linout = TRUE,
  trace = FALSE,
  maxit = 500
)
Warning: executing %dopar% sequentially: no parallel backend registered
nnetModel
Model Averaged Neural Network 

200 samples
 10 predictor

Pre-processing: centered (10), scaled (10) 
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
Resampling results across tuning parameters:

  size  decay  RMSE      Rsquared   MAE     
  1     0.00   2.429088  0.7619813  1.912979
  1     0.01   2.405182  0.7656206  1.888224
  1     0.10   2.414289  0.7634389  1.893089
  3     0.00   2.117307  0.8161658  1.667941
  3     0.01   2.115405  0.8160233  1.675606
  3     0.10   2.130880  0.8139318  1.678982
  5     0.00   2.531791  0.7702277  1.824101
  5     0.01   2.085032  0.8226430  1.640935
  5     0.10   2.044786  0.8301496  1.630167
  7     0.00   4.124188  0.6079310  2.654323
  7     0.01   2.383503  0.7718486  1.889833
  7     0.10   2.212877  0.8036464  1.747985
  9     0.00   6.029783  0.4685330  3.372273
  9     0.01   2.391078  0.7763434  1.897638
  9     0.10   2.322163  0.7842253  1.838205

Tuning parameter 'bag' was held constant at a value of FALSE
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were size = 5, decay = 0.1 and bag = FALSE.

Let’s compare the models performance

We will evaluate all four tuned models on the 5,000-sample test set and produce the result performance by hierarchy.

# Paired Resampling models Comparison

resamp <- resamples(list(
  KNN = knn_modelfit,
  MARS = marsModel,
  SVM = svmModel_fit,
  avNNet = nnetModel
))
summary(resamp)

Call:
summary.resamples(object = resamp)

Models: KNN, MARS, SVM, avNNet 
Number of resamples: 50 

MAE 
            Min.   1st Qu.    Median      Mean  3rd Qu.     Max. NA's
KNN    1.9277052 2.1860456 2.4468271 2.4675514 2.741798 3.294739    0
MARS   0.7326193 0.8436522 0.9704735 0.9917534 1.099570 1.395006    0
SVM    0.9476948 1.3222393 1.4473840 1.4768829 1.570700 2.180809    0
avNNet 0.9723454 1.3882895 1.5905612 1.6301673 1.828009 2.458635    0

RMSE 
            Min.  1st Qu.   Median     Mean  3rd Qu.     Max. NA's
KNN    2.4593394 2.747399 3.041286 3.064958 3.332628 3.942898    0
MARS   0.8900136 1.068202 1.203922 1.240975 1.418318 1.691489    0
SVM    1.3259377 1.694821 1.855214 1.867645 2.039788 2.639539    0
avNNet 1.3126196 1.725635 1.998357 2.044786 2.292826 3.183467    0

Rsquared 
            Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
KNN    0.4130894 0.6098219 0.7343975 0.6966689 0.7834308 0.8742705    0
MARS   0.8759205 0.9160902 0.9403257 0.9370177 0.9612560 0.9832637    0
SVM    0.6210310 0.8389267 0.8661658 0.8569921 0.9020618 0.9269289    0
avNNet 0.5280635 0.7989454 0.8483912 0.8301496 0.8939327 0.9369857    0
# Out-of-Sample Test Set Evaluation (n = 5,000)

models <- list(KNN = knn_modelfit, MARS = marsModel, SVM = svmModel_fit, avNNet = nnetModel)

testResults <- lapply(names(models), function(m) {
  preds <- predict(models[[m]], newdata = testX)
  perf <- postResample(pred = preds, obs = testY)
  data.frame(
    Model = m,
    CV_RMSE = min(models[[m]]$results$RMSE),
    Test_RMSE = perf["RMSE"],
    Test_Rsquared = perf["Rsquared"],
    row.names = NULL
  )
}) |> bind_rows() |> arrange(Test_RMSE)
print(testResults)
   Model  CV_RMSE Test_RMSE Test_Rsquared
1   MARS 1.240975  1.172263     0.9448890
2    SVM 1.867645  2.054120     0.8290353
3 avNNet 2.044786  2.140368     0.8240921
4    KNN 3.064958  3.148156     0.6747755

Best Model choice

The model with the smallest RMSE and largest R² is considered the best predictor of the Friedman data. As we see in our result, the MARS model is the best predictor of the Friedman data. It was predictable because the Friedman function is a sum of simple pieces (linear, quadratic) plus a single two-way interaction, which is exactly the kind of structure that hinge functions with degree = 2 can represent.Also, MARS drops irrelevant predictors, so the five noise variables do not affect it.

Does MARS select the informative predictors (those named X1–X5)?

To answer this question, we will inspect MARS Selected Terms and Variable Importance.

summary(marsModel$finalModel)
Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
            nprune=14)

                                coefficients
(Intercept)                        21.905319
h(0.621722-X1)                    -15.726181
h(X1-0.621722)                      9.234027
h(0.601063-X2)                    -18.253527
h(X2-0.601063)                     10.448545
h(0.447442-X3)                      9.700589
h(X3-0.606015)                     12.674694
h(0.734892-X4)                     -9.863526
h(X4-0.734892)                     10.297964
h(0.850094-X5)                     -5.324175
h(0.218266-X1) * h(X2-0.601063)   -55.358726
h(X1-0.218266) * h(X2-0.601063)   -29.100250
h(X1-0.621722) * h(X2-0.295997)   -26.833129
h(0.649253-X1) * h(0.601063-X2)    27.120721

Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
Termination condition: Reached nk 21
Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
Number of terms at each degree of interaction: 1 9 4
GCV 1.62945    RSS 225.8601    GRSq 0.9338437    RSq 0.953688
evimp(marsModel$finalModel)
   nsubsets   gcv    rss
X1       13 100.0  100.0
X4       12  85.1   84.3
X2       11  69.2   68.1
X5       10  49.3   48.6
X3        9  40.0   39.4

The result confirms that MARS model selects exclusively the five informative predictors (X1, X2, X3, X4, X5) and rejects 100% of the noise variables (X6–X10). Indeed, The importance ranking is X1 > X4 > X2 > X5 > X3. The interaction pair X1&X2 and the strongest linear term X4 dominate, which matches the coefficient sizes in the true equation. Moreover, the fitted effect shapes also agree with the truth: linear in X4 and X5, U-shaped in X3, and an interaction surface for X1 and X2. MARS therefore found both which predictors matter and, approximately, how they matter.

Problem 7.5

Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.

Loading packages and data

# Loading packages

library(AppliedPredictiveModeling)
Warning: package 'AppliedPredictiveModeling' was built under R version 4.5.3
library(caret)
library(earth)
library(kernlab)
library(nnet)
library(pls)
Warning: package 'pls' was built under R version 4.5.3

Attaching package: 'pls'
The following object is masked from 'package:caret':

    R2
The following object is masked from 'package:stats':

    loadings
library(ggplot2)
library(dplyr)


# Loading data 

set.seed(123)

if ("ChemicalManufacturingProcess" %in%
    data(package = "AppliedPredictiveModeling")$results[, "Item"]) {

  data("ChemicalManufacturingProcess")
  raw_df <- ChemicalManufacturingProcess

} else {

  data("chemicalManufacturing")
  raw_df <- data.frame(Yield = yield,
                       processPredictors)
}

# Separate predictors and response
yield <- ChemicalManufacturingProcess$Yield
processPredictors <- ChemicalManufacturingProcess[, -1]

# Dimensions of predictor matrix

dim(processPredictors)
[1] 176  57
# Total missing values in predictors:

sum(is.na(processPredictors))
[1] 106

a) Which nonlinear regression model gives the optimal resampling and test set performance?

Data Splitting, Imputation, and Pre-processing

we will split our data in a 80/20 ratio, meaning 80% allocated to training and 20% to testing.

# Data splitting

set.seed(123)

trainingRows <- createDataPartition(yield, p = 0.80, list = FALSE)

trainX <- processPredictors[trainingRows, ]
trainY <- yield[trainingRows]

testX  <- processPredictors[-trainingRows, ]
testY  <- yield[-trainingRows]


# Let's identify and remove near-zero variance predictors on training set

nzvCols <- nearZeroVar(trainX)
if (length(nzvCols) > 0) {
  cat("Removing near-zero variance predictors:", names(trainX)[nzvCols], "\\n")
  trainX_clean <- trainX[, -nzvCols]
  testX_clean  <- testX[, -nzvCols]
} else {
  trainX_clean <- trainX
  testX_clean  <- testX
}
Removing near-zero variance predictors: BiologicalMaterial07 \n
# Data PreProcessing : Let's Apply Box-Cox, Centering, Scaling, and KNN Imputation on training data

preProcValues <- preProcess(
  trainX_clean,
  method = c("BoxCox", "center", "scale", "bagImpute")
)

trainX_trans <- predict(preProcValues, trainX_clean)
testX_trans  <- predict(preProcValues, testX_clean)

# Inspect where the non-finite values are in testX_trans
non_finite_mask <- !is.finite(as.matrix(testX_trans))
cat("Total non-finite cells in testX_trans:", sum(non_finite_mask), "\n")
Total non-finite cells in testX_trans: 1 
print(which(non_finite_mask, arr.ind = TRUE))
   row col
58  11  54
# Replace any NA, NaN, or Inf in BOTH trainX_trans and testX_trans
#    using the corresponding training column median

for (col in names(trainX_trans)) {
  tr_bad <- !is.finite(trainX_trans[[col]])
  if (any(tr_bad)) {
    trainX_trans[[col]][tr_bad] <- median(trainX_trans[[col]][!tr_bad], na.rm = TRUE)
  }
  
  te_bad <- !is.finite(testX_trans[[col]])
  if (any(te_bad)) {
    testX_trans[[col]][te_bad] <- median(trainX_trans[[col]], na.rm = TRUE)
  }
}

# Create a collinearity-filtered subset (cutoff = 0.75) for Neural Networks. The cuttoff percent value is recommended in the book.

tooHigh <- findCorrelation(cor(trainX_trans),cutoff = 0.75)
length(tooHigh) # Predictors removed for Neural Network (|r| > 0.75)
[1] 20
trainX_nnet <- trainX_trans[, -tooHigh,drop = FALSE]
testX_nnet  <- testX_trans[, -tooHigh, drop = FALSE]

Resampling and Benchmark the PLS for later

# Let's tune a PLS model using the 10-fold cross validation

ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)

# Tuning Pls model

set.seed(100)
plsTune <- train(
  x = trainX_trans,
  y = trainY,
  method = "pls",
  tuneGrid = expand.grid(ncomp = 1:20),
  trControl = ctrl
)

plsTune
Partial Least Squares 

144 samples
 56 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ... 
Resampling results across tuning parameters:

  ncomp  RMSE      Rsquared   MAE      
   1     1.383853  0.4566568  1.1235914
   2     1.194807  0.5770138  0.9666330
   3     1.136956  0.6284003  0.9426221
   4     1.121200  0.6448438  0.9281977
   5     1.119484  0.6480625  0.9126347
   6     1.124826  0.6474776  0.9153996
   7     1.127778  0.6467972  0.9238947
   8     1.141770  0.6453254  0.9424724
   9     1.174216  0.6360110  0.9684752
  10     1.223800  0.6204575  0.9961525
  11     1.273874  0.6066813  1.0226059
  12     1.327426  0.5924425  1.0473229
  13     1.380056  0.5850754  1.0663420
  14     1.437527  0.5772966  1.0848342
  15     1.492608  0.5719447  1.1034697
  16     1.521274  0.5690166  1.1112816
  17     1.546510  0.5654116  1.1188745
  18     1.574607  0.5658335  1.1246613
  19     1.612056  0.5630648  1.1335052
  20     1.690166  0.5534257  1.1579944

RMSE was used to select the optimal model using the smallest value.
The final value used for the model was ncomp = 5.

Training, Evaluating Nonlinear Regression Models

# Model-Averaged Neural Network (avNNet)

nnetGrid <- expand.grid(
  decay = c(0, 0.01, 0.1),
  size  = 1:10,
  bag   = FALSE
)

maxWeights <- 10 * (ncol(trainX_nnet) + 1) + 10 + 1

set.seed(100)
nnetTune <- train(
  x = trainX_nnet,
  y = trainY,
  method = "avNNet",
  tuneGrid = nnetGrid,
  trControl = ctrl,
  linout = TRUE,
  trace = FALSE,
  MaxNWts = maxWeights,
  maxit = 500
)

nnetTune
Model Averaged Neural Network 

144 samples
 36 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ... 
Resampling results across tuning parameters:

  decay  size  RMSE      Rsquared   MAE     
  0.00    1    1.526738  0.4152809  1.230972
  0.00    2    1.524991  0.4296800  1.192223
  0.00    3    2.408060  0.3193180  1.822685
  0.00    4    2.146192  0.3247072  1.701115
  0.00    5    1.866260  0.3474943  1.510495
  0.00    6    2.335402  0.2911645  1.852956
  0.00    7    2.930589  0.2851898  2.137985
  0.00    8    4.042556  0.2173599  2.660656
  0.00    9    5.208309  0.1715327  3.176146
  0.00   10    8.063246  0.1417208  4.516320
  0.01    1    1.374637  0.4925229  1.095442
  0.01    2    1.415760  0.5215775  1.115229
  0.01    3    1.887578  0.3899549  1.451547
  0.01    4    1.838757  0.4080643  1.432465
  0.01    5    1.592516  0.4677818  1.248098
  0.01    6    1.660108  0.4526514  1.315282
  0.01    7    1.638093  0.4395323  1.277457
  0.01    8    1.690179  0.4354065  1.293886
  0.01    9    1.684988  0.4413174  1.294873
  0.01   10    2.023186  0.3516872  1.528464
  0.10    1    1.419365  0.5120027  1.109588
  0.10    2    1.638400  0.4836425  1.202953
  0.10    3    1.891362  0.4147726  1.387984
  0.10    4    1.767371  0.4440536  1.326966
  0.10    5    1.773802  0.4244877  1.361354
  0.10    6    1.929125  0.4006095  1.436827
  0.10    7    1.871719  0.4235188  1.359512
  0.10    8    1.991375  0.3884432  1.443426
  0.10    9    1.906527  0.4001066  1.390245
  0.10   10    1.769225  0.4347128  1.328510

Tuning parameter 'bag' was held constant at a value of FALSE
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were size = 1, decay = 0.01 and bag = FALSE.
plot(nnetTune, main = "Model-Averaged Neural Network Resampling Profile")

# Multivariate Adaptive Regression Splines (MARS) model

marsGrid <- expand.grid(
  degree = 1:2,
  nprune = 2:38
)

set.seed(100)
marsTune <- train(
  x = trainX_trans,
  y = trainY,
  method = "earth",
  tuneGrid = marsGrid,
  trControl = ctrl
)

marsTune
Multivariate Adaptive Regression Spline 

144 samples
 56 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ... 
Resampling results across tuning parameters:

  degree  nprune  RMSE      Rsquared   MAE      
  1        2      1.402864  0.4619625  1.1087966
  1        3      1.239401  0.5637876  0.9955056
  1        4      1.262146  0.5456137  1.0043498
  1        5      1.270198  0.5449490  1.0105695
  1        6      1.285660  0.5367001  1.0302453
  1        7      1.274536  0.5511765  1.0163310
  1        8      1.278560  0.5472020  1.0169211
  1        9      1.286212  0.5450611  1.0244001
  1       10      1.339148  0.5267728  1.0702534
  1       11      1.350207  0.5316864  1.0661152
  1       12      1.368616  0.5255469  1.0870832
  1       13      1.375955  0.5251526  1.0874769
  1       14      1.731813  0.5112637  1.1802589
  1       15      1.715256  0.5098606  1.1794587
  1       16      1.816463  0.5122582  1.2014535
  1       17      1.818940  0.5110925  1.2030931
  1       18      1.819186  0.5108866  1.2033138
  1       19      1.817651  0.5119013  1.2011249
  1       20      1.817651  0.5119013  1.2011249
  1       21      1.817651  0.5119013  1.2011249
  1       22      1.817651  0.5119013  1.2011249
  1       23      1.817651  0.5119013  1.2011249
  1       24      1.817651  0.5119013  1.2011249
  1       25      1.817651  0.5119013  1.2011249
  1       26      1.817651  0.5119013  1.2011249
  1       27      1.817651  0.5119013  1.2011249
  1       28      1.817651  0.5119013  1.2011249
  1       29      1.817651  0.5119013  1.2011249
  1       30      1.817651  0.5119013  1.2011249
  1       31      1.817651  0.5119013  1.2011249
  1       32      1.817651  0.5119013  1.2011249
  1       33      1.817651  0.5119013  1.2011249
  1       34      1.817651  0.5119013  1.2011249
  1       35      1.817651  0.5119013  1.2011249
  1       36      1.817651  0.5119013  1.2011249
  1       37      1.817651  0.5119013  1.2011249
  1       38      1.817651  0.5119013  1.2011249
  2        2      1.402864  0.4619625  1.1087966
  2        3      1.278735  0.5396584  1.0139660
  2        4      1.281393  0.5380525  1.0112680
  2        5      1.335311  0.5155767  1.0560350
  2        6      1.313706  0.5299083  1.0505676
  2        7      1.443790  0.5352022  1.0965858
  2        8      1.414011  0.5494207  1.0681631
  2        9      1.406748  0.5543883  1.0588182
  2       10      1.457705  0.5254577  1.0892643
  2       11      1.506776  0.5133700  1.1062359
  2       12      1.509556  0.5091590  1.1100589
  2       13      1.534318  0.4980103  1.1234360
  2       14      1.564814  0.4958830  1.1308841
  2       15      1.612341  0.4959623  1.1668334
  2       16      1.609196  0.4996484  1.1576107
  2       17      1.638725  0.4964588  1.1732176
  2       18      1.577319  0.5119547  1.1600758
  2       19      1.570975  0.5132788  1.1515172
  2       20      1.581195  0.5108002  1.1581386
  2       21      1.592801  0.5065599  1.1702771
  2       22      1.607156  0.5020322  1.1790769
  2       23      2.104922  0.4846656  1.3360063
  2       24      2.089819  0.4847881  1.3344808
  2       25      2.098483  0.4827853  1.3363581
  2       26      2.107680  0.4800812  1.3405904
  2       27      2.107051  0.4787019  1.3410179
  2       28      2.108357  0.4787559  1.3426877
  2       29      2.107413  0.4789573  1.3408933
  2       30      2.104041  0.4808557  1.3360273
  2       31      2.103084  0.4814124  1.3355609
  2       32      2.103084  0.4814124  1.3355609
  2       33      2.103084  0.4814124  1.3355609
  2       34      2.103084  0.4814124  1.3355609
  2       35      2.103084  0.4814124  1.3355609
  2       36      2.103084  0.4814124  1.3355609
  2       37      2.103084  0.4814124  1.3355609
  2       38      2.103084  0.4814124  1.3355609

RMSE was used to select the optimal model using the smallest value.
The final values used for the model were nprune = 3 and degree = 1.
plot(marsTune, main = "MARS Resampling Profile across Degree and Retained Terms")

# Support Vector Machine model with Radial Basis Kernel (svmR)

set.seed(100)
svmRTune <- train(
  x = trainX_trans,
  y = trainY,
  method = "svmRadial",
  tuneLength = 14,
  trControl = ctrl
)

svmRTune
Support Vector Machines with Radial Basis Function Kernel 

144 samples
 56 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ... 
Resampling results across tuning parameters:

  C        RMSE      Rsquared   MAE      
     0.25  1.378190  0.5223701  1.1110099
     0.50  1.263961  0.5698465  1.0193896
     1.00  1.167992  0.6199923  0.9396418
     2.00  1.138538  0.6322834  0.9220901
     4.00  1.109223  0.6462412  0.8996210
     8.00  1.096357  0.6548280  0.8895809
    16.00  1.096357  0.6548280  0.8895809
    32.00  1.096357  0.6548280  0.8895809
    64.00  1.096357  0.6548280  0.8895809
   128.00  1.096357  0.6548280  0.8895809
   256.00  1.096357  0.6548280  0.8895809
   512.00  1.096357  0.6548280  0.8895809
  1024.00  1.096357  0.6548280  0.8895809
  2048.00  1.096357  0.6548280  0.8895809

Tuning parameter 'sigma' was held constant at a value of 0.01691486
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were sigma = 0.01691486 and C = 8.
plot(svmRTune, scales = list(x = list(log = 2)),
     main = "Radial Basis Function SVM Resampling Profile")

# K-Nearest Neighbors (KNN) model

set.seed(100)
knnTune <- train(
  x = trainX_trans,
  y = trainY,
  method = "knn",
  tuneGrid = data.frame(k = 1:20),
  trControl = ctrl
)

knnTune
k-Nearest Neighbors 

144 samples
 56 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 129, 130, 130, 130, 130, 130, ... 
Resampling results across tuning parameters:

  k   RMSE      Rsquared   MAE      
   1  1.287542  0.5346454  1.0078153
   2  1.282679  0.5283690  1.0312421
   3  1.233852  0.5613118  0.9902430
   4  1.259767  0.5524228  0.9989688
   5  1.271359  0.5381109  1.0223896
   6  1.276441  0.5359752  1.0269244
   7  1.297789  0.5189795  1.0453212
   8  1.333438  0.4886811  1.0756940
   9  1.327318  0.5022821  1.0689361
  10  1.331763  0.5032410  1.0778037
  11  1.331852  0.5067123  1.0817101
  12  1.339905  0.5033552  1.0928018
  13  1.343580  0.5029629  1.0972452
  14  1.353473  0.4964769  1.1043683
  15  1.363085  0.4920052  1.1101127
  16  1.374256  0.4847915  1.1160828
  17  1.377099  0.4837612  1.1177292
  18  1.385684  0.4782693  1.1205679
  19  1.391376  0.4795634  1.1262610
  20  1.400789  0.4710035  1.1303387

RMSE was used to select the optimal model using the smallest value.
The final value used for the model was k = 3.
plot(knnTune, main = "K-Nearest Neighbors Resampling Profile")

Resampling and Test Set Performance Comparison

# Resampling comparison
resampList <- resamples(list(
  SVM_Radial = svmRTune,
  Neural_Net = nnetTune,
  MARS       = marsTune,
  KNN        = knnTune,
  PLS_Linear = plsTune
))

summary(resampList)

Call:
summary.resamples(object = resampList)

Models: SVM_Radial, Neural_Net, MARS, KNN, PLS_Linear 
Number of resamples: 50 

MAE 
                Min.   1st Qu.    Median      Mean  3rd Qu.     Max. NA's
SVM_Radial 0.4756940 0.7828983 0.9091418 0.8895809 1.010406 1.216072    0
Neural_Net 0.6692961 0.9495471 1.0556203 1.0954421 1.195130 1.638424    0
MARS       0.6729365 0.8679177 0.9983652 0.9955056 1.107992 1.433199    0
KNN        0.6062222 0.8694322 1.0037351 0.9902430 1.150056 1.322857    0
PLS_Linear 0.4006002 0.7973510 0.9427296 0.9126347 1.052786 1.204312    0

RMSE 
                Min.   1st Qu.   Median     Mean  3rd Qu.     Max. NA's
SVM_Radial 0.6085102 0.9791980 1.062667 1.096357 1.210015 1.550405    0
Neural_Net 0.7775011 1.1768963 1.376522 1.374637 1.494546 2.189232    0
MARS       0.7463047 1.0980821 1.212018 1.239401 1.394306 1.792399    0
KNN        0.7544034 1.0389920 1.240172 1.233852 1.437532 1.737376    0
PLS_Linear 0.6104034 0.9708719 1.139747 1.119484 1.292257 1.418129    0

Rsquared 
                 Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
SVM_Radial 0.35792848 0.5657407 0.6648553 0.6548280 0.7504176 0.9118547    0
Neural_Net 0.14352818 0.3957844 0.5201341 0.4925229 0.5975457 0.8293627    0
MARS       0.15807855 0.4957368 0.5824836 0.5637876 0.6630155 0.7811301    0
KNN        0.08223089 0.4588368 0.5855461 0.5613118 0.6821407 0.8527014    0
PLS_Linear 0.29649025 0.5465241 0.6647315 0.6480625 0.7421875 0.9199257    0
bwplot(resampList, metric = "RMSE",
       main = "Repeated 10-Fold CV RMSE Distribution Across Models")

# Paired hypothesis tests of resampling differences

modelDiffs <- diff(resampList)
summary(modelDiffs)

Call:
summary.diff.resamples(object = modelDiffs)

p-value adjustment: bonferroni 
Upper diagonal: estimates of the difference
Lower diagonal: p-value for H0: difference = 0

MAE 
           SVM_Radial Neural_Net MARS      KNN       PLS_Linear
SVM_Radial            -0.205861  -0.105925 -0.100662 -0.023054 
Neural_Net 9.194e-08              0.099936  0.105199  0.182807 
MARS       0.0004346  0.0186155             0.005263  0.082871 
KNN        0.0002948  0.0091828  1.0000000            0.077608 
PLS_Linear 1.0000000  1.031e-06  0.0181265 0.0705449           

RMSE 
           SVM_Radial Neural_Net MARS      KNN       PLS_Linear
SVM_Radial            -0.278281  -0.143044 -0.137495 -0.023127 
Neural_Net 5.350e-11              0.135236  0.140786  0.255154 
MARS       2.252e-05  0.0015885             0.005549  0.119917 
KNN        2.201e-05  0.0018277  1.0000000            0.114368 
PLS_Linear 1.0000000  8.495e-09  0.0005037 0.0122713           

Rsquared 
           SVM_Radial Neural_Net MARS      KNN       PLS_Linear
SVM_Radial             0.162305   0.091040  0.093516  0.006766 
Neural_Net 2.020e-10             -0.071265 -0.068789 -0.155540 
MARS       2.804e-05  0.0057648             0.002476 -0.084275 
KNN        0.0001671  0.0593179  1.0000000           -0.086751 
PLS_Linear 1.0000000  9.353e-10  3.722e-05 0.0027104           
# Test set predictions and evaluation
pred_svm  <- predict(svmRTune, newdata = testX_trans)
pred_nnet <- predict(nnetTune, newdata = testX_nnet)
pred_mars <- predict(marsTune, newdata = testX_trans)
pred_knn  <- predict(knnTune,  newdata = testX_trans)
pred_pls  <- predict(plsTune,  newdata = testX_trans)

testResults <- rbind(
  SVM_Radial = postResample(pred = pred_svm,  obs = testY),
  Neural_Net = postResample(pred = pred_nnet, obs = testY),
  MARS       = postResample(pred = pred_mars, obs = testY),
  KNN        = postResample(pred = pred_knn,  obs = testY),
  PLS_Linear = postResample(pred = pred_pls,  obs = testY)
)

# Combine CV means with Test set metrics

cvMeans <- summary(resampList)$statistics$RMSE[, "Mean"]
cvR2    <- summary(resampList)$statistics$Rsquared[, "Mean"]

perfTable <- data.frame(
  Model     = rownames(testResults),
  CV_RMSE   = round(cvMeans[rownames(testResults)], 4),
  CV_R2     = round(cvR2[rownames(testResults)], 4),
  Test_RMSE = round(testResults[, "RMSE"], 4),
  Test_R2   = round(testResults[, "Rsquared"], 4),
  Test_MAE  = round(testResults[, "MAE"], 4)
)
rownames(perfTable) <- NULL
knitr::kable(perfTable, caption = "Resampling and Held-Out Test Set Performance")
Resampling and Held-Out Test Set Performance
Model CV_RMSE CV_R2 Test_RMSE Test_R2 Test_MAE
SVM_Radial 1.0964 0.6548 1.2086 0.5666 1.0029
Neural_Net 1.3746 0.4925 1.7433 0.2049 1.4235
MARS 1.2394 0.5638 1.2232 0.5734 0.9757
KNN 1.2339 0.5613 1.4520 0.3700 1.1376
PLS_Linear 1.1195 0.6481 1.3858 0.4780 1.1987

b) Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?

Variable Importance analysis & Comparison with PLS Linear Model

We are going to compute variable importance scores for the optimal nonlinear model (svmRadial which uses a LOESS smoother R^2 measure for model-independent regression importance when internal weights are nonlinear) and compare its top 10 predictors against the top 10 predictors of the optimal linear model (plsTune, which uses weighted sums of absolute regression coefficients / VIP scores).

# Let's check the variable importance for Optimal Nonlinear Model (SVM)

svmImp <- varImp(svmRTune, scale = TRUE)
plsImp <- varImp(plsTune, scale = TRUE)

plot(svmImp, top = 15, main = "Top 15 Predictors: Optimal Nonlinear Model (SVM Radial)")

plot(plsImp, top = 15, main = "Top 15 Predictors: Optimal Linear Model (PLS)")

# Extract Top 10 predictor names and importance scores
getTopN <- function(impObj, n = 10) {
  df <- impObj$importance
  df$Predictor <- rownames(df)
  df <- df[order(-df$Overall), ]
  head(df[, c("Predictor", "Overall")], n)
}

top10_svm <- getTopN(svmImp, 10)
top10_pls <- getTopN(plsImp, 10)

compTop10 <- data.frame(
  Rank          = 1:10,
  SVM_Predictor = top10_svm$Predictor,
  SVM_Score     = round(top10_svm$Overall, 2),
  PLS_Predictor = top10_pls$Predictor,
  PLS_Score     = round(top10_pls$Overall, 2)
)

knitr::kable(compTop10, caption = "Top 10 Most Important Predictors: SVM Radial vs. PLS")
Top 10 Most Important Predictors: SVM Radial vs. PLS
Rank SVM_Predictor SVM_Score PLS_Predictor PLS_Score
1 ManufacturingProcess32 100.00 ManufacturingProcess32 100.00
2 BiologicalMaterial06 95.00 ManufacturingProcess17 90.70
3 BiologicalMaterial03 83.55 ManufacturingProcess13 89.73
4 ManufacturingProcess36 83.52 ManufacturingProcess09 88.86
5 ManufacturingProcess13 83.25 ManufacturingProcess36 86.16
6 BiologicalMaterial02 78.48 ManufacturingProcess06 74.68
7 ManufacturingProcess17 78.25 ManufacturingProcess33 63.68
8 ManufacturingProcess31 77.21 BiologicalMaterial03 62.08
9 ManufacturingProcess09 75.13 BiologicalMaterial06 61.84
10 BiologicalMaterial12 71.85 BiologicalMaterial08 60.35
# Let's categorize Biological vs. Process predictors in Top 10

svm_bio  <- sum(grepl("^BiologicalMaterial", top10_svm$Predictor))
svm_proc <- sum(grepl("^ManufacturingProcess", top10_svm$Predictor))

pls_bio  <- sum(grepl("^BiologicalMaterial", top10_pls$Predictor))
pls_proc <- sum(grepl("^ManufacturingProcess", top10_pls$Predictor))

cat(sprintf("SVM Top 10 Composition: %d Process, %d Biological\\n", svm_proc, svm_bio))
SVM Top 10 Composition: 6 Process, 4 Biological\n
cat(sprintf("PLS Top 10 Composition: %d Process, %d Biological\\n", pls_proc, pls_bio))
PLS Top 10 Composition: 7 Process, 3 Biological\n
sharedPreds <- intersect(top10_svm$Predictor, top10_pls$Predictor)
uniqueSVM   <- setdiff(top10_svm$Predictor, top10_pls$Predictor)
uniquePLS   <- setdiff(top10_pls$Predictor, top10_pls$Predictor)

cat("Shared Top-10 Predictors:", paste(sharedPreds, collapse = ", "), "\\n")
Shared Top-10 Predictors: ManufacturingProcess32, BiologicalMaterial06, BiologicalMaterial03, ManufacturingProcess36, ManufacturingProcess13, ManufacturingProcess17, ManufacturingProcess09 \n
cat("Unique to SVM Top-10:    ", paste(uniqueSVM, collapse = ", "), "\\n")
Unique to SVM Top-10:     BiologicalMaterial02, ManufacturingProcess31, BiologicalMaterial12 \n
cat("Unique to PLS Top-10:    ", paste(uniquePLS, collapse = ", "), "\\n")
Unique to PLS Top-10:      \n

Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?

To answer this question, we examine scatter plots with LOESS smoothers relating yield to the predictors that appear in the top 10 of svmRadial but not in the top 10 of plsTune. We plot these both on the pre-processed scale and on the original imputed scale so physical process interpretations remain transparent.

# Plot relationships for predictors unique to SVM Top 10
if (length(uniqueSVM) > 0) {
  plotData <- data.frame(
    Yield = trainY,
    trainX_trans[, uniqueSVM, drop = FALSE]
  )
  plotDataLong <- reshape2::melt(plotData, id.vars = "Yield",
                                 variable.name = "Predictor",
                                 value.name = "Standardized_Value")
  
  ggplot(plotDataLong, aes(x = Standardized_Value, y = Yield)) +
    geom_point(alpha = 0.7, color = "#2c3e50") +
    geom_smooth(method = "loess", se = TRUE, color = "#e74c3c", fill = "#f5b7b1") +
    geom_smooth(method = "lm", se = FALSE, linetype = "dashed", color = "#2980b9") +
    facet_wrap(~ Predictor, scales = "free_x", ncol = 2) +
    labs(
      title = "Nonlinear vs. Linear Trends for Predictors Unique to Optimal SVM Top 10",
      subtitle = "Solid red curve: LOESS smoother | Dashed blue line: Ordinary linear fit",
      x = "Pre-processed Predictor Value (Standardized)",
      y = "Manufacturing Yield (%)"
    ) +
    theme_bw(base_size = 12)
}
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'

We also inspect partial dependence / model response profiles from the fitted svmR model across these unique predictors to confirm how the SVM exploits non-monotonic, threshold, or saturation effects.

if (length(uniqueSVM) > 0) {
  medProfile <- as.data.frame(lapply(trainX_trans, median))
  pdpList <- lapply(uniqueSVM, function(varName) {
    gridVals <- seq(min(trainX_trans[[varName]]),
                    max(trainX_trans[[varName]]),
                    length.out = 60)
    newGrid <- medProfile[rep(1, 60), ]
    newGrid[[varName]] <- gridVals
    data.frame(
      Predictor = varName,
      Value     = gridVals,
      Predicted_Yield = predict(svmRTune, newdata = newGrid)
    )
  })
  pdpDF <- do.call(rbind, pdpList)
  
  ggplot(pdpDF, aes(x = Value, y = Predicted_Yield)) +
    geom_line(color = "#16a085", size = 1.2) +
    facet_wrap(~ Predictor, scales = "free_x", ncol = 2) +
    labs(
      title = "SVM Radial Partial Response Profiles (Other Predictors Fixed at Median)",
      x = "Pre-processed Predictor Value (Standardized)",
      y = "SVM Predicted Yield (%)"
    ) +
    theme_bw(base_size = 12)
}
Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.

Conclusion

Bivariate LOESS smoothers and SVM partial response profiles demonstrate that predictors unique to the SVM top 10 exhibit non-monotonic curvature, diminishing returns (plateaus), or threshold regimes with respect to yield—patterns that linear projections in PLS average out, explaining why kernel-based nonlinear regression captures substantial additional predictive signal.