Both exercises are from Chapter 7 of Kuhn and Johnson, Applied Predictive Modeling. Exercise 7.2 uses simulated data with a known nonlinear formula, so the true answer is known. Exercise 7.5 returns to the chemical manufacturing data. The models are the four from the chapter: K-nearest neighbors (KNN), a single-hidden-layer neural network (averaged over several random starts), multivariate adaptive regression splines (MARS), and a support vector machine (SVM) with a radial basis kernel.

Exercise 7.2

7.2. Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data: \(y = 10 \sin(\pi x_1 x_2) + 20 (x_3 - 0.5)^2 + 10 x_4 + 5 x_5 + N(0, \sigma^2)\), where the \(x\) values are random variables uniformly distributed between \([0, 1]\) (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:

library(mlbench)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)

## or other methods.

## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)

Tune several models on these data. For example:

library(caret)
knnModel <- train(x = trainingData$x,
                  y = trainingData$y,
                  method = "knn",
                  preProc = c("center", "scale"),
                  tuneLength = 10)
knnModel
knnPred <- predict(knnModel, newdata = testData$x)
## The function 'postResample' can be used to get the test set
## perforamnce values
postResample(pred = knnPred, obs = testData$y)

The book’s printed result for this call is a test set RMSE of 3.2286834 and R-squared of 0.6871735.

Which models appear to give the best performance? Does MARS select the informative predictors (those named X1-X5)?

The training set has 200 samples and 10 predictors; only X1 to X5 enter the formula, and X6 to X10 are noise. The test set of 5,000 samples gives a precise estimate of the true error. The error that cannot be removed is the simulated noise (standard deviation 1), so the best possible test RMSE is about 1.

All four models are tuned on the same resamples (the seed is set before each train() call) with 10-fold cross-validation repeated 5 times. KNN, the neural network and the SVM are sensitive to predictor scale, so they are centered and scaled. MARS needs no pre-processing (Section 7.2).

f1_ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)

f1_models <- fit_once("f1_models", {
  ## KNN, as in the exercise but with repeated cross-validation instead of the bootstrap
  set.seed(200)
  knnModel <- train(x = trainingData$x, y = trainingData$y,
                    method = "knn",
                    preProc = c("center", "scale"),
                    tuneLength = 10,
                    trControl = f1_ctrl)

  ## single-hidden-layer neural network, averaged over 5 random starts (avNNet)
  ## the predictors are independent uniform variables, so no correlation filtering is needed
  nnetGrid <- expand.grid(.decay = c(0, 0.01, 0.1),
                          .size = 1:10,
                          .bag = FALSE)
  set.seed(200)
  nnetModel <- train(x = trainingData$x, y = trainingData$y,
                     method = "avNNet",
                     preProc = c("center", "scale"),
                     tuneGrid = nnetGrid,
                     linout = TRUE, trace = FALSE,
                     MaxNWts = 10 * (ncol(trainingData$x) + 1) + 10 + 1,
                     maxit = 2000,
                     trControl = f1_ctrl)

  ## MARS: degree of the features and number of retained terms
  marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:20)
  set.seed(200)
  marsModel <- train(x = trainingData$x, y = trainingData$y,
                     method = "earth",
                     tuneGrid = marsGrid,
                     trControl = f1_ctrl)

  ## SVM, radial basis kernel (sigma estimated analytically, cost tuned)
  set.seed(200)
  svmModel <- train(x = trainingData$x, y = trainingData$y,
                    method = "svmRadial",
                    preProc = c("center", "scale"),
                    tuneLength = 14,
                    trControl = f1_ctrl)

  list(KNN = knnModel, NNet = nnetModel, MARS = marsModel, SVM = svmModel)
})
knnModel <- f1_models$KNN; nnetModel <- f1_models$NNet
marsModel <- f1_models$MARS; svmModel <- f1_models$SVM
lapply(f1_models, function(m) m$bestTune)
## $KNN
##    k
## 5 13
## 
## $NNet
##   size decay   bag
## 4    4     0 FALSE
## 
## $MARS
##    nprune degree
## 32     14      2
## 
## $SVM
##        sigma C
## 6 0.06299324 8
# convergence of the five networks averaged in the final model: 0 = converged,
# 1 = stopped at maxit (the iteration limit) before converging
sapply(nnetModel$finalModel$model, function(m) m$convergence)
## [1] 0 0 0 0 0

Why these grids, and whether the optimum sits on an edge.

par(mfrow = c(2, 2))
plot(knnModel$results$k, knnModel$results$RMSE, type = "b", pch = 19,
     xlab = "Neighbors (k)", ylab = "RMSE (repeated CV)", main = "KNN")
plot(svmModel$results$C, svmModel$results$RMSE, type = "b", pch = 19, log = "x",
     xlab = "Cost (log scale)", ylab = "RMSE (repeated CV)", main = "SVM, radial kernel")
mars_res <- marsModel$results
plot(mars_res$nprune[mars_res$degree == 1], mars_res$RMSE[mars_res$degree == 1],
     type = "b", pch = 19,
     ylim = range(mars_res$RMSE) + c(0, 0.3 * diff(range(mars_res$RMSE))),
     xlab = "Terms retained (nprune)", ylab = "RMSE (repeated CV)", main = "MARS")
lines(mars_res$nprune[mars_res$degree == 2], mars_res$RMSE[mars_res$degree == 2],
      type = "b", pch = 17, col = "firebrick")
legend("top", c("degree 1", "degree 2"), pch = c(19, 17),
       col = c("black", "firebrick"), bty = "n", horiz = TRUE)
nn_res <- nnetModel$results
plot(nn_res$size[nn_res$decay == 0], nn_res$RMSE[nn_res$decay == 0], type = "b", pch = 19,
     ylim = range(nn_res$RMSE) + c(0, 0.3 * diff(range(nn_res$RMSE))),
     xlab = "Hidden units", ylab = "RMSE (repeated CV)", main = "Neural network")
lines(nn_res$size[nn_res$decay == 0.01], nn_res$RMSE[nn_res$decay == 0.01],
      type = "b", pch = 17, col = "steelblue")
lines(nn_res$size[nn_res$decay == 0.1], nn_res$RMSE[nn_res$decay == 0.1],
      type = "b", pch = 15, col = "firebrick")
legend("top", c("decay 0", "decay 0.01", "decay 0.1"), pch = c(19, 17, 15),
       col = c("black", "steelblue", "firebrick"), bty = "n", horiz = TRUE)

par(mfrow = c(1, 1))

Each profile has a clear minimum or plateau: KNN is lowest at k = 13, the SVM is flat from a cost of about 8, MARS is lowest at 14 terms with degree 2, and the neural network at 4 hidden units with decay 0.

Resampled and test set performance of each model at its best tuning values:

f1_resampled <- t(sapply(f1_models, function(m) {
  best <- m$results[which.min(m$results$RMSE), ]
  c(RMSE = best$RMSE, Rsquared = best$Rsquared, MAE = best$MAE)
}))

f1_test <- t(sapply(f1_models, function(m)
  postResample(pred = predict(m, newdata = testData$x), obs = testData$y)))

f1_perf <- cbind(f1_resampled, f1_test)
colnames(f1_perf) <- c("CV_RMSE", "CV_R2", "CV_MAE", "Test_RMSE", "Test_R2", "Test_MAE")
round(f1_perf, 3)
##      CV_RMSE CV_R2 CV_MAE Test_RMSE Test_R2 Test_MAE
## KNN    3.065 0.697  2.468     3.148   0.675    2.524
## NNet   1.950 0.844  1.544     2.459   0.775    1.779
## MARS   1.241 0.937  0.992     1.172   0.945    0.932
## SVM    1.868 0.857  1.477     2.054   0.829    1.559
f1_resamp <- resamples(f1_models)
bwplot(f1_resamp, metric = "RMSE")

summary(diff(f1_resamp))
## 
## Call:
## summary.diff.resamples(object = diff(f1_resamp))
## 
## p-value adjustment: bonferroni 
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
## 
## MAE 
##      KNN       NNet      MARS      SVM     
## KNN             0.92357   1.47580   0.99066
## NNet < 2.2e-16            0.55223   0.06709
## MARS < 2.2e-16 < 2.2e-16           -0.48514
## SVM  < 2.2e-16 0.7452    3.537e-16         
## 
## RMSE 
##      KNN       NNet      MARS      SVM     
## KNN             1.11505   1.82398   1.19730
## NNet < 2.2e-16            0.70893   0.08225
## MARS < 2.2e-16 4.727e-16           -0.62668
## SVM  < 2.2e-16 0.6167    < 2.2e-16         
## 
## Rsquared 
##      KNN       NNet      MARS      SVM     
## KNN            -0.14743  -0.24035  -0.16032
## NNet 7.124e-16           -0.09292  -0.01289
## MARS < 2.2e-16 1.262e-11            0.08003
## SVM  < 2.2e-16 0.668     3.369e-13
# the KNN model from the exercise's call, tuned with repeated cross-validation
knnModel
## k-Nearest Neighbors 
## 
## 200 samples
##  10 predictor
## 
## Pre-processing: centered (10), scaled (10) 
## Resampling: Cross-Validated (10 fold, repeated 5 times) 
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
## Resampling results across tuning parameters:
## 
##   k   RMSE      Rsquared   MAE     
##    5  3.199670  0.5910371  2.649336
##    7  3.108444  0.6299684  2.537062
##    9  3.090673  0.6553105  2.506983
##   11  3.070640  0.6798579  2.489940
##   13  3.064958  0.6966689  2.467551
##   15  3.082000  0.7111449  2.491659
##   17  3.102544  0.7142288  2.506028
##   19  3.115994  0.7231515  2.517837
##   21  3.148223  0.7237919  2.560250
##   23  3.177131  0.7236183  2.589908
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 13.
knnPred <- predict(knnModel, newdata = testData$x)
postResample(pred = knnPred, obs = testData$y)
##      RMSE  Rsquared       MAE 
## 3.1481557 0.6747755 2.5236041

Which models perform best. MARS is clearly the best: its resampled RMSE is 1.241 and its test RMSE is 1.172 (R-squared 0.945), close to the noise floor of about 1. The SVM is second (test RMSE 2.054), the neural network third (2.459) and KNN last (3.148). The diff() table says MARS differs from each of the other three with adjusted p-values below 0.0001. The SVM and the neural network cannot be told apart on the resamples (adjusted p-value about 0.62 for RMSE), although the SVM is ahead on the large test set. The resampled and test RMSE are close for KNN and MARS, differ by about 0.2 for the SVM, and differ most for the neural network (test 2.46 against resampled 1.95), so the 5,000-sample test set is the more trustworthy referee here.

The ranking has a reason. The true function is additive in X3, X4 and X5 and has one interaction, sin(pi * X1 * X2), which is the structure a degree-2 MARS model with hinge terms can build, and MARS also drops predictors it does not need. KNN, the SVM and the neural network use all ten predictors, so the five noise predictors may dilute the distances (Section 7.4 notes that noisy predictors push similar samples apart) and add parameters to fit; that explanation is consistent with the results, though this comparison did not isolate the effect of the noise predictors. KNN’s test RMSE of 3.15 is close to the 3.23 the book reports for its example call (which used the bootstrap). All of this comes from one simulated training set of 200 samples; a different seed would move the numbers.

Checking the SVM parameters by hand

train() reports the best SVM settings, but it helps to see what each parameter does. The chunk below fits the SVM directly with ksvm() on the scaled training data and changes one setting at a time, reporting the 10-fold cross-validated RMSE and the number of support vectors. This is a supplementary sensitivity analysis: it uses ksvm()’s internal 10-fold cross-validation after scaling on the full training set, so its errors are not directly comparable with the main repeated cross-validation results, where preprocessing is fitted within each resample. It only illustrates how the parameters affect the fit. The external test set is not used here.

x_scaled <- scale(trainingData$x)

try_svm <- function(cost, sigma, epsilon) {
  set.seed(1)
  fit <- ksvm(x_scaled, trainingData$y, type = "eps-svr", kernel = "rbfdot",
              kpar = list(sigma = sigma), C = cost, epsilon = epsilon, cross = 10)
  c(cost = cost, sigma = sigma, epsilon = epsilon,
    CV_RMSE = sqrt(cross(fit)), support_vectors = nSV(fit))
}

# cost and sigma, with epsilon fixed at 0.1
svm_cs <- expand.grid(cost = c(0.25, 1, 8, 64), sigma = c(0.005, 0.063, 0.5))
svm_cs <- t(apply(svm_cs, 1, function(p) try_svm(p[["cost"]], p[["sigma"]], 0.1)))
round(svm_cs, 3)
##        cost sigma epsilon CV_RMSE support_vectors
##  [1,]  0.25 0.005     0.1   3.797             175
##  [2,]  1.00 0.005     0.1   2.674             174
##  [3,]  8.00 0.005     0.1   2.345             171
##  [4,] 64.00 0.005     0.1   1.969             161
##  [5,]  0.25 0.063     0.1   2.526             167
##  [6,]  1.00 0.063     0.1   2.072             160
##  [7,]  8.00 0.063     0.1   1.786             152
##  [8,] 64.00 0.063     0.1   1.794             150
##  [9,]  0.25 0.500     0.1   4.702             187
## [10,]  1.00 0.500     0.1   4.254             183
## [11,]  8.00 0.500     0.1   4.143             184
## [12,] 64.00 0.500     0.1   4.143             184
# epsilon, with cost 8 and sigma 0.063
svm_eps <- t(sapply(c(0.01, 0.1, 0.5, 1, 2), function(e) try_svm(8, 0.063, e)))
round(svm_eps, 3)
##      cost sigma epsilon CV_RMSE support_vectors
## [1,]    8 0.063    0.01   1.810             198
## [2,]    8 0.063    0.10   1.786             152
## [3,]    8 0.063    0.50   2.279              52
## [4,]    8 0.063    1.00   3.077              19
## [5,]    8 0.063    2.00   4.656               4
  • Cost is the penalty for points that fall outside the epsilon tube. At the middle sigma (0.063) the cross-validated RMSE falls from 2.53 at a cost of 0.25 to 1.79 at 8 and does not improve at 64: the validation error decreased as cost increased to 8, with no improvement at 64 in this experiment, which matches the plateau in the cost profile above.
  • Sigma sets how far each training point’s influence reaches. A very small value (0.005) gives a very smooth surface and performed worse than the middle setting even at a high cost (RMSE 1.97 at cost 64). A large value (0.5) gives more localized predictions and a much higher validation error (RMSE about 4.1). The analytic estimate used by train() (0.063) sits in the good middle region. These results show poorer validation performance at the extreme settings; they do not by themselves show why.
  • Epsilon is the half-width of the tube inside which errors cost nothing. ksvm() standardizes the response by default, so these epsilon values are in standardized units: with a response standard deviation of 4.95, epsilon 0.1 is about 0.5 in original units and epsilon 1 is about 4.95, while the RMSE is reported on the original scale. A tiny epsilon makes almost every point a support vector (198 of 200 at 0.01); a wide one leaves only a few (19 at 1) and the fit is much worse, with RMSE 3.1 (4.7 at 2, close to the standard deviation of y, about 5). The default of 0.1 gives the lowest error of those tried, and train() keeps epsilon fixed at that default and tunes only the cost.

The values train() chose (cost 8, sigma 0.063) land where the by-hand exploration shows the lowest error. The set.seed(1) inside try_svm() only fixes the cross-validation folds so the table is reproducible.

Does MARS select the informative predictors?

marsModel$finalModel
## Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 9 4
## GCV 1.62945    RSS 225.8601    GRSq 0.9338437    RSq 0.953688
summary(marsModel$finalModel)
## Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
##             nprune=14)
## 
##                                 coefficients
## (Intercept)                        21.905319
## h(0.621722-X1)                    -15.726181
## h(X1-0.621722)                      9.234027
## h(0.601063-X2)                    -18.253527
## h(X2-0.601063)                     10.448545
## h(0.447442-X3)                      9.700589
## h(X3-0.606015)                     12.674694
## h(0.734892-X4)                     -9.863526
## h(X4-0.734892)                     10.297964
## h(0.850094-X5)                     -5.324175
## h(0.218266-X1) * h(X2-0.601063)   -55.358726
## h(X1-0.218266) * h(X2-0.601063)   -29.100250
## h(X1-0.621722) * h(X2-0.295997)   -26.833129
## h(0.649253-X1) * h(0.601063-X2)    27.120721
## 
## Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 9 4
## GCV 1.62945    RSS 225.8601    GRSq 0.9338437    RSq 0.953688
varImp(marsModel, scale = FALSE)   # scale = FALSE: show earth's own values, not caret's 0-100 rescaling
## earth variable importance
## 
##    Overall
## X1  100.00
## X4   85.13
## X2   69.22
## X5   49.28
## X3   39.95
plotmo(marsModel$finalModel, caption = "MARS fitted response profiles: Friedman data")
##  plotmo grid:    X1        X2       X3        X4        X5        X6        X7
##           0.5139349 0.5106664 0.537307 0.4445841 0.5343299 0.4975981 0.4688035
##        X8        X9       X10
##  0.497961 0.5288716 0.5359218

# predictors that appear in at least one retained (not pruned) MARS term
fm <- marsModel$finalModel
retained <- fm$dirs[fm$selected.terms, , drop = FALSE]
used <- colnames(retained)[colSums(retained != 0) > 0]
used
## [1] "X1" "X2" "X3" "X4" "X5"

Does MARS select the informative predictors? Yes. The final model keeps 5 of the 10 predictors, X1, X2, X3, X4, X5, and none of X6 to X10 appears in any term. The equation has two hinge terms for each of X1 to X4, one hinge for X5, and four degree-2 terms that are products of X1 and X2 hinges, which matches the X1 * X2 interaction in the formula. The two X3 hinges have different knots (0.447 and 0.606, on either side of 0.5 where the true quadratic term 20 * (X3 - 0.5)^2 has its minimum), so together they give a falling segment, a flat middle segment and a rising segment, a rough piecewise-linear version of that curve. X3 is retained in the final equation, yet the default importance table gives it 0. That zero comes from caret’s extra rescaling, which maps the lowest reported importance to 0; with scale = FALSE (table above) X3 has an importance of about 40, the lowest of the five. Which predictors were selected is therefore determined from the retained terms in the equation, not from whether the scaled importance is positive.

Exercise 7.5

7.5. Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.

The data, the 80/20 split (seed 624), the removal of near-zero-variance predictors, and the KNN imputation fitted on the training rows only are the same as in Exercise 6.3.

data(ChemicalManufacturingProcess)
process_predictors <- ChemicalManufacturingProcess[, -1]
yield <- ChemicalManufacturingProcess$Yield

set.seed(624)
cmp_rows <- createDataPartition(yield, p = 0.80, list = FALSE)

cmp_train_x <- process_predictors[cmp_rows, ]
cmp_test_x  <- process_predictors[-cmp_rows, ]
cmp_train_y <- yield[cmp_rows]
cmp_test_y  <- yield[-cmp_rows]

cmp_nzv <- nearZeroVar(cmp_train_x, names = TRUE)
cmp_nzv
## [1] "BiologicalMaterial07"
cmp_train_x <- cmp_train_x[, !(names(cmp_train_x) %in% cmp_nzv)]
cmp_test_x  <- cmp_test_x[, !(names(cmp_test_x) %in% cmp_nzv)]

c(train = nrow(cmp_train_x), test = nrow(cmp_test_x), predictors = ncol(cmp_train_x),
  missing_cells_train = sum(is.na(cmp_train_x)))
##               train                test          predictors missing_cells_train 
##                 144                  32                  56                  99

As in Exercise 6.3, the imputation is fitted inside each resample with preProc = "knnImpute" (which also centers and scales), so a held-out fold never influences the values its own predictors receive, and predict() applies the training imputation to the test rows.

The neural network is the model most hurt by correlated predictors (Section 7.1), so for it the book removes predictors until no pair has a correlation above 0.75. Here that filter ("corr") is applied inside each training resample through preProcOptions. caret runs it first, on pairwise complete observations with an absolute-correlation cutoff of 0.75; the remaining predictors are then centered, scaled and imputed using the training data of that resample (writing c("knnImpute", "corr") does not set the order).

(a) Which nonlinear model gives the best resampling and test set performance?

Which nonlinear regression model gives the optimal resampling and test set performance?

cmp_ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)

cmp_models <- fit_once("cmp_models", {
  set.seed(100)
  cmp_knn <- train(cmp_train_x, cmp_train_y, method = "knn",
                   preProc = "knnImpute",
                   tuneGrid = data.frame(k = 1:20),
                   trControl = cmp_ctrl)

  set.seed(100)
  cmp_nnet <- train(cmp_train_x, cmp_train_y, method = "avNNet",
                    # "corr" is not in the book's code (the book calls findCorrelation() directly);
                    # it does the same filtering inside each resample, with the cutoff set below
                    preProc = c("knnImpute", "corr"),
                    tuneGrid = expand.grid(decay = c(0, 0.01, 0.1), size = 1:7, bag = FALSE),
                    linout = TRUE, trace = FALSE, maxit = 500,
                    MaxNWts = 7 * (ncol(cmp_train_x) + 1) + 7 + 1,
                    trControl = trainControl(method = "repeatedcv", number = 10, repeats = 5,
                                             preProcOptions = list(thresh = 0.95, ICAcomp = 3,
                                                                   k = 5, freqCut = 19,
                                                                   uniqueCut = 10,
                                                                   cutoff = 0.75)))

  set.seed(100)
  cmp_mars <- train(cmp_train_x, cmp_train_y, method = "earth",
                    preProc = "knnImpute",
                    tuneGrid = expand.grid(degree = 1:2, nprune = 2:30),
                    trControl = cmp_ctrl)

  set.seed(100)
  cmp_svm <- train(cmp_train_x, cmp_train_y, method = "svmRadial",
                   preProc = "knnImpute",
                   tuneLength = 14,
                   trControl = cmp_ctrl)

  list(KNN = cmp_knn, NNet = cmp_nnet, MARS = cmp_mars, SVM = cmp_svm)
})
cmp_knn <- cmp_models$KNN; cmp_nnet <- cmp_models$NNet
cmp_mars <- cmp_models$MARS; cmp_svm <- cmp_models$SVM
lapply(cmp_models, function(m) m$bestTune)
## $KNN
##   k
## 5 5
## 
## $NNet
##   size decay   bag
## 8    1  0.01 FALSE
## 
## $MARS
##    nprune degree
## 32      4      2
## 
## $SVM
##        sigma C
## 6 0.01396075 8

Why these grids, and whether the optimum sits on an edge.

  • KNN, k from 1 to 20: the full range of small neighbourhoods for 144 runs, including k = 1 as the extreme. The best k is 5, inside the grid.
  • Neural network, decay 0, 0.01, 0.1 and 1 to 7 hidden units: the same decay values as in 7.2, and a smaller maximum size than there (7 instead of 10) because 144 runs and 56 predictors give a network with many weights per observation (the MaxNWts limit is set to fit the largest network; the correlation filter leaves 35 of the 56 predictors for this model). The best network has 1 hidden unit, which is the lower edge of the size grid. One unit is the smallest possible network, so the grid cannot go lower, and the profile shows every larger size doing worse; the decay of 0.01 is inside its grid.
  • MARS, degree 1 to 2 and 2 to 30 retained terms: degree is limited to 1 and 2 as in the chapter, and the forward pass builds up to 50 terms, so 30 reaches well beyond the best model. The best model keeps 4 terms, near the low end but not on the edge, with degree 2, the top of the degree grid. That edge is harmless here: the final equation has no interaction terms, so a degree-1 model would give essentially the same fit.
  • SVM, tuneLength = 14: the same cost grid as in 7.2 (0.25 to 2048). The best cost is 8; larger costs give nearly identical resampled RMSE, and train() selected 8 because it had the lowest value.
cmp_resampled <- t(sapply(cmp_models, function(m) {
  best <- m$results[which.min(m$results$RMSE), ]
  c(RMSE = best$RMSE, Rsquared = best$Rsquared, MAE = best$MAE)
}))
cmp_test_perf <- t(sapply(cmp_models, function(m)
  postResample(pred = predict(m, cmp_test_x), obs = cmp_test_y)))

cmp_perf <- cbind(cmp_resampled, cmp_test_perf)
colnames(cmp_perf) <- c("CV_RMSE", "CV_R2", "CV_MAE", "Test_RMSE", "Test_R2", "Test_MAE")
round(cmp_perf, 3)
##      CV_RMSE CV_R2 CV_MAE Test_RMSE Test_R2 Test_MAE
## KNN    1.294 0.500  1.063     1.402   0.613    1.082
## NNet   1.413 0.423  1.150     1.496   0.499    1.183
## MARS   1.193 0.570  0.971     1.201   0.713    0.918
## SVM    1.130 0.603  0.930     1.147   0.748    0.796
cmp_resamp <- resamples(cmp_models)
bwplot(cmp_resamp, metric = "RMSE")

summary(diff(cmp_resamp))
## 
## Call:
## summary.diff.resamples(object = diff(cmp_resamp))
## 
## p-value adjustment: bonferroni 
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
## 
## MAE 
##      KNN       NNet      MARS     SVM     
## KNN            -0.08781   0.09128  0.13274
## NNet 0.16378              0.17909  0.22055
## MARS 0.01382   4.618e-06           0.04145
## SVM  4.558e-05 9.029e-10 0.75487          
## 
## RMSE 
##      KNN       NNet      MARS     SVM     
## KNN            -0.11900   0.10090  0.16348
## NNet 0.06635              0.21990  0.28248
## MARS 0.03578   3.251e-06           0.06258
## SVM  1.120e-05 3.574e-10 0.37273          
## 
## Rsquared 
##      KNN       NNet      MARS      SVM     
## KNN             0.07695  -0.07014  -0.10274
## NNet 0.1584712           -0.14709  -0.17969
## MARS 0.0470613 9.632e-07           -0.03259
## SVM  0.0005413 8.860e-09 0.9500346
par(mfrow = c(2, 2))
plot(cmp_knn$results$k, cmp_knn$results$RMSE, type = "b", pch = 19,
     xlab = "Neighbors (k)", ylab = "RMSE (repeated CV)", main = "KNN")
plot(cmp_svm$results$C, cmp_svm$results$RMSE, type = "b", pch = 19, log = "x",
     xlab = "Cost (log scale)", ylab = "RMSE (repeated CV)", main = "SVM, radial kernel")
cm <- cmp_mars$results
plot(cm$nprune[cm$degree == 1], cm$RMSE[cm$degree == 1], type = "b", pch = 19,
     ylim = range(cm$RMSE) + c(0, 0.3 * diff(range(cm$RMSE))),
     xlab = "Terms retained (nprune)", ylab = "RMSE (repeated CV)", main = "MARS")
lines(cm$nprune[cm$degree == 2], cm$RMSE[cm$degree == 2], type = "b", pch = 17, col = "firebrick")
legend("top", c("degree 1", "degree 2"), pch = c(19, 17), col = c("black", "firebrick"), bty = "n",
       horiz = TRUE)
cn <- cmp_nnet$results
plot(cn$size[cn$decay == 0], cn$RMSE[cn$decay == 0], type = "b", pch = 19,
     ylim = range(cn$RMSE) + c(0, 0.3 * diff(range(cn$RMSE))),
     xlab = "Hidden units", ylab = "RMSE (repeated CV)", main = "Neural network")
lines(cn$size[cn$decay == 0.01], cn$RMSE[cn$decay == 0.01], type = "b", pch = 17, col = "steelblue")
lines(cn$size[cn$decay == 0.1], cn$RMSE[cn$decay == 0.1], type = "b", pch = 15, col = "firebrick")
legend("top", c("decay 0", "decay 0.01", "decay 0.1"), pch = c(19, 17, 15),
       col = c("black", "steelblue", "firebrick"), bty = "n", horiz = TRUE)

par(mfrow = c(1, 1))

The SVM with a radial kernel gives the best resampling and test set performance: resampled RMSE 1.13 (R-squared 0.603) and test RMSE 1.147 (R-squared 0.748). The ranking by RMSE is the same on resampling and on the test set: SVM, then MARS (1.193 and 1.201), KNN (1.294 and 1.402) and the neural network (1.413 and 1.496). The SVM’s resampled and test RMSE agree closely. The diff() table shows the SVM is clearly better than KNN and the neural network (adjusted p-values near 0.00001 and below), but the SVM and MARS cannot be separated (adjusted p-value about 0.37 for RMSE; about 0.19 in the three-model comparison further down, because the adjustment depends on how many models are compared), so the paired comparison did not detect a difference between the SVM and MARS, and the SVM’s lead of about 0.06 RMSE should not be treated as established. The mean cross-validated R-squared is approximately 0.60, while the test-set R-squared is approximately 0.75, so the best model’s predictions are only moderately correlated with yield. A test RMSE near 1.15 yield points is the lowest among the tested candidates, but whether that is accurate enough for plant decisions depends on an error tolerance the exercise does not give.

The tuning profiles show how each model was chosen. The SVM’s RMSE falls as the cost rises and is essentially flat from a cost of about 8 upward. train() picked the cost with the lowest resampled RMSE (8, with sigma estimated analytically as 0.014); the differences among the larger costs are tiny, so the plateau supports the choice but does not single out one value. KNN is best at 5 neighbors, with RMSE worse for a single neighbor and rising slowly beyond. MARS is best as a very small additive model: 4 terms involving only two predictors (shown in the MARS output in part b). Its degree-2 profile rises steeply with more terms and ends near RMSE 2.1, the instability with higher-degree models that the chapter warns about. The neural network is best at 1 hidden unit with decay 0.01, the edge of the grid, and no larger network does better: RMSE is higher than its one-unit value for every larger network at every decay value, with the decay 0.01 profile peaking at four units and partly recovering at six and seven but staying above its one-unit value. Larger networks generally did worse on resampling; over-fitting with only 144 training runs and several dozen predictors is a plausible reason, though the profile alone does not separate it from other limits of the fit. A one-unit network cannot be shrunk further, so the grid did not need to extend below it.

The test set holds only 32 runs, so before reading too much into the differences, the largest absolute test errors of each model show whether a few runs drive the results (row numbers are the run’s position in the original data):

lapply(cmp_models, function(m) {
  err <- predict(m, cmp_test_x) - cmp_test_y
  names(err) <- rownames(cmp_test_x)
  round(head(sort(abs(err), decreasing = TRUE), 3), 2)
})
## $KNN
##   49   44  130 
## 4.13 3.61 2.16 
## 
## $NNet
##   49   28   44 
## 4.42 2.46 2.35 
## 
## $MARS
##   49   44  121 
## 3.52 2.93 1.92 
## 
## $SVM
##   49   44   77 
## 4.42 2.51 1.50

Run 49 is the worst-predicted test run for all four models (errors of about 3.5 to 4.4 yield points) and run 44 is second for three of them, so these two runs account for a large share of every model’s test RMSE. Differences of a few hundredths between models on this test set mean little; the resampled comparison, which uses all 144 training runs, is the firmer evidence.

(b) Most important predictors, and comparison with the optimal linear model

Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?

cmp_best_name <- names(which.min(cmp_resampled[, "RMSE"]))
cmp_best <- cmp_models[[cmp_best_name]]
cmp_best_name
## [1] "SVM"
cmp_imp <- varImp(cmp_best)
cmp_imp
## loess r-squared variable importance
## 
##   only 20 most important variables shown (out of 56)
## 
##                        Overall
## ManufacturingProcess32  100.00
## BiologicalMaterial06     89.32
## ManufacturingProcess36   79.78
## BiologicalMaterial03     77.48
## ManufacturingProcess09   73.90
## ManufacturingProcess13   73.23
## ManufacturingProcess31   70.84
## BiologicalMaterial02     66.91
## ManufacturingProcess11   65.19
## BiologicalMaterial12     64.94
## ManufacturingProcess06   61.04
## ManufacturingProcess17   53.06
## ManufacturingProcess29   50.91
## BiologicalMaterial11     49.10
## BiologicalMaterial04     48.27
## ManufacturingProcess33   47.62
## ManufacturingProcess30   47.38
## BiologicalMaterial01     40.69
## BiologicalMaterial08     38.18
## ManufacturingProcess25   34.07
plot(cmp_imp, top = 10,
     main = paste("Top ten predictors,", cmp_best_name, "(marginal loess R-squared)"))

imp_order <- order(cmp_imp$importance$Overall, decreasing = TRUE)
nl_top10 <- rownames(cmp_imp$importance)[imp_order][1:10]
nl_top10
##  [1] "ManufacturingProcess32" "BiologicalMaterial06"   "ManufacturingProcess36"
##  [4] "BiologicalMaterial03"   "ManufacturingProcess09" "ManufacturingProcess13"
##  [7] "ManufacturingProcess31" "BiologicalMaterial02"   "ManufacturingProcess11"
## [10] "BiologicalMaterial12"
table(ifelse(grepl("^Biological", nl_top10), "Biological", "Process"))
## 
## Biological    Process 
##          4          6

varImp() is model-specific for MARS (caret rescales it to 0-100) but, for the SVM, caret has no model-based measure and falls back to a filter: the R-squared of a loess fit of each predictor against the response, computed one predictor at a time (the “loess r-squared variable importance” in the heading above). That ranking therefore shows which predictors relate to yield on their own, not which predictors the fitted SVM leans on. As a model-based cross-check, the MARS model (second best on the test set) has its own importance, based on the reduction in GCV error each predictor’s terms give (shown with scale = FALSE, earth’s own relative values):

varImp(cmp_mars, scale = FALSE)
## earth variable importance
## 
##                        Overall
## ManufacturingProcess32  100.00
## ManufacturingProcess09   58.09
summary(cmp_mars$finalModel)
## Call: earth(x=data.frame[144,56], y=c(38,42.44,42.0...), keepxy=TRUE, degree=2,
##             nprune=4)
## 
##                                     coefficients
## (Intercept)                            37.822440
## h(-1.16089-ManufacturingProcess09)     -1.043508
## h(ManufacturingProcess09- -1.16089)     0.742130
## h(ManufacturingProcess32- -1.24119)     1.160811
## 
## Selected 4 of 50 terms, and 2 of 56 predictors (nprune=4)
## Termination condition: RSq changed by less than 0.001 at 50 terms
## Importance: ManufacturingProcess32, ManufacturingProcess09, ...
## Number of terms at each degree of interaction: 1 3 (additive model)
## GCV 1.423486    RSS 181.497    GRSq 0.5542264    RSq 0.5997596

The tuned MARS model is a four-term additive model that uses only ManufacturingProcess32 and ManufacturingProcess09, so no other predictor appears in the model. With caret’s default rescaling ManufacturingProcess09 would be shown as 0 even though it has two terms in the equation, because that rescaling maps the lowest reported importance to 0; its importance before rescaling is about 58, against 100 for ManufacturingProcess32. The model equation is the direct evidence of which predictors it uses. That is far fewer predictors than the SVM works with, yet its test RMSE is within about 0.05 of the SVM’s.

The linear model used for comparison is the elastic net, the best of the linear models fitted to these data in Exercise 6.3, refit here with exactly the same resampling so the comparison is fair. Its importance is the absolute size of the coefficients at the selected fraction, on the standardized predictors (Section 6.5). Its grid is lambda 0.01, 0.1 and 1 (two orders of magnitude of ridge penalty) crossed with fraction 0.05 to 1 in 20 steps (this spans most of the elastic-net coefficient path for each penalty); the selected values are listed below and neither is on the edge of its grid.

cmp_enet <- fit_once("cmp_enet", {
  set.seed(100)
  train(cmp_train_x, cmp_train_y, method = "enet",
        preProc = "knnImpute",
        tuneGrid = expand.grid(lambda = c(0.01, 0.1, 1),
                               fraction = seq(0.05, 1, length = 20)),
        trControl = cmp_ctrl)
})
cmp_enet$bestTune
##    fraction lambda
## 26      0.3    0.1
c(resampled_RMSE = min(cmp_enet$results$RMSE),
  test_RMSE = unname(postResample(predict(cmp_enet, cmp_test_x), cmp_test_y)["RMSE"]))
## resampled_RMSE      test_RMSE 
##       1.196128       1.219538
lin_coef <- predict(cmp_enet$finalModel, s = cmp_enet$bestTune$fraction,
                    mode = "fraction", type = "coefficients")$coefficients
lin_rank <- sort(abs(lin_coef[lin_coef != 0]), decreasing = TRUE)
lin_top10 <- names(lin_rank)[1:10]
c(nonzero_coefficients = length(lin_rank))
## nonzero_coefficients 
##                   13

The elastic net has a resampled RMSE of 1.196 and a test RMSE of 1.22, against 1.13 and 1.147 for the SVM. The SVM is lower by about 0.07 RMSE on both measures, and MARS (resampled 1.193) is no better than the linear model on resampling. The elastic net was tuned on the same resamples (same seed), so the three models can be compared on paired resampled RMSE like the models above:

cmp_vs_linear <- resamples(list(SVM = cmp_svm, MARS = cmp_mars, Enet = cmp_enet))
summary(diff(cmp_vs_linear), metric = "RMSE")
## 
## Call:
## summary.diff.resamples(object = diff(cmp_vs_linear), metric = "RMSE")
## 
## p-value adjustment: bonferroni 
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
## 
## MAE 
##      SVM    MARS     Enet    
## SVM         -0.04145 -0.03943
## MARS 0.3774           0.00202
## Enet 0.4599 1.0000           
## 
## RMSE 
##      SVM    MARS      Enet     
## SVM         -0.062585 -0.065702
## MARS 0.1864           -0.003117
## Enet 0.3965 1.0000             
## 
## Rsquared 
##      SVM    MARS      Enet     
## SVM          0.032595  0.026803
## MARS 0.4750           -0.005791
## Enet 0.8051 1.0000

The SVM’s RMSE is lower than the elastic net’s by about 0.07, but the paired comparison gives an adjusted p-value of about 0.40 for that difference, and MARS is practically equal to the elastic net, so the resamples give no evidence that either nonlinear model predicts better than the linear one. The tested nonlinear models did not demonstrate a clear improvement over the elastic net with this sample and tuning procedure; similar predictive performance does not establish the underlying form of the relationship. The SVM is still the best of the four nonlinear models, and its test RMSE is also about 0.07 lower than the elastic net’s, but one small test set cannot confirm a gain the resamples do not show.

Choosing between near-ties. The resamples cannot separate the SVM, MARS and the elastic net, so a simpler model is the better practical choice when predictive error is tied. The SVM is called the best model above because it has the lowest resampled RMSE and the exercise asks for the optimal nonlinear model, and its importance list is used for part (b). If one model had to be used, though, MARS (four terms, two predictors) or the elastic net (linear, 13 nonzero coefficients) would be preferable to the SVM, which uses all 56 predictors and cannot be written as a short equation. MARS is the smallest of the three; the elastic net is the easiest to explain to a plant engineer. In Exercise 7.2 there is no such tie at the top: MARS beats the other three clearly, and it is also the model that matches the true structure.

top10_table <- data.frame(
  Rank = 1:10,
  Nonlinear = nl_top10,
  Linear = lin_top10)
top10_table
##    Rank              Nonlinear                 Linear
## 1     1 ManufacturingProcess32 ManufacturingProcess32
## 2     2   BiologicalMaterial06 ManufacturingProcess09
## 3     3 ManufacturingProcess36 ManufacturingProcess36
## 4     4   BiologicalMaterial03 ManufacturingProcess17
## 5     5 ManufacturingProcess09 ManufacturingProcess34
## 6     6 ManufacturingProcess13   BiologicalMaterial06
## 7     7 ManufacturingProcess31 ManufacturingProcess11
## 8     8   BiologicalMaterial02 ManufacturingProcess06
## 9     9 ManufacturingProcess11 ManufacturingProcess13
## 10   10   BiologicalMaterial12   BiologicalMaterial03
shared <- intersect(nl_top10, lin_top10)
unique_nonlinear <- setdiff(nl_top10, lin_top10)
unique_linear <- setdiff(lin_top10, nl_top10)
list(shared = shared, only_nonlinear = unique_nonlinear, only_linear = unique_linear)
## $shared
## [1] "ManufacturingProcess32" "BiologicalMaterial06"   "ManufacturingProcess36"
## [4] "BiologicalMaterial03"   "ManufacturingProcess09" "ManufacturingProcess13"
## [7] "ManufacturingProcess11"
## 
## $only_nonlinear
## [1] "ManufacturingProcess31" "BiologicalMaterial02"   "BiologicalMaterial12"  
## 
## $only_linear
## [1] "ManufacturingProcess17" "ManufacturingProcess34" "ManufacturingProcess06"
table(Linear = ifelse(grepl("^Biological", lin_top10), "Biological", "Process"))
## Linear
## Biological    Process 
##          2          8

Neither group dominates the SVM’s list: six of its top ten are process predictors and four are biological, with ManufacturingProcess32 first and BiologicalMaterial06 second. Process predictors lead in the linear model’s list (eight of ten, two biological). Process predictors also lead in the MARS model, which uses only two of them. Because the SVM score is a one-predictor-at-a-time filter, the biological predictors rank high partly because they are correlated with each other and with yield individually; the elastic net, which holds the other predictors fixed, has only two biological predictors among its top ten.

The two top-ten lists share seven predictors: ManufacturingProcess32, 09, 36, 13 and 11, and BiologicalMaterial06 and 03. All three approaches (the SVM filter, MARS and the elastic net) place ManufacturingProcess32 first or near the top and ManufacturingProcess09 high. Three predictors are in the nonlinear top ten only: ManufacturingProcess31, BiologicalMaterial02 and BiologicalMaterial12. Three are in the linear top ten only: ManufacturingProcess17, 34 and 06. Since the lists are built from different measures (a filter on one predictor at a time against coefficients of a penalized multiple regression), differences between them partly reflect the measures and not only linearity versus nonlinearity.

(c) Relationships for the predictors unique to the nonlinear model

Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?

The plots use the original, un-imputed and un-scaled values, so the axes are in process units. The smooth curve is a loess smoother, as in the book’s featurePlot() examples. These plots are descriptive and were made after the test set evaluation; they are not used for tuning.

unique_plot <- head(unique_nonlinear, 6)
featurePlot(x = ChemicalManufacturingProcess[, unique_plot, drop = FALSE],
            y = ChemicalManufacturingProcess$Yield,
            plot = "scatter", type = c("p", "smooth"),
            labels = c("Predictor value", "Yield (%)"),
            between = list(x = 1, y = 1),
            layout = c(min(3, length(unique_plot)), ceiling(length(unique_plot) / 3)))

The ManufacturingProcess31 panel is dominated by run 108, so the same plot is repeated without that run:

no108 <- ChemicalManufacturingProcess[ChemicalManufacturingProcess$ManufacturingProcess31 > 0 &
                                      !is.na(ChemicalManufacturingProcess$ManufacturingProcess31), ]
featurePlot(x = no108[, "ManufacturingProcess31", drop = FALSE], y = no108$Yield,
            plot = "scatter", type = c("p", "smooth"),
            labels = c("Predictor value", "Yield (%)"))

round(sapply(unique_nonlinear, function(v)
  cor(ChemicalManufacturingProcess[[v]], ChemicalManufacturingProcess$Yield,
      use = "complete.obs")), 3)
## ManufacturingProcess31   BiologicalMaterial02   BiologicalMaterial12 
##                 -0.069                  0.482                  0.367
# the one run with ManufacturingProcess31 equal to zero
zero31 <- which(ChemicalManufacturingProcess$ManufacturingProcess31 == 0)
zero31
## [1] 108
ChemicalManufacturingProcess[zero31, c("Yield", "ManufacturingProcess31")]
##     Yield ManufacturingProcess31
## 108 39.75                      0
c(zeros_in_that_run = sum(process_predictors[zero31, ] == 0, na.rm = TRUE),
  in_training_set = zero31 %in% cmp_rows)
## zeros_in_that_run   in_training_set 
##                11                 1
# correlation with yield with and without that run
c(with_run = cor(ChemicalManufacturingProcess$ManufacturingProcess31,
                 ChemicalManufacturingProcess$Yield, use = "complete.obs"),
  without_run = cor(ChemicalManufacturingProcess$ManufacturingProcess31[-zero31],
                    ChemicalManufacturingProcess$Yield[-zero31], use = "complete.obs"))
##    with_run without_run 
## -0.06890417 -0.36587278
# how the two biological predictors relate to the other biological predictors
bio <- ChemicalManufacturingProcess[, grep("^Biological", names(ChemicalManufacturingProcess))]
round(cor(bio)[c("BiologicalMaterial02", "BiologicalMaterial12"),
               c("BiologicalMaterial01", "BiologicalMaterial03", "BiologicalMaterial06",
                 "BiologicalMaterial12", "BiologicalMaterial02")], 2)
##                      BiologicalMaterial01 BiologicalMaterial03
## BiologicalMaterial02                 0.74                 0.86
## BiologicalMaterial12                 0.52                 0.70
##                      BiologicalMaterial06 BiologicalMaterial12
## BiologicalMaterial02                 0.95                 0.78
## BiologicalMaterial12                 0.81                 1.00
##                      BiologicalMaterial02
## BiologicalMaterial02                 1.00
## BiologicalMaterial12                 0.78

The unique predictors are ManufacturingProcess31, BiologicalMaterial02 and BiologicalMaterial12.

ManufacturingProcess31 is a process predictor and can be adjusted. Almost every run has a value between about 68 and 73, so the plot is a narrow vertical band, and one run (run 108, yield 39.75) has the value 0 along with zeros in ten other process predictors. That one run bends the loess curve in the first panel so far that it leaves the plotted range, and a second run near 60 stretches the left end of the curve. Without run 108 (second plot) the smooth curve falls steadily from the low end of the band and flattens near its upper end, so higher values go with lower yield over that range. Run 108 pulls the overall correlation with yield to -0.07. Without it the correlation is -0.37, a clearly negative relationship: runs with a higher value tend to have lower yield. A linear model fitted with that run present sees a weak relationship, which is consistent with ManufacturingProcess31 missing from the elastic net’s top ten, although this plot cannot prove that is the reason. The run is in the training set, so it was seen by every model during fitting. In practice a run with eleven predictors at exactly zero looks like a recording or sensor problem and not a real operating point, and it should be checked in the plant records before it is trusted.

BiologicalMaterial02 rises steadily with yield: the smooth curve climbs from the lowest to the highest values (roughly 38 to 42.5 on the yield axis), a correlation of about 0.48, with a lot of scatter around it. BiologicalMaterial12 is curved: the smooth curve rises toward a peak near a value of 21 and then declines modestly at the high end, where there are relatively few runs (correlation about 0.37). That bend is the kind of shape a nonlinear model can follow and a straight line cannot, but with so few runs at the high end the downturn should be interpreted cautiously. Both biological predictors are strongly correlated with the other biological predictors (BiologicalMaterial02 with 06 at 0.95 and with 03 at 0.86, BiologicalMaterial12 with 06 at 0.81), so each is partly the same signal as BiologicalMaterial06 and 03, which both lists already rank highly. Their separate effects cannot be read from these plots.

Taken together, the plots give some intuition but not a recipe. The biological predictors cannot be changed, but higher BiologicalMaterial02 and a middle-to-high BiologicalMaterial12 go with higher yield on average, so they may be worth watching when raw material batches are screened, although the plots do not establish a threshold. The only unique process predictor, ManufacturingProcess31, points to something that could be controlled (lower values go with higher yield once the zero run is set aside), but the data are observational, so the way to confirm it is a designed experiment in the plant. With a 1% gain in yield worth about one hundred thousand dollars per batch, a confirmed improvement from one process setting would be worth that experiment.