Exercise 7.2: Friedman simulation

Step 1: Simulate and examine the data

Question: What is the relationship between the predictors and the response, and which predictors actually contain signal?

Following the book’s simulation, I use 200 observations for training and 5,000 new observations for testing. The larger test set lets me estimate prediction error more precisely. I use all ten predictors as inputs, even though the simulation tells us that only five are informative.

set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
trainingData$x <- data.frame(trainingData$x)

testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)

dim(trainingData$x)
## [1] 200  10
dim(testData$x)
## [1] 5000   10
featurePlot(x = trainingData$x, y = trainingData$y, plot = "scatter",
            layout = c(5, 2))

Answer: X1 and X2 act together through the sine term, X3 has a curved effect, and X4 and X5 have linear effects. X6 through X10 are noise predictors. The scatterplots show one predictor at a time, so they do not fully show the interaction of X1 and X2.

Step 2: Tune several nonlinear models

Question: How do KNN, MARS, and a radial support vector machine compare?

I use the same ten cross-validation folds for each model and select tuning values by the lowest cross-validation RMSE. KNN and the radial SVM are sensitive to distance, so I center and scale their predictors. Putting preprocessing in train() lets caret estimate it separately in each fitting fold.

set.seed(200)
friedmanFolds <- createFolds(trainingData$y, k = 10, returnTrain = TRUE)
friedmanControl <- trainControl(method = "cv", number = 10,
                                index = friedmanFolds, allowParallel = FALSE)

I start with the book’s KNN example using train(method = "knn") and tuneLength = 10. The book’s example uses bootstrap resampling by default; here I specify shared cross-validation folds for a fair comparison.

set.seed(200)
knnModel <- train(x = trainingData$x, y = trainingData$y,
                  method = "knn", preProcess = c("center", "scale"),
                  tuneLength = 10, trControl = friedmanControl,
                  metric = "RMSE")
knnModel$bestTune
##    k
## 4 11

For MARS, degree controls whether the model can include interactions, and nprune limits the number of retained terms.

marsGrid <- expand.grid(degree = 1:2, nprune = c(4, 6, 8, 10, 12, 15))
set.seed(200)
marsModel <- train(x = trainingData$x, y = trainingData$y,
                   method = "earth", tuneGrid = marsGrid,
                   trControl = friedmanControl, metric = "RMSE")
marsModel$bestTune
##    nprune degree
## 11     12      2

For the radial SVM, sigma controls how local the kernel is, and C controls the penalty for errors outside the allowed margin. I try a small grid of both values.

svmGrid <- expand.grid(sigma = c(0.01, 0.05, 0.1), C = c(1, 4, 16))
set.seed(200)
svmModel <- train(x = trainingData$x, y = trainingData$y,
                  method = "svmRadial", preProcess = c("center", "scale"),
                  tuneGrid = svmGrid, trControl = friedmanControl,
                  metric = "RMSE")
svmModel$bestTune
##   sigma  C
## 6  0.05 16

Step 3: Compare resampling and test performance

Question: Which models appear to perform best?

knnPred <- predict(knnModel, newdata = testData$x)
marsPred <- predict(marsModel, newdata = testData$x)
svmPred <- predict(svmModel, newdata = testData$x)

knnTest <- postResample(pred = knnPred, obs = testData$y)
marsTest <- postResample(pred = marsPred, obs = testData$y)
svmTest <- postResample(pred = svmPred, obs = testData$y)

friedmanResults <- data.frame(
  Model = c("KNN", "MARS", "Radial SVM"),
  CV_RMSE = c(min(knnModel$results$RMSE), min(marsModel$results$RMSE),
              min(svmModel$results$RMSE)),
  Test_RMSE = c(knnTest[["RMSE"]], marsTest[["RMSE"]], svmTest[["RMSE"]]),
  Test_R2 = c(knnTest[["Rsquared"]], marsTest[["Rsquared"]], svmTest[["Rsquared"]])
)
knitr::kable(friedmanResults, digits = 3, row.names = FALSE)
Model CV_RMSE Test_RMSE Test_R2
KNN 3.044 3.122 0.669
MARS 1.233 1.280 0.934
Radial SVM 1.842 2.027 0.834

Answer: MARS had the lowest cross-validation RMSE. MARS had the lowest test RMSE (1.28). I use RMSE to compare prediction errors and report caret’s squared-correlation R² as a second measure.

Step 4: Check which predictors MARS selected

Question: Does MARS select the informative predictors X1–X5?

# Each selected non-intercept term has a nonzero entry for its predictors.
selectedTerms <- marsModel$finalModel$dirs[
  marsModel$finalModel$selected.terms, , drop = FALSE
]
selectedVariables <- colnames(trainingData$x)[colSums(abs(selectedTerms)) > 0]

signalVariables <- paste0("X", 1:5)
missedSignal <- setdiff(signalVariables, selectedVariables)
selectedNoise <- setdiff(selectedVariables, signalVariables)

data.frame(
  Predictor = names(trainingData$x),
  Informative = names(trainingData$x) %in% signalVariables,
  Selected_by_MARS = names(trainingData$x) %in% selectedVariables
)
##    Predictor Informative Selected_by_MARS
## 1         X1        TRUE             TRUE
## 2         X2        TRUE             TRUE
## 3         X3        TRUE             TRUE
## 4         X4        TRUE             TRUE
## 5         X5        TRUE             TRUE
## 6         X6       FALSE            FALSE
## 7         X7       FALSE            FALSE
## 8         X8       FALSE            FALSE
## 9         X9       FALSE            FALSE
## 10       X10       FALSE            FALSE

Answer: MARS selected X1, X2, X3, X4, X5. It recovered all five informative predictors. It did not retain any noise predictors.

Exercise 7.5: Chemical manufacturing yield

Step 1: Reuse the Exercise 6.3 data and split

I use the same 80/20 split as the step-by-step HW7, with seed 6243. The response is yield in percent, and the predictors measure the raw materials and the manufacturing process. As in HW7, the model preprocessing removes constant predictors, fills in missing predictors with fitting-set medians, and centers and scales the predictors. None of these steps uses the test set to estimate preprocessing values.

data("ChemicalManufacturingProcess", package = "AppliedPredictiveModeling")
manufacturing <- ChemicalManufacturingProcess
x_mfg <- manufacturing[, names(manufacturing) != "Yield"]
y_mfg <- manufacturing$Yield

set.seed(6243)
train_rows_mfg <- sort(sample.int(nrow(x_mfg), floor(0.8 * nrow(x_mfg))))
x_train_mfg <- x_mfg[train_rows_mfg, ]
x_test_mfg <- x_mfg[-train_rows_mfg, ]
y_train_mfg <- y_mfg[train_rows_mfg]
y_test_mfg <- y_mfg[-train_rows_mfg]

c(Training = nrow(x_train_mfg), Test = nrow(x_test_mfg),
  Missing_predictor_values = sum(is.na(x_mfg)))
##                 Training                     Test Missing_predictor_values 
##                      140                       36                      106
set.seed(16243)
mfgFolds <- createFolds(y_train_mfg, k = 10, returnTrain = TRUE)
mfgControl <- trainControl(method = "cv", number = 10,
                           index = mfgFolds, allowParallel = FALSE)
mfgPrep <- c("zv", "medianImpute", "center", "scale")

(a) Which nonlinear model performs best?

I tune KNN, MARS, and radial SVM on the manufacturing data using the same folds. I also refit the HW7 ridge baseline on the same split and folds. Ridge is a linear reference, not a candidate for choosing the best nonlinear model.

set.seed(6243)
mfgKnn <- train(x = x_train_mfg, y = y_train_mfg,
                method = "knn", preProcess = mfgPrep, tuneLength = 10,
                trControl = mfgControl, metric = "RMSE")
mfgKnn$bestTune
##   k
## 1 5
set.seed(6243)
mfgMars <- train(x = x_train_mfg, y = y_train_mfg,
                 method = "earth", preProcess = mfgPrep, tuneGrid = marsGrid,
                 trControl = mfgControl, metric = "RMSE")
mfgMars$bestTune
##   nprune degree
## 1      4      1
mfgSvmGrid <- expand.grid(sigma = c(0.005, 0.01, 0.025), C = c(0.5, 2, 8))
set.seed(6243)
mfgSvm <- train(x = x_train_mfg, y = y_train_mfg,
                method = "svmRadial", preProcess = mfgPrep,
                tuneGrid = mfgSvmGrid, trControl = mfgControl,
                metric = "RMSE")
mfgSvm$bestTune
##   sigma C
## 3 0.005 8
ridgeGrid <- expand.grid(alpha = 0, lambda = 10^seq(-3, 2, length.out = 26))
set.seed(16243)
mfgRidge <- train(x = x_train_mfg, y = y_train_mfg,
                  method = "glmnet", preProcess = mfgPrep,
                  tuneGrid = ridgeGrid, trControl = mfgControl, metric = "RMSE",
                  standardize = FALSE,
                  lambda = sort(ridgeGrid$lambda, decreasing = TRUE))
mfgRidge$bestTune
##    alpha   lambda
## 18     0 2.511886
mfgKnnTest <- postResample(predict(mfgKnn, x_test_mfg), y_test_mfg)
mfgMarsTest <- postResample(predict(mfgMars, x_test_mfg), y_test_mfg)
mfgSvmTest <- postResample(predict(mfgSvm, x_test_mfg), y_test_mfg)
mfgRidgeTest <- postResample(predict(mfgRidge, x_test_mfg), y_test_mfg)

mfgResults <- data.frame(
  Model = c("KNN", "MARS", "Radial SVM", "Ridge"),
  CV_RMSE = c(min(mfgKnn$results$RMSE), min(mfgMars$results$RMSE),
              min(mfgSvm$results$RMSE), min(mfgRidge$results$RMSE)),
  Test_RMSE = c(mfgKnnTest[["RMSE"]], mfgMarsTest[["RMSE"]],
                mfgSvmTest[["RMSE"]], mfgRidgeTest[["RMSE"]]),
  Test_R2 = c(mfgKnnTest[["Rsquared"]], mfgMarsTest[["Rsquared"]],
              mfgSvmTest[["Rsquared"]], mfgRidgeTest[["Rsquared"]])
)
knitr::kable(mfgResults, digits = 3, row.names = FALSE)
Model CV_RMSE Test_RMSE Test_R2
KNN 1.289 1.385 0.534
MARS 1.211 1.093 0.688
Radial SVM 1.175 1.023 0.717
Ridge 1.296 1.277 0.661
bestCV <- which.min(mfgResults$CV_RMSE[1:3])
bestTest <- which.min(mfgResults$Test_RMSE[1:3])
nonlinearModels <- list(mfgKnn, mfgMars, mfgSvm)
bestNonlinear <- nonlinearModels[[bestCV]]

Answer: Among the nonlinear models, Radial SVM had the best resampling performance (RMSE 1.175 percentage points). Radial SVM had the best held-out test RMSE (1.023 percentage points). The ridge test RMSE was 1.277. I choose the nonlinear model for the next parts using cross-validation, so the test set remains an evaluation rather than a tuning tool.

(b) Which predictors are most important?

For the nonlinear winner, I use model-based importance if MARS wins. KNN and radial SVM do not have a model-specific importance calculation in caret, so if one of them wins, I measure how much its five-fold RMSE increases when each predictor is shuffled in validation rows. I keep the selected tuning values fixed for this exploratory ranking. For the HW7 ridge model, I use the same absolute standardized-coefficient ranking as before. The numerical scores are not on the same scale, so I compare the ranked names rather than the magnitudes.

if (bestCV == 2) {
  nonlinearImportance <- varImp(bestNonlinear, scale = FALSE)$importance
} else {
  # Estimate a prediction-based ranking using only the training set.
  set.seed(71)
  importanceFolds <- createFolds(y_train_mfg, k = 5)
  foldIncrease <- matrix(0, nrow = length(importanceFolds),
                         ncol = ncol(x_train_mfg))
  for (i in seq_along(importanceFolds)) {
    valid <- importanceFolds[[i]]
    foldModel <- train(
      x = x_train_mfg[-valid, ], y = y_train_mfg[-valid],
      method = bestNonlinear$method, preProcess = mfgPrep,
      tuneGrid = bestNonlinear$bestTune, trControl = trainControl(method = "none")
    )
    foldData <- x_train_mfg[valid, ]
    actual <- y_train_mfg[valid]
    baseline <- sqrt(mean((predict(foldModel, foldData) - actual)^2))
    for (j in seq_len(ncol(foldData))) {
      shuffled <- foldData
      shuffled[[j]] <- sample(shuffled[[j]])
      shuffledRMSE <- sqrt(mean((predict(foldModel, shuffled) - actual)^2))
      foldIncrease[i, j] <- shuffledRMSE - baseline
    }
  }
  nonlinearImportance <- data.frame(Overall = colMeans(foldIncrease),
                                     row.names = names(x_train_mfg))
}

ridgeImportance <- varImp(mfgRidge, scale = FALSE)$importance
nlNames <- rownames(nonlinearImportance)[order(-nonlinearImportance$Overall)]
if (bestCV == 2) {
  # A zero score means MARS did not give the predictor positive importance.
  nlNames <- nlNames[nonlinearImportance[nlNames, "Overall"] > 0]
}
linearNames <- rownames(ridgeImportance)[order(-ridgeImportance$Overall)]
topNonlinear <- head(nlNames, 10)
topLinear <- head(linearNames, 10)
common <- intersect(topNonlinear, topLinear)
uniqueNonlinear <- setdiff(topNonlinear, topLinear)
nProcess <- sum(grepl("^ManufacturingProcess", topNonlinear))

knitr::kable(data.frame(
  Rank = 1:10,
  Nonlinear = c(topNonlinear, rep("—", 10 - length(topNonlinear))),
  Ridge = topLinear
), row.names = FALSE)
Rank Nonlinear Ridge
1 ManufacturingProcess32 ManufacturingProcess32
2 ManufacturingProcess04 ManufacturingProcess09
3 ManufacturingProcess09 ManufacturingProcess13
4 ManufacturingProcess36 ManufacturingProcess36
5 ManufacturingProcess37 ManufacturingProcess17
6 ManufacturingProcess11 ManufacturingProcess33
7 ManufacturingProcess28 ManufacturingProcess12
8 ManufacturingProcess17 ManufacturingProcess06
9 ManufacturingProcess12 ManufacturingProcess37
10 ManufacturingProcess13 ManufacturingProcess34

Answer: The nonlinear list contains 10 manufacturing process predictors and 0 biological predictors. Manufacturing process predictors dominate by count. There are 7 overlapping predictors: ManufacturingProcess32, ManufacturingProcess09, ManufacturingProcess36, ManufacturingProcess37, ManufacturingProcess17, ManufacturingProcess12, ManufacturingProcess13. The two models can rank correlated predictors differently, and the data contain more process variables to begin with.

(c) What do the unique nonlinear predictors tell us about yield?

I plot the predictors on the nonlinear list that do not appear in ridge’s top ten. I use actual measured values from the training set rather than displaying imputed medians as observations. The smoother shows a marginal pattern, not the effect of changing a variable while holding everything else constant.

if (length(uniqueNonlinear) > 0) {
  oldPar <- par(mfrow = c(ceiling(length(uniqueNonlinear) / 3),
                         min(3, length(uniqueNonlinear))),
                mar = c(4, 4, 2.5, 0.5))
  for (variable in uniqueNonlinear) {
    observed <- !is.na(x_train_mfg[[variable]])
    x <- x_train_mfg[[variable]][observed]
    y <- y_train_mfg[observed]
    plot(x, y, pch = 19, cex = 0.55, col = "steelblue",
         xlab = "Measured predictor", ylab = "Yield (%)",
         main = variable, cex.main = 0.75)
    lines(lowess(x, y), col = "darkorange3", lwd = 2)
  }
  par(oldPar)

  uniqueCor <- sapply(uniqueNonlinear, function(variable) {
    cor(x_train_mfg[[variable]], y_train_mfg, use = "complete.obs")
  })
  knitr::kable(data.frame(Predictor = uniqueNonlinear,
                         Correlation_with_yield = unname(uniqueCor)),
               digits = 3, row.names = FALSE)
} else {
  print("There are no unique nonlinear predictors in the ranked lists.")
}

Predictor Correlation_with_yield
ManufacturingProcess04 -0.257
ManufacturingProcess11 0.308
ManufacturingProcess28 0.278

Answer: Among the unique predictors, ManufacturingProcess11 had the strongest straight-line association with yield (r = 0.308). A low correlation does not rule out a curve or an interaction. For a process variable, I would use a visible pattern to identify settings to investigate in controlled trials. For a biological variable, I would use the pattern to flag raw-material batches for closer monitoring. These plots alone do not show that changing a setting will improve yield.

On this run, the smoother for ManufacturingProcess04 falls through much of the observed range and then turns slightly upward. ManufacturingProcess11 shows a generally upward trend, especially at higher observed values. ManufacturingProcess28 has two separated groups of observations: one near zero and one near 10–11. I would compare those observed groups rather than interpreting the line drawn across the gap.

Reference

Kuhn, M., & Johnson, K. (2013). Applied Predictive Modeling. Springer. Chapter 7, Section 7.5 and Exercises 7.2 and 7.5, pp. 169–171. The simulation, model-fitting calls, and test scoring follow the book’s examples, with tuning grids and shared cross-validation folds adapted for this assignment. Companion Chapter 7 R code.