Question: What is the relationship between the predictors and the response, and which predictors actually contain signal?
Following the book’s simulation, I use 200 observations for training and 5,000 new observations for testing. The larger test set lets me estimate prediction error more precisely. I use all ten predictors as inputs, even though the simulation tells us that only five are informative.
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
trainingData$x <- data.frame(trainingData$x)
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)
dim(trainingData$x)
## [1] 200 10
dim(testData$x)
## [1] 5000 10
featurePlot(x = trainingData$x, y = trainingData$y, plot = "scatter",
layout = c(5, 2))
Answer: X1 and X2 act together through the sine term, X3 has a curved effect, and X4 and X5 have linear effects. X6 through X10 are noise predictors. The scatterplots show one predictor at a time, so they do not fully show the interaction of X1 and X2.
Question: How do KNN, MARS, and a radial support vector machine compare?
I use the same ten cross-validation folds for each model and select
tuning values by the lowest cross-validation RMSE. KNN and the radial
SVM are sensitive to distance, so I center and scale their predictors.
Putting preprocessing in train() lets caret estimate it
separately in each fitting fold.
set.seed(200)
friedmanFolds <- createFolds(trainingData$y, k = 10, returnTrain = TRUE)
friedmanControl <- trainControl(method = "cv", number = 10,
index = friedmanFolds, allowParallel = FALSE)
I start with the book’s KNN example using
train(method = "knn") and tuneLength = 10. The
book’s example uses bootstrap resampling by default; here I specify
shared cross-validation folds for a fair comparison.
set.seed(200)
knnModel <- train(x = trainingData$x, y = trainingData$y,
method = "knn", preProcess = c("center", "scale"),
tuneLength = 10, trControl = friedmanControl,
metric = "RMSE")
knnModel$bestTune
## k
## 4 11
For MARS, degree controls whether the model can include
interactions, and nprune limits the number of retained
terms.
marsGrid <- expand.grid(degree = 1:2, nprune = c(4, 6, 8, 10, 12, 15))
set.seed(200)
marsModel <- train(x = trainingData$x, y = trainingData$y,
method = "earth", tuneGrid = marsGrid,
trControl = friedmanControl, metric = "RMSE")
marsModel$bestTune
## nprune degree
## 11 12 2
For the radial SVM, sigma controls how local the kernel
is, and C controls the penalty for errors outside the
allowed margin. I try a small grid of both values.
svmGrid <- expand.grid(sigma = c(0.01, 0.05, 0.1), C = c(1, 4, 16))
set.seed(200)
svmModel <- train(x = trainingData$x, y = trainingData$y,
method = "svmRadial", preProcess = c("center", "scale"),
tuneGrid = svmGrid, trControl = friedmanControl,
metric = "RMSE")
svmModel$bestTune
## sigma C
## 6 0.05 16
Question: Which models appear to perform best?
knnPred <- predict(knnModel, newdata = testData$x)
marsPred <- predict(marsModel, newdata = testData$x)
svmPred <- predict(svmModel, newdata = testData$x)
knnTest <- postResample(pred = knnPred, obs = testData$y)
marsTest <- postResample(pred = marsPred, obs = testData$y)
svmTest <- postResample(pred = svmPred, obs = testData$y)
friedmanResults <- data.frame(
Model = c("KNN", "MARS", "Radial SVM"),
CV_RMSE = c(min(knnModel$results$RMSE), min(marsModel$results$RMSE),
min(svmModel$results$RMSE)),
Test_RMSE = c(knnTest[["RMSE"]], marsTest[["RMSE"]], svmTest[["RMSE"]]),
Test_R2 = c(knnTest[["Rsquared"]], marsTest[["Rsquared"]], svmTest[["Rsquared"]])
)
knitr::kable(friedmanResults, digits = 3, row.names = FALSE)
| Model | CV_RMSE | Test_RMSE | Test_R2 |
|---|---|---|---|
| KNN | 3.044 | 3.122 | 0.669 |
| MARS | 1.233 | 1.280 | 0.934 |
| Radial SVM | 1.842 | 2.027 | 0.834 |
Answer: MARS had the lowest cross-validation RMSE. MARS had the lowest test RMSE (1.28). I use RMSE to compare prediction errors and report caret’s squared-correlation R² as a second measure.
Question: Does MARS select the informative predictors X1–X5?
# Each selected non-intercept term has a nonzero entry for its predictors.
selectedTerms <- marsModel$finalModel$dirs[
marsModel$finalModel$selected.terms, , drop = FALSE
]
selectedVariables <- colnames(trainingData$x)[colSums(abs(selectedTerms)) > 0]
signalVariables <- paste0("X", 1:5)
missedSignal <- setdiff(signalVariables, selectedVariables)
selectedNoise <- setdiff(selectedVariables, signalVariables)
data.frame(
Predictor = names(trainingData$x),
Informative = names(trainingData$x) %in% signalVariables,
Selected_by_MARS = names(trainingData$x) %in% selectedVariables
)
## Predictor Informative Selected_by_MARS
## 1 X1 TRUE TRUE
## 2 X2 TRUE TRUE
## 3 X3 TRUE TRUE
## 4 X4 TRUE TRUE
## 5 X5 TRUE TRUE
## 6 X6 FALSE FALSE
## 7 X7 FALSE FALSE
## 8 X8 FALSE FALSE
## 9 X9 FALSE FALSE
## 10 X10 FALSE FALSE
Answer: MARS selected X1, X2, X3, X4, X5. It recovered all five informative predictors. It did not retain any noise predictors.
I use the same 80/20 split as the step-by-step HW7, with seed 6243. The response is yield in percent, and the predictors measure the raw materials and the manufacturing process. As in HW7, the model preprocessing removes constant predictors, fills in missing predictors with fitting-set medians, and centers and scales the predictors. None of these steps uses the test set to estimate preprocessing values.
data("ChemicalManufacturingProcess", package = "AppliedPredictiveModeling")
manufacturing <- ChemicalManufacturingProcess
x_mfg <- manufacturing[, names(manufacturing) != "Yield"]
y_mfg <- manufacturing$Yield
set.seed(6243)
train_rows_mfg <- sort(sample.int(nrow(x_mfg), floor(0.8 * nrow(x_mfg))))
x_train_mfg <- x_mfg[train_rows_mfg, ]
x_test_mfg <- x_mfg[-train_rows_mfg, ]
y_train_mfg <- y_mfg[train_rows_mfg]
y_test_mfg <- y_mfg[-train_rows_mfg]
c(Training = nrow(x_train_mfg), Test = nrow(x_test_mfg),
Missing_predictor_values = sum(is.na(x_mfg)))
## Training Test Missing_predictor_values
## 140 36 106
set.seed(16243)
mfgFolds <- createFolds(y_train_mfg, k = 10, returnTrain = TRUE)
mfgControl <- trainControl(method = "cv", number = 10,
index = mfgFolds, allowParallel = FALSE)
mfgPrep <- c("zv", "medianImpute", "center", "scale")
I tune KNN, MARS, and radial SVM on the manufacturing data using the same folds. I also refit the HW7 ridge baseline on the same split and folds. Ridge is a linear reference, not a candidate for choosing the best nonlinear model.
set.seed(6243)
mfgKnn <- train(x = x_train_mfg, y = y_train_mfg,
method = "knn", preProcess = mfgPrep, tuneLength = 10,
trControl = mfgControl, metric = "RMSE")
mfgKnn$bestTune
## k
## 1 5
set.seed(6243)
mfgMars <- train(x = x_train_mfg, y = y_train_mfg,
method = "earth", preProcess = mfgPrep, tuneGrid = marsGrid,
trControl = mfgControl, metric = "RMSE")
mfgMars$bestTune
## nprune degree
## 1 4 1
mfgSvmGrid <- expand.grid(sigma = c(0.005, 0.01, 0.025), C = c(0.5, 2, 8))
set.seed(6243)
mfgSvm <- train(x = x_train_mfg, y = y_train_mfg,
method = "svmRadial", preProcess = mfgPrep,
tuneGrid = mfgSvmGrid, trControl = mfgControl,
metric = "RMSE")
mfgSvm$bestTune
## sigma C
## 3 0.005 8
ridgeGrid <- expand.grid(alpha = 0, lambda = 10^seq(-3, 2, length.out = 26))
set.seed(16243)
mfgRidge <- train(x = x_train_mfg, y = y_train_mfg,
method = "glmnet", preProcess = mfgPrep,
tuneGrid = ridgeGrid, trControl = mfgControl, metric = "RMSE",
standardize = FALSE,
lambda = sort(ridgeGrid$lambda, decreasing = TRUE))
mfgRidge$bestTune
## alpha lambda
## 18 0 2.511886
mfgKnnTest <- postResample(predict(mfgKnn, x_test_mfg), y_test_mfg)
mfgMarsTest <- postResample(predict(mfgMars, x_test_mfg), y_test_mfg)
mfgSvmTest <- postResample(predict(mfgSvm, x_test_mfg), y_test_mfg)
mfgRidgeTest <- postResample(predict(mfgRidge, x_test_mfg), y_test_mfg)
mfgResults <- data.frame(
Model = c("KNN", "MARS", "Radial SVM", "Ridge"),
CV_RMSE = c(min(mfgKnn$results$RMSE), min(mfgMars$results$RMSE),
min(mfgSvm$results$RMSE), min(mfgRidge$results$RMSE)),
Test_RMSE = c(mfgKnnTest[["RMSE"]], mfgMarsTest[["RMSE"]],
mfgSvmTest[["RMSE"]], mfgRidgeTest[["RMSE"]]),
Test_R2 = c(mfgKnnTest[["Rsquared"]], mfgMarsTest[["Rsquared"]],
mfgSvmTest[["Rsquared"]], mfgRidgeTest[["Rsquared"]])
)
knitr::kable(mfgResults, digits = 3, row.names = FALSE)
| Model | CV_RMSE | Test_RMSE | Test_R2 |
|---|---|---|---|
| KNN | 1.289 | 1.385 | 0.534 |
| MARS | 1.211 | 1.093 | 0.688 |
| Radial SVM | 1.175 | 1.023 | 0.717 |
| Ridge | 1.296 | 1.277 | 0.661 |
bestCV <- which.min(mfgResults$CV_RMSE[1:3])
bestTest <- which.min(mfgResults$Test_RMSE[1:3])
nonlinearModels <- list(mfgKnn, mfgMars, mfgSvm)
bestNonlinear <- nonlinearModels[[bestCV]]
Answer: Among the nonlinear models, Radial SVM had the best resampling performance (RMSE 1.175 percentage points). Radial SVM had the best held-out test RMSE (1.023 percentage points). The ridge test RMSE was 1.277. I choose the nonlinear model for the next parts using cross-validation, so the test set remains an evaluation rather than a tuning tool.
For the nonlinear winner, I use model-based importance if MARS wins. KNN and radial SVM do not have a model-specific importance calculation in caret, so if one of them wins, I measure how much its five-fold RMSE increases when each predictor is shuffled in validation rows. I keep the selected tuning values fixed for this exploratory ranking. For the HW7 ridge model, I use the same absolute standardized-coefficient ranking as before. The numerical scores are not on the same scale, so I compare the ranked names rather than the magnitudes.
if (bestCV == 2) {
nonlinearImportance <- varImp(bestNonlinear, scale = FALSE)$importance
} else {
# Estimate a prediction-based ranking using only the training set.
set.seed(71)
importanceFolds <- createFolds(y_train_mfg, k = 5)
foldIncrease <- matrix(0, nrow = length(importanceFolds),
ncol = ncol(x_train_mfg))
for (i in seq_along(importanceFolds)) {
valid <- importanceFolds[[i]]
foldModel <- train(
x = x_train_mfg[-valid, ], y = y_train_mfg[-valid],
method = bestNonlinear$method, preProcess = mfgPrep,
tuneGrid = bestNonlinear$bestTune, trControl = trainControl(method = "none")
)
foldData <- x_train_mfg[valid, ]
actual <- y_train_mfg[valid]
baseline <- sqrt(mean((predict(foldModel, foldData) - actual)^2))
for (j in seq_len(ncol(foldData))) {
shuffled <- foldData
shuffled[[j]] <- sample(shuffled[[j]])
shuffledRMSE <- sqrt(mean((predict(foldModel, shuffled) - actual)^2))
foldIncrease[i, j] <- shuffledRMSE - baseline
}
}
nonlinearImportance <- data.frame(Overall = colMeans(foldIncrease),
row.names = names(x_train_mfg))
}
ridgeImportance <- varImp(mfgRidge, scale = FALSE)$importance
nlNames <- rownames(nonlinearImportance)[order(-nonlinearImportance$Overall)]
if (bestCV == 2) {
# A zero score means MARS did not give the predictor positive importance.
nlNames <- nlNames[nonlinearImportance[nlNames, "Overall"] > 0]
}
linearNames <- rownames(ridgeImportance)[order(-ridgeImportance$Overall)]
topNonlinear <- head(nlNames, 10)
topLinear <- head(linearNames, 10)
common <- intersect(topNonlinear, topLinear)
uniqueNonlinear <- setdiff(topNonlinear, topLinear)
nProcess <- sum(grepl("^ManufacturingProcess", topNonlinear))
knitr::kable(data.frame(
Rank = 1:10,
Nonlinear = c(topNonlinear, rep("—", 10 - length(topNonlinear))),
Ridge = topLinear
), row.names = FALSE)
| Rank | Nonlinear | Ridge |
|---|---|---|
| 1 | ManufacturingProcess32 | ManufacturingProcess32 |
| 2 | ManufacturingProcess04 | ManufacturingProcess09 |
| 3 | ManufacturingProcess09 | ManufacturingProcess13 |
| 4 | ManufacturingProcess36 | ManufacturingProcess36 |
| 5 | ManufacturingProcess37 | ManufacturingProcess17 |
| 6 | ManufacturingProcess11 | ManufacturingProcess33 |
| 7 | ManufacturingProcess28 | ManufacturingProcess12 |
| 8 | ManufacturingProcess17 | ManufacturingProcess06 |
| 9 | ManufacturingProcess12 | ManufacturingProcess37 |
| 10 | ManufacturingProcess13 | ManufacturingProcess34 |
Answer: The nonlinear list contains 10 manufacturing process predictors and 0 biological predictors. Manufacturing process predictors dominate by count. There are 7 overlapping predictors: ManufacturingProcess32, ManufacturingProcess09, ManufacturingProcess36, ManufacturingProcess37, ManufacturingProcess17, ManufacturingProcess12, ManufacturingProcess13. The two models can rank correlated predictors differently, and the data contain more process variables to begin with.
I plot the predictors on the nonlinear list that do not appear in ridge’s top ten. I use actual measured values from the training set rather than displaying imputed medians as observations. The smoother shows a marginal pattern, not the effect of changing a variable while holding everything else constant.
if (length(uniqueNonlinear) > 0) {
oldPar <- par(mfrow = c(ceiling(length(uniqueNonlinear) / 3),
min(3, length(uniqueNonlinear))),
mar = c(4, 4, 2.5, 0.5))
for (variable in uniqueNonlinear) {
observed <- !is.na(x_train_mfg[[variable]])
x <- x_train_mfg[[variable]][observed]
y <- y_train_mfg[observed]
plot(x, y, pch = 19, cex = 0.55, col = "steelblue",
xlab = "Measured predictor", ylab = "Yield (%)",
main = variable, cex.main = 0.75)
lines(lowess(x, y), col = "darkorange3", lwd = 2)
}
par(oldPar)
uniqueCor <- sapply(uniqueNonlinear, function(variable) {
cor(x_train_mfg[[variable]], y_train_mfg, use = "complete.obs")
})
knitr::kable(data.frame(Predictor = uniqueNonlinear,
Correlation_with_yield = unname(uniqueCor)),
digits = 3, row.names = FALSE)
} else {
print("There are no unique nonlinear predictors in the ranked lists.")
}
| Predictor | Correlation_with_yield |
|---|---|
| ManufacturingProcess04 | -0.257 |
| ManufacturingProcess11 | 0.308 |
| ManufacturingProcess28 | 0.278 |
Answer: Among the unique predictors, ManufacturingProcess11 had the strongest straight-line association with yield (r = 0.308). A low correlation does not rule out a curve or an interaction. For a process variable, I would use a visible pattern to identify settings to investigate in controlled trials. For a biological variable, I would use the pattern to flag raw-material batches for closer monitoring. These plots alone do not show that changing a setting will improve yield.
On this run, the smoother for ManufacturingProcess04 falls through much of the observed range and then turns slightly upward. ManufacturingProcess11 shows a generally upward trend, especially at higher observed values. ManufacturingProcess28 has two separated groups of observations: one near zero and one near 10–11. I would compare those observed groups rather than interpreting the line drawn across the gap.
Kuhn, M., & Johnson, K. (2013). Applied Predictive Modeling. Springer. Chapter 7, Section 7.5 and Exercises 7.2 and 7.5, pp. 169–171. The simulation, model-fitting calls, and test scoring follow the book’s examples, with tuning grids and shared cross-validation folds adapted for this assignment. Companion Chapter 7 R code.