# Loading required packages
library(AppliedPredictiveModeling)
library(caret)
library(mlbench)
library(earth)
library(kernlab)
library(nnet)
library(pls)
library(RANN)DATA 624 Homework 8
7.2
Friedman (1991) introduced several benchmark data sets created by simulation. One of these simulations used the following nonlinear equation to create data: where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
trainingData$x <- data.frame(trainingData$x)
featurePlot(trainingData$x, trainingData$y)testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)200 training points and a test set of 5,000 points to estimate the true error. X1 to X5 are used in the equation and X6 to X10 are noise.
Tune several models on these data. Which models appear to give the best performance? Does MARS select the informative predictors (those named X1 to X5)?
ctrl <- trainControl(method = "cv", number = 10)
set.seed(624)
knn_fit <- train(trainingData$x, trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10,
trControl = ctrl)
knn_fit$bestTune k
3 9
set.seed(624)
mars_fit <- train(trainingData$x, trainingData$y,
method = "earth",
tuneGrid = expand.grid(degree = 1:2, nprune = 2:38),
trControl = ctrl)
mars_fit$bestTune nprune degree
50 14 2
set.seed(624)
svm_fit <- train(trainingData$x, trainingData$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = ctrl)
svm_fit$bestTune sigma C
5 0.05607744 4
set.seed(624)
nnet_fit <- train(trainingData$x, trainingData$y,
method = "avNNet",
preProc = c("center", "scale"),
tuneGrid = expand.grid(decay = c(0, 0.01, 0.1),
size = 1:10, bag = FALSE),
linout = TRUE, trace = FALSE,
maxit = 500, MaxNWts = 500,
trControl = ctrl)Warning: executing %dopar% sequentially: no parallel backend registered
nnet_fit$bestTune size decay bag
4 4 0 FALSE
kNN, MARS, SVM and a neural net were tuned with 10 fold cross validation. The optimal settings are k = 9, a degree 2 MARS with 14 terms, an SVM with C = 4, and a neural net with 4 hidden units and no decay.
perf <- function(fit, name, x_new, y_new) {
cv <- getTrainPerf(fit)
te <- postResample(predict(fit, x_new), y_new)
data.frame(Model = name,
CV_RMSE = cv$TrainRMSE, CV_R2 = cv$TrainRsquared,
Test_RMSE = te["RMSE"], Test_R2 = te["Rsquared"])
}
comp <- rbind(perf(knn_fit, "kNN", testData$x, testData$y),
perf(mars_fit, "MARS", testData$x, testData$y),
perf(svm_fit, "SVM", testData$x, testData$y),
perf(nnet_fit, "Neural net", testData$x, testData$y))
rownames(comp) <- NULL
comp Model CV_RMSE CV_R2 Test_RMSE Test_R2
1 kNN 3.028589 0.6824162 3.117232 0.6556622
2 MARS 1.279383 0.9390664 1.172263 0.9448890
3 SVM 1.847956 0.8721190 2.044656 0.8308598
4 Neural net 1.919085 0.8612943 2.008473 0.8404012
MARS is the best model, with a test RMSE of 1.17 and R squared of 0.945. The neural net (2.00, 0.841) and SVM (2.04, 0.831) come next, and kNN is the weakest (3.12, 0.656). MARS fits the equation well because it is built from hinge terms and the degree 2 model can capture the X1 and X2 interaction. kNN uses all 10 predictors in its distances, so the 5 noise predictors blur which points are truly close.
summary(mars_fit$finalModel)Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
nprune=14)
coefficients
(Intercept) 21.905319
h(0.621722-X1) -15.726181
h(X1-0.621722) 9.234027
h(0.601063-X2) -18.253527
h(X2-0.601063) 10.448545
h(0.447442-X3) 9.700589
h(X3-0.606015) 12.674694
h(0.734892-X4) -9.863526
h(X4-0.734892) 10.297964
h(0.850094-X5) -5.324175
h(0.218266-X1) * h(X2-0.601063) -55.358726
h(X1-0.218266) * h(X2-0.601063) -29.100250
h(X1-0.621722) * h(X2-0.295997) -26.833129
h(0.649253-X1) * h(0.601063-X2) 27.120721
Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
Termination condition: Reached nk 21
Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
Number of terms at each degree of interaction: 1 9 4
GCV 1.62945 RSS 225.8601 GRSq 0.9338437 RSq 0.953688
varImp(mars_fit)earth variable importance
Overall
X1 100.00
X4 75.24
X2 48.74
X5 15.53
X3 0.00
Yes. The summary shows MARS used 5 of the 10 predictors, X1 to X5, and X6 to X10 are unused. X3 shows an importance of 0 only because caret rescales the scores so the lowest used predictor is 0, but it appears in two of the model terms.
7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and preprocessing steps as before and train several nonlinear regression models.
data(ChemicalManufacturingProcess)
cmp <- ChemicalManufacturingProcess
x <- cmp[, -1]
yield <- cmp$Yield
set.seed(624)
idx <- createDataPartition(yield, p = 0.8, list = FALSE)
x_train <- x[idx, ]
x_test <- x[-idx, ]
y_train <- yield[idx]
y_test <- yield[-idx]
pp <- c("knnImpute", "nzv", "center", "scale")Same split as Homework 7, 144 batches for training and 32 for testing, with the same imputation and preprocessing.
set.seed(624)
pls_cmp <- train(x_train, y_train,
method = "pls",
tuneLength = 15,
preProc = pp,
trControl = ctrl)The PLS model from 6.3 is refit so its top predictors can be compared in part b.
set.seed(624)
knn_cmp <- train(x_train, y_train,
method = "knn",
tuneLength = 10,
preProc = pp,
trControl = ctrl)
set.seed(624)
mars_cmp <- train(x_train, y_train,
method = "earth",
tuneGrid = expand.grid(degree = 1:2, nprune = 2:30),
preProc = pp,
trControl = ctrl)
set.seed(624)
svm_cmp <- train(x_train, y_train,
method = "svmRadial",
tuneLength = 14,
preProc = pp,
trControl = ctrl)
set.seed(624)
nnet_cmp <- train(x_train, y_train,
method = "avNNet",
tuneGrid = expand.grid(decay = c(0, 0.01, 0.1),
size = 1:5, bag = FALSE),
linout = TRUE, trace = FALSE,
maxit = 500, MaxNWts = 1000,
preProc = pp,
trControl = ctrl)a. Which nonlinear regression model gives the optimal resampling and test set performance?
comp_cmp <- rbind(perf(knn_cmp, "kNN", x_test, y_test),
perf(mars_cmp, "MARS", x_test, y_test),
perf(svm_cmp, "SVM", x_test, y_test),
perf(nnet_cmp, "Neural net", x_test, y_test),
perf(pls_cmp, "PLS", x_test, y_test))
rownames(comp_cmp) <- NULL
comp_cmp Model CV_RMSE CV_R2 Test_RMSE Test_R2
1 kNN 1.315749 0.5090465 1.402164 0.6129980
2 MARS 1.193540 0.5808758 1.282667 0.6705045
3 SVM 1.167241 0.5916503 1.146870 0.7474025
4 Neural net 1.403694 0.4615988 1.322341 0.6094226
5 PLS 1.368345 0.5231989 1.201579 0.6756127
The SVM is the optimal nonlinear model. It has the lowest resampled RMSE (1.167) and the best test results (RMSE 1.147, R squared 0.747). MARS has a slightly higher resampled R squared (0.597 vs 0.592), but its test RMSE is worse (1.288). The SVM also beats the PLS model from 6.3 on both resampled and test RMSE.
b. Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
imp_svm <- varImp(svm_cmp)
top_svm <- rownames(imp_svm$importance)[order(-imp_svm$importance$Overall)][1:10]
imp_svmloess r-squared variable importance
only 20 most important variables shown (out of 57)
Overall
ManufacturingProcess32 100.00
BiologicalMaterial06 89.32
ManufacturingProcess36 79.78
BiologicalMaterial03 77.48
ManufacturingProcess09 73.90
ManufacturingProcess13 73.23
ManufacturingProcess31 70.84
BiologicalMaterial02 66.91
ManufacturingProcess11 65.19
BiologicalMaterial12 64.94
ManufacturingProcess06 61.04
ManufacturingProcess17 53.06
ManufacturingProcess29 50.91
BiologicalMaterial11 49.10
BiologicalMaterial04 48.27
ManufacturingProcess33 47.62
ManufacturingProcess30 47.38
BiologicalMaterial01 40.69
BiologicalMaterial08 38.18
ManufacturingProcess25 34.07
table(substr(top_svm, 1, 10))
Biological Manufactur
4 6
6 of the top 10 are process predictors and 4 are biological, so the process predictors still lead, but less than in PLS (7 of 10). caret has no built in importance for SVMs, so it ranks each predictor by how well a loess curve of yield on that predictor alone fits.
imp_pls <- varImp(pls_cmp)
top_pls <- rownames(imp_pls$importance)[order(-imp_pls$importance$Overall)][1:10]
intersect(top_svm, top_pls)[1] "ManufacturingProcess32" "BiologicalMaterial06" "ManufacturingProcess36"
[4] "BiologicalMaterial03" "ManufacturingProcess09" "ManufacturingProcess13"
[7] "BiologicalMaterial02" "ManufacturingProcess11"
setdiff(top_svm, top_pls)[1] "ManufacturingProcess31" "BiologicalMaterial12"
setdiff(top_pls, top_svm)[1] "ManufacturingProcess17" "ManufacturingProcess06"
8 of the top 10 are shared with PLS, and ManufacturingProcess32 is first in both. BiologicalMaterial06 moves up to second in the SVM list. ManufacturingProcess31 and BiologicalMaterial12 are only in the SVM list, while ManufacturingProcess17 and 06 are only in the PLS list.
c. Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
unique_svm <- setdiff(top_svm, top_pls)
summary(cmp[, unique_svm]) ManufacturingProcess31 BiologicalMaterial12
Min. : 0.00 Min. :18.35
1st Qu.:70.10 1st Qu.:19.73
Median :70.80 Median :20.12
Mean :70.18 Mean :20.20
3rd Qu.:71.40 3rd Qu.:20.75
Max. :72.50 Max. :22.21
NA's :5
ManufacturingProcess31 has one batch at 0 while its first quartile is 70.1, so that batch is likely a recording error. I leave it out of the plot so it does not stretch the axis.
keep <- which(cmp$ManufacturingProcess31 > 0)
round(cor(cmp[keep, unique_svm], cmp$Yield[keep], use = "complete.obs"), 3) [,1]
ManufacturingProcess31 -0.366
BiologicalMaterial12 0.372
featurePlot(x = cmp[keep, unique_svm], y = cmp$Yield[keep],
plot = "scatter", type = c("p", "smooth"),
layout = c(2, 1))ManufacturingProcess31 has a negative relationship with yield (-0.37), and the curve shows yield dropping once it goes above about 68. It is a process predictor, so keeping it lower is worth testing. BiologicalMaterial12 (0.37) rises with yield up to about 21 and then dips slightly, a curved shape a linear model like PLS cannot fully use. It cannot be changed, but it could help screen raw material lots. These are correlations, so any change should be tested before assuming it improves yield.