7.2. Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data: y = 10 sin(πx1x2) + 20(x3 − 0.5)2 + 10x4 + 5x5 + N (0, σ2) where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
library(mlbench)library(caret)
Loading required package: ggplot2
Loading required package: lattice
library(earth)
Loading required package: Formula
Loading required package: plotmo
Loading required package: plotrix
library(kernlab)
Attaching package: 'kernlab'
The following object is masked from 'package:ggplot2':
alpha
library(nnet)
set.seed(200)trainingData <-mlbench.friedman1(200, sd =1)## We convert the 'x' data from a matrix to a data frame## One reason is that this will give the columns names.trainingData$x <-data.frame(trainingData$x)## Look at the data usingfeaturePlot(trainingData$x, trainingData$y)
## or other methods.## This creates a list with a vector 'y' and a matrix## of predictors 'x'. Also simulate a large test set to## estimate the true error rate with good precision:testData <-mlbench.friedman1(5000, sd =1)testData$x <-data.frame(testData$x)
k-Nearest Neighbors
200 samples
10 predictor
Pre-processing: centered (10), scaled (10)
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
Resampling results across tuning parameters:
k RMSE Rsquared MAE
5 3.466085 0.5121775 2.816838
7 3.349428 0.5452823 2.727410
9 3.264276 0.5785990 2.660026
11 3.214216 0.6024244 2.603767
13 3.196510 0.6176570 2.591935
15 3.184173 0.6305506 2.577482
17 3.183130 0.6425367 2.567787
19 3.198752 0.6483184 2.592683
21 3.188993 0.6611428 2.588787
23 3.200458 0.6638353 2.604529
RMSE was used to select the optimal model using the smallest value.
The final value used for the model was k = 17.
knnPred <-predict(knnModel, newdata = testData$x)## The function 'postResample' can be used to get the test set performance valuespostResample(pred = knnPred, obs = testData$y)
RMSE Rsquared MAE
3.2040595 0.6819919 2.5683461
Which models appear to give the best performance? Does MARS select the informative predictors (those named X1–X5)?
Support Vector Machines with Radial Basis Function Kernel
200 samples
10 predictor
Pre-processing: centered (10), scaled (10)
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
Resampling results across tuning parameters:
C RMSE Rsquared MAE
0.25 2.635010 0.7685188 2.074977
0.50 2.423373 0.7839086 1.902162
1.00 2.284133 0.8001542 1.791776
2.00 2.196624 0.8126474 1.713560
4.00 2.143035 0.8209820 1.668024
8.00 2.119156 0.8246312 1.649384
16.00 2.117438 0.8248677 1.648568
32.00 2.117438 0.8248677 1.648568
64.00 2.117438 0.8248677 1.648568
128.00 2.117438 0.8248677 1.648568
Tuning parameter 'sigma' was held constant at a value of 0.06299324
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were sigma = 0.06299324 and C = 16.
The MARS model appears to give the best performance. According to the RMSE values, the MARS model had the lowest RMSE followed by the SVM model and then the Neural Network model. The example KNN model actually performed the worst out of the 4 models. The MARS model does select the informative predictors (those named X1–X5) according to the summary statistics above.
7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
Support Vector Machines with Radial Basis Function Kernel
144 samples
56 predictor
No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 130, 129, 129, 130, 131, 131, ...
Resampling results across tuning parameters:
C RMSE Rsquared MAE
0.25 0.7907526 0.5150952 0.6459562
0.50 0.7308700 0.5555113 0.6006713
1.00 0.6906014 0.5855178 0.5631447
2.00 0.6597609 0.6101799 0.5340240
4.00 0.6473585 0.6163341 0.5179436
8.00 0.6301217 0.6339842 0.5056210
16.00 0.6283767 0.6362071 0.5041480
32.00 0.6283767 0.6362071 0.5041480
64.00 0.6283767 0.6362071 0.5041480
128.00 0.6283767 0.6362071 0.5041480
Tuning parameter 'sigma' was held constant at a value of 0.01473086
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were sigma = 0.01473086 and C = 16.
k-Nearest Neighbors
144 samples
56 predictor
No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 130, 129, 129, 130, 131, 131, ...
Resampling results across tuning parameters:
k RMSE Rsquared MAE
5 0.7582754 0.4689249 0.5958651
7 0.7567786 0.4860430 0.5988680
9 0.7656574 0.4791745 0.6129237
11 0.7676219 0.4867971 0.6218451
13 0.7671042 0.4938682 0.6143599
15 0.7772014 0.4762563 0.6203014
17 0.7932353 0.4571005 0.6377444
19 0.7895134 0.4608770 0.6348365
21 0.7889588 0.4668767 0.6347396
23 0.7975105 0.4598221 0.6427061
RMSE was used to select the optimal model using the smallest value.
The final value used for the model was k = 7.
resamps <-resamples(list(SVM = svm_model, MARS = mars_model, KNN = knn_model, NN = nn_model))summary(resamps)
Call:
summary.resamples(object = resamps)
Models: SVM, MARS, KNN, NN
Number of resamples: 10
MAE
Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
SVM 0.3779863 0.4297056 0.4962653 0.5041480 0.5741598 0.6677082 0
MARS 0.3982727 0.4786543 0.5299589 0.5412634 0.5758744 0.8070333 0
KNN 0.4299905 0.5192368 0.5889512 0.5988680 0.6600127 0.7662737 0
NN 0.4104028 0.4701289 0.6019592 0.5945290 0.6669310 0.8840452 0
RMSE
Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
SVM 0.4701273 0.5624376 0.6163458 0.6283767 0.7164603 0.8132034 0
MARS 0.4823963 0.6314224 0.6804129 0.6710648 0.7057017 0.9084705 0
KNN 0.5736776 0.7081903 0.7480029 0.7567786 0.8249372 0.9086932 0
NN 0.4935170 0.6020204 0.7315897 0.7426006 0.8035316 1.0854727 0
Rsquared
Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
SVM 0.36239724 0.6000560 0.6541328 0.6362071 0.7267933 0.8821837 0
MARS 0.31826066 0.5490608 0.6043979 0.5810654 0.6537452 0.7732053 0
KNN 0.15458364 0.4554774 0.4955779 0.4860430 0.6102074 0.6714464 0
NN 0.05798509 0.4241956 0.5180183 0.5226842 0.6834319 0.8204222 0
Based on the results, the SVM model performed the best in terms of having the lowest RMSE value and the highest Rsquared value. Next was the MARS model and then the Neural Network model and finally the KNN model.
(b) Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
svm_imp <-varImp(svm_model)plot(svm_imp, top =10, main ="Top 10 Important Variables (SVM)")
set.seed(8585)pls_mod <-train(x = training_x, y = training_y, method ="pls", tuneLength =10, trControl = ctrl)plot(varImp(pls_mod), top =10, main ="Top 10 Important Variables (PLS)")
Attaching package: 'pls'
The following object is masked from 'package:caret':
R2
The following object is masked from 'package:stats':
loadings
The top five most important predictors in the SVM model were the ManufacturingProcess32, ManufacturingProcess13, BiologicalMaterial06, ManufacturingProcess17, and ManufacturingProcess36. The ManufactoringProcess variables dominate the list with only one BiologicalMaterial variable in the top 5. The optimal linear model from Exercise 6.3 was the PLS model and that was dominated by ManufactoringProcess predictors. The two models share 8 of the same predictors in the top 10.
(c) Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
Between the top 10 predictors for the linear and nonlinear regression models, there were two that were unique to the SVM model. The two predictors were the ManufacturingProcess31 and BiologicalMaterial12.
library(ggplot2)ggplot(training_data, aes(x = ManufacturingProcess31, y = Yield)) +geom_point(alpha =0.5) +geom_smooth(method ="loess", color ="blue", se =FALSE) +theme_minimal() +labs(title ="Yield vs ManufacturingProcess31")
`geom_smooth()` using formula = 'y ~ x'
Looking at the plot above, the nonlinear regression model was able to capture the relationship between ManufacturingProcess31 and Yield while a linear regression model would probably just go straight through the plot and capture nothing.
library(ggplot2)ggplot(training_data, aes(x = BiologicalMaterial12, y = Yield)) +geom_point(alpha =0.5) +geom_smooth(method ="loess", color ="blue", se =FALSE) +theme_minimal() +labs(title ="Yield vs BiologicalMaterial12")
`geom_smooth()` using formula = 'y ~ x'
Looking at the plot above, the nonlinear regression model does a good job at capturing the relationship between the Yield and BiologicalMaterial12 because it is able to capture the upswing in the middle and then the downward curve at the end. A linear model would be able to capture some of the relationship but would not do as well of a job compared to the nonlinear model.