Data 624 - Homework 8
Abdelmalek Hajjam
4/24/2021
Exercise 7.2
Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data:
\(y = 10 sin(\pi x_1 x_2) + 20(x_3 - 0.5)^2 + 10x_4 + 5x_5 + N(0, \sigma^2)\)
where the \(x\) values are random variables uniformly distributed between \([0, 1]\) (there are also \(5\) other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
library(caret)## Loading required package: lattice## Loading required package: ggplot2library(mlbench)
help("mlbench.friedman1")## starting httpd help server ...## doneset.seed(200)
training.data <- mlbench.friedman1(200, sd=1)
training.data$x <- data.frame(training.data$x)
featurePlot(training.data$x, training.data$y)
Estimating the true error rate with good precision:
test.data <- mlbench.friedman1(5000, sd = 1)
test.data$x <- data.frame(test.data$x)KNN Model
set.seed(200)
knnModel <- train(x = training.data$x,
y = training.data$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.654912 0.4779838 2.958475
## 7 3.529432 0.5118581 2.861742
## 9 3.446330 0.5425096 2.780756
## 11 3.378049 0.5723793 2.719410
## 13 3.332339 0.5953773 2.692863
## 15 3.309235 0.6111389 2.663046
## 17 3.317408 0.6201421 2.678898
## 19 3.311667 0.6333800 2.682098
## 21 3.316340 0.6407537 2.688887
## 23 3.326040 0.6491480 2.705915
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 15.plot(knnModel)
knn.Predict <- predict(knnModel, newdata = test.data$x)
postResample(pred = knn.Predict, obs = test.data$y)## RMSE Rsquared MAE
## 3.1750657 0.6785946 2.5443169The result of KNN MRSE is 3.1947
MARS
library(earth)## Loading required package: Formula## Loading required package: plotmo## Loading required package: plotrix## Loading required package: TeachingDemosset.seed(200)
marsGrid <- expand.grid(degree =1:2, nprune=seq(2,14,by=2))
marsModel <- train(x = training.data$x, y = training.data$y, method='earth', tuneGrid = marsGrid, trControl = trainControl(method = "cv"))
marsModel## Multivariate Adaptive Regression Spline
##
## 200 samples
## 10 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## degree nprune RMSE Rsquared MAE
## 1 2 4.188280 0.3042527 3.460689
## 1 4 2.653143 0.7167280 2.128222
## 1 6 2.295006 0.7754603 1.853199
## 1 8 1.647182 0.8774867 1.299564
## 1 10 1.635035 0.8798236 1.309436
## 1 12 1.571561 0.8898750 1.253077
## 1 14 1.571673 0.8909652 1.245508
## 2 2 4.188280 0.3042527 3.460689
## 2 4 2.615256 0.7216809 2.128763
## 2 6 2.275048 0.7762472 1.807779
## 2 8 1.641647 0.8839822 1.288520
## 2 10 1.473254 0.9101555 1.158761
## 2 12 1.285380 0.9283193 1.033426
## 2 14 1.261797 0.9327541 1.009821
##
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 14 and degree = 2.plot(marsModel)
Feature Importance
varImp(marsModel)## earth variable importance
##
## Overall
## X1 100.00
## X4 75.24
## X2 48.74
## X5 15.53
## X3 0.00mars.Predict <- predict (marsModel, test.data$x)
postResample(pred = mars.Predict, obs = test.data$y)## RMSE Rsquared MAE
## 1.1722635 0.9448890 0.9324923MARS_RMSE = 1.1487 is definitely better than KNN_RMSE = 3.1947.
Let’s look at other models:
SVM
set.seed(200)
svmTuned <- train(x = training.data$x,
y = training.data$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmTuned## Support Vector Machines with Radial Basis Function Kernel
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 2.525164 0.7810576 2.010680
## 0.50 2.270567 0.7944850 1.794902
## 1.00 2.099356 0.8155574 1.659376
## 2.00 2.005858 0.8302852 1.578799
## 4.00 1.934650 0.8435677 1.528373
## 8.00 1.915665 0.8475605 1.528648
## 16.00 1.923914 0.8463074 1.535991
## 32.00 1.923914 0.8463074 1.535991
## 64.00 1.923914 0.8463074 1.535991
## 128.00 1.923914 0.8463074 1.535991
## 256.00 1.923914 0.8463074 1.535991
## 512.00 1.923914 0.8463074 1.535991
## 1024.00 1.923914 0.8463074 1.535991
## 2048.00 1.923914 0.8463074 1.535991
##
## Tuning parameter 'sigma' was held constant at a value of 0.06299324
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.06299324 and C = 8.svmTuned$finalModel## Support Vector Machine object of class "ksvm"
##
## SV type: eps-svr (regression)
## parameter : epsilon = 0.1 cost C = 8
##
## Gaussian Radial Basis kernel function.
## Hyperparameter : sigma = 0.0629932410345396
##
## Number of Support Vectors : 152
##
## Objective Function Value : -72.63
## Training error : 0.009177svm.Predict <- predict(svmTuned, newdata = test.data$x)
postResample(pred = svm.Predict, obs = test.data$y)## RMSE Rsquared MAE
## 2.0541197 0.8290353 1.5586411SVM_RMSE = 2.0601, not performing well compared to MARS (=1.1487).
Neural Networks
library(nnet)
set.seed(200)
nnetFit <- nnet(training.data$x,
training.data$y,
size = 5,
decay = 0.01,
linout = TRUE,
trace = FALSE,
maxit = 500,
MaxNWts = 5 * (ncol(training.data$x) + 1) + 5 + 1)
nnetFit## a 10-5-1 network with 61 weights
## options were - linear output units decay=0.01nnet.Predict <- predict(nnetFit, newdata = test.data$x)
postResample(pred = nnet.Predict, obs = test.data$y)## RMSE Rsquared MAE
## 2.8134222 0.7083039 2.1815463varImp(nnetFit)## Overall
## X1 10.152501
## X2 7.395536
## X3 23.690690
## X4 14.521013
## X5 11.405236
## X6 4.833571
## X7 10.246607
## X8 6.754084
## X9 6.013435
## X10 4.987327Neural Networks has an RMSE = 2.8435. It’s also not performing well against MARS (=1.1487). Therefore MARS ids the the best model with a lower MRSE.
Exercise 7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
A. Which nonlinear regression model gives the optimal resampling and test set performance?
library(AppliedPredictiveModeling)
data("ChemicalManufacturingProcess")library(DMwR)## Loading required package: grid## Registered S3 method overwritten by 'quantmod':
## method from
## as.zoo.data.frame zooknn.df <- knnImputation(ChemicalManufacturingProcess[, 1:57], k = 3, meth = "weighAvg")
anyNA(knn.df)## [1] FALSELet’s take out the missing values:
near_zero <- nearZeroVar(knn.df)
knn.df <- knn.df[,-near_zero]
knn.df[,2:(ncol(knn.df))] <- scale(knn.df[,2:(ncol(knn.df))])set.seed(200)
inTraining <- createDataPartition(knn.df$Yield, p = 0.80, list=FALSE)
training <- knn.df[ inTraining,]
testing <- knn.df[-inTraining,]
X_train <- training[,2:(length(training))]
Y_train <- training$Yield
X_test <- testing[,2:(length(testing))]
Y_test <- testing$YieldKNN
set.seed(200)
knnModel <- train(x = X_train,
y = Y_train,
method = "knn",
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 144 samples
## 55 predictor
##
## No pre-processing
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 144, 144, 144, 144, 144, 144, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 1.358724 0.4629696 1.072222
## 7 1.371312 0.4458838 1.083336
## 9 1.376625 0.4401890 1.094604
## 11 1.383306 0.4312044 1.096783
## 13 1.390086 0.4264260 1.103710
## 15 1.390433 0.4275496 1.102569
## 17 1.395259 0.4266876 1.109112
## 19 1.400306 0.4244713 1.116016
## 21 1.403813 0.4240472 1.122619
## 23 1.414156 0.4155829 1.133073
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 5.plot(knnModel)
KNN RMSE value is 1.358724. Let’s see what other models have to say.
MARS
set.seed(200)
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
marsModel <- train(x = X_train,
y = Y_train,
method = "earth",
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))
marsModel## Multivariate Adaptive Regression Spline
##
## 144 samples
## 55 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 130, 129, 129, 130, 131, 131, ...
## Resampling results across tuning parameters:
##
## degree nprune RMSE Rsquared MAE
## 1 2 1.358846 0.4736659 1.0773044
## 1 3 1.296657 0.5106767 1.0478494
## 1 4 1.273441 0.5201153 1.0263780
## 1 5 1.288455 0.5178118 1.0606103
## 1 6 1.308078 0.5076130 1.0565894
## 1 7 1.255087 0.5425926 1.0041199
## 1 8 1.271841 0.5331827 1.0000354
## 1 9 1.279745 0.5250474 1.0154927
## 1 10 1.292175 0.5223808 1.0323971
## 1 11 1.283229 0.5325098 1.0418353
## 1 12 1.269256 0.5402115 1.0194497
## 1 13 1.272423 0.5369788 1.0090722
## 1 14 1.269055 0.5410239 0.9984376
## 1 15 1.281957 0.5327002 1.0105145
## 1 16 1.276377 0.5367464 1.0037746
## 1 17 1.276377 0.5367464 1.0037746
## 1 18 1.276377 0.5367464 1.0037746
## 1 19 1.276377 0.5367464 1.0037746
## 1 20 1.276377 0.5367464 1.0037746
## 1 21 1.276377 0.5367464 1.0037746
## 1 22 1.276377 0.5367464 1.0037746
## 1 23 1.276377 0.5367464 1.0037746
## 1 24 1.276377 0.5367464 1.0037746
## 1 25 1.276377 0.5367464 1.0037746
## 1 26 1.276377 0.5367464 1.0037746
## 1 27 1.276377 0.5367464 1.0037746
## 1 28 1.276377 0.5367464 1.0037746
## 1 29 1.276377 0.5367464 1.0037746
## 1 30 1.276377 0.5367464 1.0037746
## 1 31 1.276377 0.5367464 1.0037746
## 1 32 1.276377 0.5367464 1.0037746
## 1 33 1.276377 0.5367464 1.0037746
## 1 34 1.276377 0.5367464 1.0037746
## 1 35 1.276377 0.5367464 1.0037746
## 1 36 1.276377 0.5367464 1.0037746
## 1 37 1.276377 0.5367464 1.0037746
## 1 38 1.276377 0.5367464 1.0037746
## 2 2 1.358846 0.4736659 1.0773044
## 2 3 1.229696 0.5642792 0.9983440
## 2 4 1.207160 0.5806614 0.9594554
## 2 5 1.235032 0.5545250 0.9899618
## 2 6 1.256267 0.5494292 1.0110015
## 2 7 1.235968 0.5612134 0.9864113
## 2 8 1.217343 0.5692671 0.9789297
## 2 9 1.201919 0.5765427 0.9640088
## 2 10 1.282515 0.5236294 1.0327008
## 2 11 1.227825 0.5647696 0.9841350
## 2 12 1.258767 0.5411971 1.0000580
## 2 13 1.343413 0.4965445 1.0695792
## 2 14 1.362048 0.4986443 1.0872110
## 2 15 1.344378 0.5036945 1.0506216
## 2 16 1.398320 0.4888881 1.0848642
## 2 17 1.473546 0.4854781 1.1120501
## 2 18 1.478332 0.4919094 1.1127982
## 2 19 1.467072 0.5005203 1.1060668
## 2 20 1.470965 0.4987451 1.1024792
## 2 21 1.462591 0.5027102 1.1016735
## 2 22 1.428628 0.5120665 1.0834251
## 2 23 1.443942 0.5094336 1.0865845
## 2 24 1.443942 0.5094336 1.0865845
## 2 25 1.443942 0.5094336 1.0865845
## 2 26 1.443942 0.5094336 1.0865845
## 2 27 1.484623 0.5002102 1.1223371
## 2 28 1.484623 0.5002102 1.1223371
## 2 29 1.458591 0.5056408 1.1186604
## 2 30 1.451215 0.5086060 1.1113881
## 2 31 1.451215 0.5086060 1.1113881
## 2 32 1.451215 0.5086060 1.1113881
## 2 33 1.451215 0.5086060 1.1113881
## 2 34 1.451215 0.5086060 1.1113881
## 2 35 1.451215 0.5086060 1.1113881
## 2 36 1.451215 0.5086060 1.1113881
## 2 37 1.451215 0.5086060 1.1113881
## 2 38 1.451215 0.5086060 1.1113881
##
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 9 and degree = 2.plot(marsModel)
marsPredict <- predict(marsModel, newdata = X_test)
postResample(pred = marsPredict, obs = Y_test)## RMSE Rsquared MAE
## 5.36423843 0.08214788 1.91091003MARS_RMSE = 5.36423843
Neural Networks
set.seed(200)
nnetFit <- nnet(X_train,
Y_train,
size = 5,
decay = 0.01,
linout = TRUE,
trace = FALSE,
maxit = 500,
MaxNWts = 5 * (ncol(X_train) + 1) + 5 + 1)
nnetFit## a 55-5-1 network with 286 weights
## options were - linear output units decay=0.01nnetFit.Predict <- predict(nnetFit, newdata = X_test)
postResample(pred = nnetFit.Predict, obs = Y_test)## RMSE Rsquared MAE
## 3.2274418 0.1310242 2.1838495NN_RMSE = 3.2274418
SVM
set.seed(200)
svmModel <- train(x = X_train,
y = Y_train,
method = "svmRadial",
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmModel## Support Vector Machines with Radial Basis Function Kernel
##
## 144 samples
## 55 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 130, 129, 129, 130, 131, 131, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 1.364309 0.4898533 1.1098152
## 0.50 1.264003 0.5399353 1.0212339
## 1.00 1.167489 0.6082610 0.9280668
## 2.00 1.128452 0.6416772 0.8950842
## 4.00 1.128531 0.6427525 0.8999622
## 8.00 1.130919 0.6396080 0.8970457
## 16.00 1.125978 0.6427766 0.8930774
## 32.00 1.125978 0.6427766 0.8930774
## 64.00 1.125978 0.6427766 0.8930774
## 128.00 1.125978 0.6427766 0.8930774
## 256.00 1.125978 0.6427766 0.8930774
## 512.00 1.125978 0.6427766 0.8930774
## 1024.00 1.125978 0.6427766 0.8930774
## 2048.00 1.125978 0.6427766 0.8930774
##
## Tuning parameter 'sigma' was held constant at a value of 0.01389398
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.01389398 and C = 16.svm.Predict <- predict(svmModel, newdata = X_test)
postResample(pred = svm.Predict, obs = Y_test)## RMSE Rsquared MAE
## 1.3404715 0.6124804 1.0007174SVM_RMSE = 1.3404715
Looks like SVM is the winner as it has the lowest RMSE of all of those NN and MARS models.
B. Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
Feature Importance
varImp(marsModel)## earth variable importance
##
## Overall
## ManufacturingProcess32 100.00
## ManufacturingProcess09 52.54
## ManufacturingProcess29 27.09
## ManufacturingProcess39 22.39
## ManufacturingProcess28 22.39
## ManufacturingProcess17 16.62
## ManufacturingProcess30 0.00
## BiologicalMaterial09 0.00ManufacturingProcess32 is nthe highest important variable. Looks like the ManufactingProcess seem to be leading the feature importance.
varImp(svmModel)## loess r-squared variable importance
##
## only 20 most important variables shown (out of 55)
##
## Overall
## ManufacturingProcess32 100.00
## BiologicalMaterial06 87.81
## ManufacturingProcess13 78.23
## BiologicalMaterial03 76.45
## BiologicalMaterial12 69.16
## ManufacturingProcess17 68.44
## ManufacturingProcess31 67.67
## ManufacturingProcess36 67.63
## ManufacturingProcess09 64.12
## ManufacturingProcess06 60.89
## BiologicalMaterial02 53.50
## ManufacturingProcess29 52.89
## BiologicalMaterial11 48.66
## ManufacturingProcess33 47.00
## ManufacturingProcess11 46.92
## ManufacturingProcess30 44.62
## BiologicalMaterial09 38.34
## BiologicalMaterial04 37.89
## BiologicalMaterial08 36.89
## ManufacturingProcess12 36.65Looks like for SVM, both Manufacturing process and Biological Material are on the top feature importance list. ManufacturingProcess32 is also the highest in this model.
C. Explore the relationship between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
Let’s look at SVM:
svmModel$finalModel## Support Vector Machine object of class "ksvm"
##
## SV type: eps-svr (regression)
## parameter : epsilon = 0.1 cost C = 16
##
## Gaussian Radial Basis kernel function.
## Hyperparameter : sigma = 0.0138939782317795
##
## Number of Support Vectors : 125
##
## Objective Function Value : -85.2452
## Training error : 0.008962xyplot(Y_test ~ predict(svmModel, X_test),
type = c("p", "r"),
xlab = "Predicted", ylab = "Observed")
predicted <- predict(svmModel, X_test)
actual <-Y_test
xyplot((predicted-actual) ~ predicted,
type = c("p", "g"),
xlab = "Predicted radial svm", ylab = "Residuals")
No significant pattern has been indicated by the residual plots.But the actual vs predict graph shows there is a close correlation.
dotPlot(varImp(svmModel), top=20)
top 20 features rank for SVM Model is shown in the above plot.ManufacturingProcess32 is always at the top.
Plotting top 6 features and revealing correlation:
predictors.val <- c("ManufacturingProcess32", "BiologicalMaterial06", "ManufacturingProcess36", "BiologicalMaterial03", "ManufacturingProcess13", "BiologicalMaterial02")
featurePlot(X_train[,predictors.val], Y_train)
cor(X_train[, predictors.val], Y_train)## [,1]
## ManufacturingProcess32 0.6217081
## BiologicalMaterial06 0.4480502
## ManufacturingProcess36 -0.5112720
## BiologicalMaterial03 0.4235631
## ManufacturingProcess13 -0.4901456
## BiologicalMaterial02 0.4511409We can clearly see that the predictors and the yield indeed have some correlation, with ManufacturingProcess32 being the most positively correlated. Some of the Manufacturing Process have negative impact (negative correlation) on Yield, while most of the Biological Maeterial maintain a positive and balanced correlation.