Assignment 8
Mary Anna Kivenson
April 11, 2020
library(caret)
library(pls)
library(tidyverse)
library(AppliedPredictiveModeling)
library(corrplot)
library(nnet)Assignment 8 - Non-Linear Regression Models
Question 7.2
Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data:
\[y = 10sin(πx_1x_2) + 20(x_3 − 0.5)^2 + 10x_4 +5x_5 + N(0,σ^2)\]
where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
library(mlbench)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y) 
## or other methods.
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x) Tune several models on these data. For example:
library(caret)
knnModel <- train(x = trainingData$x,
y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.466085 0.5121775 2.816838
## 7 3.349428 0.5452823 2.727410
## 9 3.264276 0.5785990 2.660026
## 11 3.214216 0.6024244 2.603767
## 13 3.196510 0.6176570 2.591935
## 15 3.184173 0.6305506 2.577482
## 17 3.183130 0.6425367 2.567787
## 19 3.198752 0.6483184 2.592683
## 21 3.188993 0.6611428 2.588787
## 23 3.200458 0.6638353 2.604529
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.knnPred <- predict(knnModel, newdata = testData$x)
postResample(pred = knnPred, obs = testData$y)## RMSE Rsquared MAE
## 3.2040595 0.6819919 2.5683461Neural Network Model
nnetFit## Model Averaged Neural Network with 5 Repeats
##
## a 10-5-1 network with 61 weights
## options were - linear output units decay=0.01nnetPred <- predict(nnetFit, newdata = testData$x)
postResample(pred = nnetPred, obs = testData$y)## RMSE Rsquared MAE
## 1.7760172 0.8732967 1.3436096Multivariate Adaptive Regression Splines
library(earth)
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
marsTuned <- train(x = trainingData$x,
y = trainingData$y,
method = "earth",
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))
summary(marsTuned)## Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
## nprune=12)
##
## coefficients
## (Intercept) 22.050690
## h(0.621722-X1) -15.001651
## h(X1-0.621722) 10.878737
## h(0.601063-X2) -18.830135
## h(0.447442-X3) 9.940077
## h(X3-0.606015) 12.999390
## h(0.734892-X4) -9.877554
## h(X4-0.734892) 10.414930
## h(0.850094-X5) -5.604897
## h(X1-0.621722) * h(X2-0.295997) -43.245766
## h(0.649253-X1) * h(0.601063-X2) 26.218297
##
## Selected 11 of 18 terms, and 5 of 10 predictors
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 8 2
## GCV 1.747495 RSS 264.5358 GRSq 0.929051 RSq 0.9457576marsPred <- predict(marsTuned, newdata = testData$x)
postResample(pred = marsPred, obs = testData$y)## RMSE Rsquared MAE
## 1.2803060 0.9335241 1.0168673Support Vector Machines
library(kernlab)
svmRTuned <- train(x = trainingData$x,
y = trainingData$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmRTuned## Support Vector Machines with Radial Basis Function Kernel
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 2.530435 0.7901963 2.004487
## 0.50 2.255696 0.8090387 1.792581
## 1.00 2.093099 0.8300166 1.660175
## 2.00 2.008122 0.8446842 1.571400
## 4.00 1.946207 0.8556897 1.519964
## 8.00 1.905850 0.8611268 1.505200
## 16.00 1.907569 0.8600929 1.509775
## 32.00 1.907569 0.8600929 1.509775
## 64.00 1.907569 0.8600929 1.509775
## 128.00 1.907569 0.8600929 1.509775
## 256.00 1.907569 0.8600929 1.509775
## 512.00 1.907569 0.8600929 1.509775
## 1024.00 1.907569 0.8600929 1.509775
## 2048.00 1.907569 0.8600929 1.509775
##
## Tuning parameter 'sigma' was held constant at a value of 0.06431588
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.06431588 and C = 8.svmPred <- predict(svmRTuned, newdata = testData$x)
postResample(pred = svmPred, obs = testData$y)## RMSE Rsquared MAE
## 2.0596874 0.8281385 1.5633494The Multivariate Adaptive Regression Splines model produces the lowest RMSE, highest Rsquared, and lowest MAE. The MARS model selected 5 out of 10 predictors (X1-X5); these are all the informative predictors.
Question 7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
data(ChemicalManufacturingProcess)
cmp <- ChemicalManufacturingProcess[, -nearZeroVar(ChemicalManufacturingProcess)]
#kNN imputation will be used to fill the missing values in the dataset.
cmp <- preProcess(as.data.frame(cmp), method = "knnImpute", k = 10)$data
# test train split
set.seed(0)
smp_size <- floor(0.8 * nrow(cmp))
train_ind <- sample(seq_len(nrow(cmp)), size = smp_size)
Xtrain <- cmp[train_ind, -1]
Xtest <- cmp[-train_ind, -1]
ytrain <- cmp[train_ind, 1]
ytest <- cmp[-train_ind, 1]- Which nonlinear regression model gives the optimal resampling and test set performance?
K Nearest Neighbor
knnModel <- train(x = Xtrain,
y = ytrain,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 121 samples
## 56 predictor
##
## Pre-processing: centered (56), scaled (56)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 121, 121, 121, 121, 121, 121, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 0.7915034 0.4601246 0.6280008
## 7 0.8044428 0.4440399 0.6386233
## 9 0.8009274 0.4583975 0.6395943
## 11 0.8034349 0.4623691 0.6448591
## 13 0.8049449 0.4666385 0.6474978
## 15 0.8136250 0.4576535 0.6541549
## 17 0.8199790 0.4567221 0.6611276
## 19 0.8240740 0.4602510 0.6647304
## 21 0.8328735 0.4565277 0.6720905
## 23 0.8394308 0.4560071 0.6778361
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 5.knnPred <- predict(knnModel, newdata = Xtest)
postResample(pred = knnPred, obs = ytest)## RMSE Rsquared MAE
## 0.6970735 0.4128862 0.5354828Neural Network Model
nnetFit## Model Averaged Neural Network with 5 Repeats
##
## a 56-5-1 network with 291 weights
## options were - linear output units decay=0.01nnetPred <- predict(nnetFit, newdata = Xtest)
postResample(pred = nnetPred, obs = ytest)## RMSE Rsquared MAE
## 0.6555705 0.5444447 0.5087601Multivariate Adaptive Regression Splines
library(earth)
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
marsTuned <- train(x = Xtrain,
y = ytrain,
method = "earth",
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))
summary(marsTuned)## Call: earth(x=data.frame[121,56], y=c(-1.011,0.446,...), keepxy=TRUE, degree=2,
## nprune=7)
##
## coefficients
## (Intercept) -0.4517631
## h(-0.090604-ManufacturingProcess09) -0.5310012
## h(-0.401815-ManufacturingProcess13) 0.7457923
## h(ManufacturingProcess32- -0.827442) 0.4893506
## h(ManufacturingProcess39-0.165304) -2.5426673
## h(BiologicalMaterial12- -0.219621) * h(ManufacturingProcess09- -0.090604) 0.4452104
## h(0.744561-ManufacturingProcess28) * h(ManufacturingProcess32- -0.827442) 0.2471084
##
## Selected 7 of 42 terms, and 6 of 56 predictors
## Termination condition: RSq changed by less than 0.001 at 42 terms
## Importance: ManufacturingProcess32, ManufacturingProcess09, ...
## Number of terms at each degree of interaction: 1 4 2
## GCV 0.3336436 RSS 30.40017 GRSq 0.6995123 RSq 0.7699391marsPred <- predict(marsTuned, newdata = Xtest)
postResample(pred = marsPred, obs = ytest)## RMSE Rsquared MAE
## 0.6141105 0.5736586 0.5134494Support Vector Machines
library(kernlab)
svmRTuned <- train(x = Xtrain,
y = ytrain,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmRTuned## Support Vector Machines with Radial Basis Function Kernel
##
## 121 samples
## 56 predictor
##
## Pre-processing: centered (56), scaled (56)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 109, 108, 109, 109, 108, 109, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 0.8302541 0.4672792 0.6777092
## 0.50 0.7646354 0.5227493 0.6194463
## 1.00 0.7271215 0.5691309 0.5957993
## 2.00 0.6741144 0.6215252 0.5553700
## 4.00 0.6558508 0.6237964 0.5336353
## 8.00 0.6500105 0.6235019 0.5257665
## 16.00 0.6488643 0.6254285 0.5241727
## 32.00 0.6488643 0.6254285 0.5241727
## 64.00 0.6488643 0.6254285 0.5241727
## 128.00 0.6488643 0.6254285 0.5241727
## 256.00 0.6488643 0.6254285 0.5241727
## 512.00 0.6488643 0.6254285 0.5241727
## 1024.00 0.6488643 0.6254285 0.5241727
## 2048.00 0.6488643 0.6254285 0.5241727
##
## Tuning parameter 'sigma' was held constant at a value of 0.01208728
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.01208728 and C = 16.svmPred <- predict(svmRTuned, newdata = Xtest)
postResample(pred = svmPred, obs = ytest)## RMSE Rsquared MAE
## 0.4799170 0.7235624 0.4032473The SVM model produces the best results when resampling on the test set, which a much higher R-squared and lower RMSE than the other models.
- Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
For the SVM model, which is the optimal nonlinear regression model, the following chart shows the ten most important predictors. Neither manufacturing nor biological variables dominate the list completely, though manufacturing processes make up the top four predictors. This is similar to the predictor importance results of the optimal linear model, which includes all the same variables other than BiologicalMaterial12. The order of importance is also different, but both models have the same manufacturing processes in the top 4.
plsTune <- train(Xtrain,
ytrain,
method = "pls",
tuneLength = 30,
preProc = c("center", "scale"),
trControl = trainControl(method = 'cv', 10))
plot(varImp(svmRTuned), top = 10, main = "SVM")
plot(varImp(plsTune), top = 10, main = "Linear")
- Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
The only predictor that is unique to the SVM model is BiologicalMaterial12 - the relationship between this predictor and yield is linear, positive, and somewhat weak.
plot(Xtrain$BiologicalMaterial12, ytrain)