Data 624 HW 8
Gabe Abreu
11/13/2021
Homework 8
Do problems 7.2 and 7.5 in Kuhn and Johnson.
7.2
Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data:
y = 10sin(πx1x2) + 20(x3 − 0.5)2 + 10x4 + 5x5 + N(0, σ2)where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
library(mlbench)## Warning: package 'mlbench' was built under R version 3.6.3set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will five the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)Tune several models on these data.
KNN
knnModel <- train(x = trainingData$x,
y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.466085 0.5121775 2.816838
## 7 3.349428 0.5452823 2.727410
## 9 3.264276 0.5785990 2.660026
## 11 3.214216 0.6024244 2.603767
## 13 3.196510 0.6176570 2.591935
## 15 3.184173 0.6305506 2.577482
## 17 3.183130 0.6425367 2.567787
## 19 3.198752 0.6483184 2.592683
## 21 3.188993 0.6611428 2.588787
## 23 3.200458 0.6638353 2.604529
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.knnPred <- predict(knnModel, newdata = testData$x)
postResample(pred = knnPred, obs = testData$y)## RMSE Rsquared MAE
## 3.2040595 0.6819919 2.5683461knnResample <- postResample(pred = knnPred, obs = testData$y)Support Vector Machine
svmModel <- train(trainingData$x, trainingData$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmModel## Support Vector Machines with Radial Basis Function Kernel
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 2.505383 0.8031869 1.999381
## 0.50 2.290725 0.8103140 1.829703
## 1.00 2.105086 0.8302040 1.677851
## 2.00 2.014620 0.8418576 1.598814
## 4.00 1.965196 0.8491165 1.567327
## 8.00 1.927649 0.8538945 1.542267
## 16.00 1.924262 0.8545293 1.539275
## 32.00 1.924262 0.8545293 1.539275
## 64.00 1.924262 0.8545293 1.539275
## 128.00 1.924262 0.8545293 1.539275
## 256.00 1.924262 0.8545293 1.539275
## 512.00 1.924262 0.8545293 1.539275
## 1024.00 1.924262 0.8545293 1.539275
## 2048.00 1.924262 0.8545293 1.539275
##
## Tuning parameter 'sigma' was held constant at a value of 0.06802164
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.06802164 and C = 16.svmPred <- predict(svmModel, newdata = testData$x)
postResample(pred = svmPred, obs = testData$y)## RMSE Rsquared MAE
## 2.0864652 0.8236735 1.5854649svResample <- postResample(pred = svmPred, obs = testData$y)Neural Net
nnetGrid <- expand.grid(.decay = c(0, 0.01, .1),
.size = c(1:10),
.bag = FALSE)set.seed(100)
nnetModel <- train(trainingData$x, trainingData$y,
method = "avNNet",
tuneGrid = nnetGrid,
trControl = trainControl(method = "cv"),
preProc = c("center", "scale"),
linout = TRUE,
trace = FALSE,
MaxNWts = 10 * (ncol(trainingData$x) + 1) + 10 + 1,
maxit = 500)## Warning: executing %dopar% sequentially: no parallel backend registerednnetModel## Model Averaged Neural Network
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## decay size RMSE Rsquared MAE
## 0.00 1 2.392711 0.7610354 1.897330
## 0.00 2 2.410532 0.7567109 1.907478
## 0.00 3 2.043957 0.8224281 1.630751
## 0.00 4 2.289347 0.8130639 1.749187
## 0.00 5 2.445600 0.7709399 1.824446
## 0.00 6 2.898295 0.7388800 2.052725
## 0.00 7 3.351563 0.6644147 2.460366
## 0.00 8 6.513566 0.4418645 3.563297
## 0.00 9 4.484215 0.5644107 2.877950
## 0.00 10 3.422545 0.6247430 2.439739
## 0.01 1 2.385381 0.7602926 1.887906
## 0.01 2 2.425125 0.7510903 1.935991
## 0.01 3 2.151209 0.8016018 1.701951
## 0.01 4 2.091925 0.8154383 1.676653
## 0.01 5 2.169742 0.7999255 1.738715
## 0.01 6 2.262032 0.8056619 1.817195
## 0.01 7 2.318301 0.7861811 1.856908
## 0.01 8 2.413847 0.7772629 1.938009
## 0.01 9 2.317190 0.7847500 1.857641
## 0.01 10 2.480407 0.7408505 1.995656
## 0.10 1 2.393965 0.7596431 1.894191
## 0.10 2 2.423612 0.7525959 1.935872
## 0.10 3 2.169914 0.7982380 1.726854
## 0.10 4 2.059080 0.8224160 1.648610
## 0.10 5 1.975656 0.8394000 1.578979
## 0.10 6 2.152198 0.8098015 1.693056
## 0.10 7 2.161512 0.8163011 1.693526
## 0.10 8 2.273716 0.7922525 1.822713
## 0.10 9 2.315333 0.7811273 1.785409
## 0.10 10 2.334803 0.7692182 1.872733
##
## Tuning parameter 'bag' was held constant at a value of FALSE
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were size = 5, decay = 0.1 and bag = FALSE.nnetPred <- predict(nnetModel, newdata = testData$x)
postResample(pred = nnetPred, obs = testData$y)## RMSE Rsquared MAE
## 2.1113956 0.8277556 1.5739011nnetResample <- postResample(pred = nnetPred, obs = testData$y)MARS
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
set.seed(100)
marsModel <- train(trainingData$x, trainingData$y,
method = "earth",
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))## Loading required package: earth## Loading required package: Formula## Loading required package: plotmo## Loading required package: plotrix## Loading required package: TeachingDemos## Warning: package 'TeachingDemos' was built under R version 3.6.3marsModel## Multivariate Adaptive Regression Spline
##
## 200 samples
## 10 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## degree nprune RMSE Rsquared MAE
## 1 2 4.327937 0.2544880 3.600474
## 1 3 3.572450 0.4912720 2.895811
## 1 4 2.596841 0.7183600 2.106341
## 1 5 2.370161 0.7659777 1.918669
## 1 6 2.276141 0.7881481 1.810001
## 1 7 1.766728 0.8751831 1.390215
## 1 8 1.780946 0.8723243 1.401345
## 1 9 1.665091 0.8819775 1.325515
## 1 10 1.663804 0.8821283 1.327657
## 1 11 1.657738 0.8822967 1.331730
## 1 12 1.653784 0.8827903 1.331504
## 1 13 1.648496 0.8823663 1.316407
## 1 14 1.639073 0.8841742 1.312833
## 1 15 1.639073 0.8841742 1.312833
## 1 16 1.639073 0.8841742 1.312833
## 1 17 1.639073 0.8841742 1.312833
## 1 18 1.639073 0.8841742 1.312833
## 1 19 1.639073 0.8841742 1.312833
## 1 20 1.639073 0.8841742 1.312833
## 1 21 1.639073 0.8841742 1.312833
## 1 22 1.639073 0.8841742 1.312833
## 1 23 1.639073 0.8841742 1.312833
## 1 24 1.639073 0.8841742 1.312833
## 1 25 1.639073 0.8841742 1.312833
## 1 26 1.639073 0.8841742 1.312833
## 1 27 1.639073 0.8841742 1.312833
## 1 28 1.639073 0.8841742 1.312833
## 1 29 1.639073 0.8841742 1.312833
## 1 30 1.639073 0.8841742 1.312833
## 1 31 1.639073 0.8841742 1.312833
## 1 32 1.639073 0.8841742 1.312833
## 1 33 1.639073 0.8841742 1.312833
## 1 34 1.639073 0.8841742 1.312833
## 1 35 1.639073 0.8841742 1.312833
## 1 36 1.639073 0.8841742 1.312833
## 1 37 1.639073 0.8841742 1.312833
## 1 38 1.639073 0.8841742 1.312833
## 2 2 4.327937 0.2544880 3.600474
## 2 3 3.572450 0.4912720 2.895811
## 2 4 2.661826 0.7070510 2.173471
## 2 5 2.404015 0.7578971 1.975387
## 2 6 2.243927 0.7914805 1.783072
## 2 7 1.856336 0.8605482 1.435682
## 2 8 1.754607 0.8763186 1.396841
## 2 9 1.603578 0.8938666 1.261361
## 2 10 1.492421 0.9084998 1.168700
## 2 11 1.317350 0.9292504 1.033926
## 2 12 1.304327 0.9320133 1.019108
## 2 13 1.277510 0.9323681 1.002927
## 2 14 1.269626 0.9350024 1.003346
## 2 15 1.266217 0.9359400 1.013893
## 2 16 1.268470 0.9354868 1.011414
## 2 17 1.268470 0.9354868 1.011414
## 2 18 1.268470 0.9354868 1.011414
## 2 19 1.268470 0.9354868 1.011414
## 2 20 1.268470 0.9354868 1.011414
## 2 21 1.268470 0.9354868 1.011414
## 2 22 1.268470 0.9354868 1.011414
## 2 23 1.268470 0.9354868 1.011414
## 2 24 1.268470 0.9354868 1.011414
## 2 25 1.268470 0.9354868 1.011414
## 2 26 1.268470 0.9354868 1.011414
## 2 27 1.268470 0.9354868 1.011414
## 2 28 1.268470 0.9354868 1.011414
## 2 29 1.268470 0.9354868 1.011414
## 2 30 1.268470 0.9354868 1.011414
## 2 31 1.268470 0.9354868 1.011414
## 2 32 1.268470 0.9354868 1.011414
## 2 33 1.268470 0.9354868 1.011414
## 2 34 1.268470 0.9354868 1.011414
## 2 35 1.268470 0.9354868 1.011414
## 2 36 1.268470 0.9354868 1.011414
## 2 37 1.268470 0.9354868 1.011414
## 2 38 1.268470 0.9354868 1.011414
##
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 15 and degree = 2.marsPred <- predict(marsModel, newdata = testData$x)
postResample(pred = marsPred, obs = testData$y)## RMSE Rsquared MAE
## 1.1589948 0.9460418 0.9250230marsResample <- postResample(pred = marsPred, obs = testData$y)display1 <- rbind(
"KNN" = knnResample,
"Neural Net" = nnetResample,
"SVM" = svResample,
"MARS" = marsResample
)
display1 %>% kable() %>% kable_paper()| RMSE | Rsquared | MAE | |
|---|---|---|---|
| KNN | 3.204060 | 0.6819919 | 2.568346 |
| Neural Net | 2.111396 | 0.8277556 | 1.573901 |
| SVM | 2.086465 | 0.8236735 | 1.585465 |
| MARS | 1.158995 | 0.9460418 | 0.925023 |
The MARS model performed very well and is the optimal choice with an Rsquared value of 0.946, Neural Net did perform well with an Rsquared of 0.8277. However, high Rsquared values such as 0.946 does make me think the model might be overfit.
7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models
library(AppliedPredictiveModeling)## Warning: package 'AppliedPredictiveModeling' was built under R version 3.6.3data("ChemicalManufacturingProcess")Processed <- preProcess(ChemicalManufacturingProcess,
method = c( "knnImpute", "center", "scale"))
complete_chem <- predict(Processed, ChemicalManufacturingProcess)
complete_chem$Yield = ChemicalManufacturingProcess$Yieldchemframe <- complete_chem[,-c(nearZeroVar(complete_chem))]
corr_data <- findCorrelation(cor(chemframe), cutoff = 0.90)
chemframe <- chemframe[, -corr_data]predictors <- subset(chemframe, select = -Yield)
yield <- subset(chemframe, select="Yield")set.seed(100)
#Create Data Partition
initsplit2 <- createDataPartition(yield$Yield, p=0.8, list=FALSE)
#Create Training Data to tune the model
training2 <- predictors[initsplit2,]
training_yield <- yield[initsplit2,]
#Create testing data to evaluate the model
test2 <- predictors[-initsplit2,]
test_yield <- yield[-initsplit2,]- Which nonlinear regression model gives the optimal resampling and test set performance?
KNN
knnModel <- train(x = training2,
y = training_yield,
method = "knn",
#preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 144 samples
## 46 predictor
##
## No pre-processing
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 144, 144, 144, 144, 144, 144, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 1.463164 0.3973650 1.162796
## 7 1.472554 0.3852359 1.181034
## 9 1.489149 0.3739195 1.204304
## 11 1.469094 0.3940889 1.192368
## 13 1.463137 0.4029152 1.190871
## 15 1.470070 0.4005951 1.198809
## 17 1.471031 0.4051010 1.200570
## 19 1.473985 0.4071509 1.200841
## 21 1.475209 0.4129944 1.200268
## 23 1.482031 0.4128982 1.201539
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 13.knnPred2 <- predict(knnModel, newdata = test2)
postResample(pred = knnPred2, obs = test_yield)## RMSE Rsquared MAE
## 1.1807297 0.5146414 0.9889663knnResample2 <- postResample(pred = knnPred2, obs = test_yield)Support Vector Machine
svmModel2 <- train(training2, training_yield,
method = "svmRadial",
#preProc = c("center", "scale"),
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmModel2## Support Vector Machines with Radial Basis Function Kernel
##
## 144 samples
## 46 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 130, 129, 129, 129, 131, 129, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 1.459600 0.4929203 1.1814370
## 0.50 1.336994 0.5366176 1.0908949
## 1.00 1.250213 0.5792581 1.0218281
## 2.00 1.201567 0.6024338 0.9665750
## 4.00 1.182357 0.6150766 0.9406424
## 8.00 1.153564 0.6323136 0.9220762
## 16.00 1.136784 0.6455573 0.9116199
## 32.00 1.136784 0.6455573 0.9116199
## 64.00 1.136784 0.6455573 0.9116199
## 128.00 1.136784 0.6455573 0.9116199
## 256.00 1.136784 0.6455573 0.9116199
## 512.00 1.136784 0.6455573 0.9116199
## 1024.00 1.136784 0.6455573 0.9116199
## 2048.00 1.136784 0.6455573 0.9116199
##
## Tuning parameter 'sigma' was held constant at a value of 0.01383934
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.01383934 and C = 16.svmPred2 <- predict(svmModel2, newdata = test2)
postResample(pred = svmPred2, obs = test_yield)## RMSE Rsquared MAE
## 1.0398948 0.6069301 0.8150863svResample2 <- postResample(pred = svmPred2, obs = test_yield)Neural Net
set.seed(100)
nnetModel2 <- train(training2, training_yield,
method = "avNNet",
tuneGrid = nnetGrid,
trControl = trainControl(method = "cv"),
#preProc = c("center", "scale"),
linout = TRUE,
trace = FALSE,
MaxNWts = 5 * (ncol(training2) + 1) + 5 + 1,
maxit = 500)## Warning: model fit failed for Fold01: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold01: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold01: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold01: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold01: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold01: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold01: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold01: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold01: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold01: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold01: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold01: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold01: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold01: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold01: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold02: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold02: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold02: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold02: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold02: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold02: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold02: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold02: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold02: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold02: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold02: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold02: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold02: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold02: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold02: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold03: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold03: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold03: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold03: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold03: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold03: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold03: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold03: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold03: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold03: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold03: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold03: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold03: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold03: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold03: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold04: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold04: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold04: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold04: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold04: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold04: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold04: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold04: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold04: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold04: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold04: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold04: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold04: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold04: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold04: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold05: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold05: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold05: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold05: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold05: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold05: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold05: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold05: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold05: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold05: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold05: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold05: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold05: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold05: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold05: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold06: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold06: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold06: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold06: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold06: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold06: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold06: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold06: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold06: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold06: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold06: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold06: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold06: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold06: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold06: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold07: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold07: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold07: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold07: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold07: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold07: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold07: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold07: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold07: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold07: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold07: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold07: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold07: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold07: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold07: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold08: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold08: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold08: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold08: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold08: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold08: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold08: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold08: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold08: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold08: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold08: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold08: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold08: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold08: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold08: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold09: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold09: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold09: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold09: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold09: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold09: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold09: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold09: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold09: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold09: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold09: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold09: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold09: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold09: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold09: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold10: decay=0.00, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold10: decay=0.01, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold10: decay=0.10, size= 6, bag=FALSE Error in { : task 1 failed - "too many (289) weights"## Warning: model fit failed for Fold10: decay=0.00, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold10: decay=0.01, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold10: decay=0.10, size= 7, bag=FALSE Error in { : task 1 failed - "too many (337) weights"## Warning: model fit failed for Fold10: decay=0.00, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold10: decay=0.01, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold10: decay=0.10, size= 8, bag=FALSE Error in { : task 1 failed - "too many (385) weights"## Warning: model fit failed for Fold10: decay=0.00, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold10: decay=0.01, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold10: decay=0.10, size= 9, bag=FALSE Error in { : task 1 failed - "too many (433) weights"## Warning: model fit failed for Fold10: decay=0.00, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold10: decay=0.01, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning: model fit failed for Fold10: decay=0.10, size=10, bag=FALSE Error in { : task 1 failed - "too many (481) weights"## Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo, :
## There were missing values in resampled performance measures.## Warning in train.default(training2, training_yield, method = "avNNet", tuneGrid
## = nnetGrid, : missing values found in aggregated resultsnnetModel2## Model Averaged Neural Network
##
## 144 samples
## 46 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 129, 130, 130, 130, 130, 130, ...
## Resampling results across tuning parameters:
##
## decay size RMSE Rsquared MAE
## 0.00 1 1.515212 0.4435963 1.237487
## 0.00 2 1.554009 0.3945294 1.228312
## 0.00 3 2.399561 0.2378128 1.812170
## 0.00 4 1.837202 0.3299899 1.481751
## 0.00 5 1.604397 0.5013362 1.296960
## 0.00 6 NaN NaN NaN
## 0.00 7 NaN NaN NaN
## 0.00 8 NaN NaN NaN
## 0.00 9 NaN NaN NaN
## 0.00 10 NaN NaN NaN
## 0.01 1 1.397503 0.4777893 1.128825
## 0.01 2 1.429382 0.5237032 1.130529
## 0.01 3 2.077119 0.3736576 1.454686
## 0.01 4 1.591085 0.4592756 1.313110
## 0.01 5 1.611632 0.4436736 1.232577
## 0.01 6 NaN NaN NaN
## 0.01 7 NaN NaN NaN
## 0.01 8 NaN NaN NaN
## 0.01 9 NaN NaN NaN
## 0.01 10 NaN NaN NaN
## 0.10 1 1.442597 0.4767379 1.093074
## 0.10 2 1.586502 0.5063729 1.132924
## 0.10 3 1.738851 0.3980161 1.292073
## 0.10 4 1.585410 0.4477952 1.229248
## 0.10 5 1.678267 0.4410342 1.272076
## 0.10 6 NaN NaN NaN
## 0.10 7 NaN NaN NaN
## 0.10 8 NaN NaN NaN
## 0.10 9 NaN NaN NaN
## 0.10 10 NaN NaN NaN
##
## Tuning parameter 'bag' was held constant at a value of FALSE
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were size = 1, decay = 0.01 and bag = FALSE.nnetPred2 <- predict(nnetModel2, newdata = test2)
postResample(pred = nnetPred2, obs = test_yield)## RMSE Rsquared MAE
## 1.1834364 0.5221843 0.9693455nnetResample2 <- postResample(pred = nnetPred2, obs = test_yield)MARS
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
set.seed(100)
marsModel2 <- train(training2, training_yield,
method = "earth",
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))
marsModel2## Multivariate Adaptive Regression Spline
##
## 144 samples
## 46 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 129, 130, 130, 130, 130, 130, ...
## Resampling results across tuning parameters:
##
## degree nprune RMSE Rsquared MAE
## 1 2 1.410080 0.4643855 1.1074276
## 1 3 1.239030 0.5566464 0.9945686
## 1 4 1.207841 0.5939211 0.9844014
## 1 5 1.234876 0.5570850 1.0013926
## 1 6 1.239626 0.5558288 0.9921505
## 1 7 1.258974 0.5468413 1.0111741
## 1 8 1.298985 0.5107353 1.0559709
## 1 9 1.274237 0.5229478 1.0169997
## 1 10 1.302989 0.5251049 1.0426334
## 1 11 1.311673 0.5208022 1.0487397
## 1 12 1.258257 0.5577426 1.0136784
## 1 13 1.273965 0.5530480 1.0179430
## 1 14 1.277223 0.5554494 1.0200166
## 1 15 1.289390 0.5484618 1.0289773
## 1 16 1.299993 0.5462074 1.0406692
## 1 17 1.299993 0.5462074 1.0406692
## 1 18 1.299993 0.5462074 1.0406692
## 1 19 1.299993 0.5462074 1.0406692
## 1 20 1.299993 0.5462074 1.0406692
## 1 21 1.299993 0.5462074 1.0406692
## 1 22 1.299993 0.5462074 1.0406692
## 1 23 1.299993 0.5462074 1.0406692
## 1 24 1.299993 0.5462074 1.0406692
## 1 25 1.299993 0.5462074 1.0406692
## 1 26 1.299993 0.5462074 1.0406692
## 1 27 1.299993 0.5462074 1.0406692
## 1 28 1.299993 0.5462074 1.0406692
## 1 29 1.299993 0.5462074 1.0406692
## 1 30 1.299993 0.5462074 1.0406692
## 1 31 1.299993 0.5462074 1.0406692
## 1 32 1.299993 0.5462074 1.0406692
## 1 33 1.299993 0.5462074 1.0406692
## 1 34 1.299993 0.5462074 1.0406692
## 1 35 1.299993 0.5462074 1.0406692
## 1 36 1.299993 0.5462074 1.0406692
## 1 37 1.299993 0.5462074 1.0406692
## 1 38 1.299993 0.5462074 1.0406692
## 2 2 1.410080 0.4643855 1.1074276
## 2 3 1.212745 0.5798908 0.9758157
## 2 4 1.249900 0.5621647 1.0073857
## 2 5 1.224339 0.5777778 0.9787115
## 2 6 1.228165 0.5897974 1.0046883
## 2 7 1.227243 0.5965559 0.9942716
## 2 8 1.244180 0.5920297 0.9977763
## 2 9 1.232961 0.5961477 0.9906836
## 2 10 1.227433 0.6246864 0.9812973
## 2 11 1.249074 0.6136522 1.0024685
## 2 12 1.410488 0.5295066 1.1295953
## 2 13 1.426352 0.5204188 1.1360183
## 2 14 1.419745 0.5274791 1.1371241
## 2 15 1.469690 0.5157714 1.1991982
## 2 16 1.484487 0.5086667 1.2059565
## 2 17 1.469187 0.5187982 1.1886410
## 2 18 1.470405 0.5244441 1.1693801
## 2 19 1.486552 0.5146094 1.1790512
## 2 20 1.491106 0.5125274 1.1797963
## 2 21 1.496110 0.5114684 1.1755267
## 2 22 2.361196 0.4575574 1.4151527
## 2 23 2.400882 0.4639091 1.4171723
## 2 24 2.400390 0.4651612 1.4121857
## 2 25 2.400390 0.4651612 1.4121857
## 2 26 2.400390 0.4651612 1.4121857
## 2 27 2.400390 0.4651612 1.4121857
## 2 28 2.400390 0.4651612 1.4121857
## 2 29 2.400390 0.4651612 1.4121857
## 2 30 2.400390 0.4651612 1.4121857
## 2 31 2.400390 0.4651612 1.4121857
## 2 32 2.400390 0.4651612 1.4121857
## 2 33 2.400390 0.4651612 1.4121857
## 2 34 2.400390 0.4651612 1.4121857
## 2 35 2.400390 0.4651612 1.4121857
## 2 36 2.400390 0.4651612 1.4121857
## 2 37 2.400390 0.4651612 1.4121857
## 2 38 2.400390 0.4651612 1.4121857
##
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 4 and degree = 1.marsPred2 <- predict(marsModel2, newdata = test2)
postResample(pred = marsPred2, obs = test_yield)## RMSE Rsquared MAE
## 1.1657015 0.5172382 0.9320261marsResample2 <- postResample(pred = marsPred2, obs = test_yield)display <- rbind(
"KNN" = knnResample2,
"Neural Net" = nnetResample2,
"SVM" = svResample2,
"MARS" = marsResample2
)
display %>% kable() %>% kable_paper()| RMSE | Rsquared | MAE | |
|---|---|---|---|
| KNN | 1.180730 | 0.5146414 | 0.9889663 |
| Neural Net | 1.183436 | 0.5221843 | 0.9693455 |
| SVM | 1.039895 | 0.6069301 | 0.8150863 |
| MARS | 1.165701 | 0.5172382 | 0.9320261 |
Based on the RMSE and Rsqured, the preferred model is SVM (Support Vector Machine). The Rsquared is realistic and pretty good at 0.617. The other three models were pretty close with their respective Rsquared values and RMSE values.
- Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
plot(varImp(svmModel2), top = 10)
There are minor differences between the optimal linear model and the optimal nonlinear model.
The most important variables are the Manufacturing predictors for both model types, the most important being Manufacturing Process 13 and Manufacturing Process 32. The only biological predictor in the linear model is BiologicalMaterial06, while the nonlinear model also includes BiologicalMaterial03.
Both models place large emphasis on Manufacturing predictors.
- Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
The plots below show that the top variables have a clear relationship, either negative or postive. One of the least important variables has no clear relationship with the response variable.
ggplot(ChemicalManufacturingProcess, aes(BiologicalMaterial06, Yield)) +
geom_point()
ggplot(ChemicalManufacturingProcess, aes(ManufacturingProcess13, Yield)) +
geom_point()
ggplot(ChemicalManufacturingProcess, aes(ManufacturingProcess32, Yield)) +
geom_point()
ggplot(ChemicalManufacturingProcess, aes(ManufacturingProcess11, Yield)) +
geom_point()## Warning: Removed 10 rows containing missing values (geom_point).