Assignment

Do problems 7.2 and 7.5 in Kuhn and Johnson. There are only two but they have many parts. Please submit both a link to your Rpubs and the .rmd file.

7.2

Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data:

where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:

library(mlbench)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)

## or other methods.
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)

Which models appear to give the best performance? Does MARS select the informative predictors (those named X1–X5)?

KNN Model

Tune several models on these data. For example:

library(caret)
knnModel <- train(x = trainingData$x,
 y = trainingData$y,
 method = "knn",
 preProc = c("center", "scale"),
 tuneLength = 10)

knnModel
## k-Nearest Neighbors 
## 
## 200 samples
##  10 predictor
## 
## Pre-processing: centered (10), scaled (10) 
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ... 
## Resampling results across tuning parameters:
## 
##   k   RMSE      Rsquared   MAE     
##    5  3.466085  0.5121775  2.816838
##    7  3.349428  0.5452823  2.727410
##    9  3.264276  0.5785990  2.660026
##   11  3.214216  0.6024244  2.603767
##   13  3.196510  0.6176570  2.591935
##   15  3.184173  0.6305506  2.577482
##   17  3.183130  0.6425367  2.567787
##   19  3.198752  0.6483184  2.592683
##   21  3.188993  0.6611428  2.588787
##   23  3.200458  0.6638353  2.604529
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.
knnPred <- predict(knnModel, newdata = testData$x)
## The function 'postResample' can be used to get the test set
## perforamnce values
postResample(pred = knnPred, obs = testData$y)
##      RMSE  Rsquared       MAE 
## 3.2040595 0.6819919 2.5683461

MARS

mars_grid <- expand.grid(.degree = 1:2, .nprune = 2:15)
mars_model <- train(
  x = trainingData$x,
  y = trainingData$y,
  method = "earth",
  tuneGrid = mars_grid,
  preProcess = c("center","scale"),
  tuneLength = 10
        )
## Loading required package: earth
## Loading required package: Formula
## Loading required package: plotmo
## Loading required package: plotrix
## Loading required package: TeachingDemos
## 
## Attaching package: 'plotmo'
## The following object is masked from 'package:urca':
## 
##     plotres
mars_model
## Multivariate Adaptive Regression Spline 
## 
## 200 samples
##  10 predictor
## 
## Pre-processing: centered (10), scaled (10) 
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ... 
## Resampling results across tuning parameters:
## 
##   degree  nprune  RMSE      Rsquared   MAE     
##   1        2      4.383438  0.2405683  3.597961
##   1        3      3.645469  0.4745962  2.930453
##   1        4      2.727602  0.7035031  2.184240
##   1        5      2.449243  0.7611230  1.939231
##   1        6      2.331605  0.7835496  1.833420
##   1        7      1.976830  0.8421599  1.562591
##   1        8      1.870959  0.8585503  1.464551
##   1        9      1.804342  0.8683110  1.410395
##   1       10      1.787676  0.8711960  1.386944
##   1       11      1.790700  0.8707740  1.393076
##   1       12      1.821005  0.8670619  1.419893
##   1       13      1.858688  0.8617344  1.445459
##   1       14      1.862343  0.8623072  1.446050
##   1       15      1.871033  0.8607099  1.457618
##   2        2      4.383438  0.2405683  3.597961
##   2        3      3.644919  0.4742570  2.929647
##   2        4      2.730222  0.7028372  2.183075
##   2        5      2.481291  0.7545789  1.965749
##   2        6      2.338369  0.7827873  1.825542
##   2        7      2.030065  0.8328250  1.602024
##   2        8      1.890997  0.8551326  1.477422
##   2        9      1.742626  0.8757904  1.371910
##   2       10      1.608221  0.8943432  1.255416
##   2       11      1.474325  0.9111463  1.157848
##   2       12      1.437483  0.9157967  1.120977
##   2       13      1.439395  0.9164721  1.128309
##   2       14      1.428565  0.9184503  1.118634
##   2       15      1.434093  0.9182413  1.121622
## 
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 14 and degree = 2.
marspredictions <- predict(mars_model, newdata = testData$x)
postResample(pred = marspredictions, obs = testData$y)
##      RMSE  Rsquared       MAE 
## 1.2779993 0.9338365 1.0147070
varImp(mars_model)
## earth variable importance
## 
##    Overall
## X1  100.00
## X4   75.40
## X2   49.00
## X5   15.72
## X3    0.00

Neural Network

nngrid <- expand.grid(size = c(1:10),
                        decay = c(0, 0.01, 0.1),
                        bag = FALSE)

ctrl <- trainControl(method = "cv")

nnmodel <- train(trainingData$x, trainingData$y,
                  method = "avNNet",
                  tuneGrid = nngrid,
                  trControl = ctrl,
                  preProcess = c("center", "scale"),
                  linout = TRUE,
                  trace = FALSE,
                  MaxNWts = 10 * (ncol(trainingData$x) + 1) + 10 + 1,
                  maxit = 500
                  )

nnmodel
## Model Averaged Neural Network 
## 
## 200 samples
##  10 predictor
## 
## Pre-processing: centered (10), scaled (10) 
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
## Resampling results across tuning parameters:
## 
##   size  decay  RMSE      Rsquared   MAE     
##    1    0.00   2.441157  0.7691574  1.892681
##    1    0.01   2.426398  0.7699515  1.888137
##    1    0.10   2.438901  0.7664037  1.895615
##    2    0.00   2.484581  0.7568920  1.985532
##    2    0.01   2.508404  0.7515178  1.935093
##    2    0.10   2.507056  0.7489516  1.923413
##    3    0.00   1.983590  0.8426798  1.560308
##    3    0.01   2.137990  0.8159179  1.671067
##    3    0.10   2.111035  0.8180658  1.658184
##    4    0.00   2.128153  0.8179021  1.674522
##    4    0.01   2.056207  0.8288975  1.626287
##    4    0.10   2.100844  0.8215190  1.655587
##    5    0.00   2.604555  0.7303838  1.882041
##    5    0.01   2.150383  0.8161630  1.704747
##    5    0.10   2.081498  0.8234780  1.641661
##    6    0.00   2.800390  0.7047613  2.073259
##    6    0.01   2.292557  0.7907212  1.784558
##    6    0.10   2.125132  0.8174755  1.691236
##    7    0.00   3.079954  0.6644465  2.242235
##    7    0.01   2.406552  0.7787017  1.910135
##    7    0.10   2.256381  0.7949568  1.807820
##    8    0.00   6.171313  0.4237324  3.379596
##    8    0.01   2.450917  0.7584732  1.919138
##    8    0.10   2.344427  0.7822189  1.859629
##    9    0.00   5.161083  0.5138704  3.113548
##    9    0.01   2.276530  0.7919343  1.808944
##    9    0.10   2.254537  0.7952287  1.726870
##   10    0.00   3.068783  0.6663547  2.300797
##   10    0.01   2.504300  0.7483431  1.959052
##   10    0.10   2.392290  0.7769650  1.893188
## 
## Tuning parameter 'bag' was held constant at a value of FALSE
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were size = 3, decay = 0 and bag
##  = FALSE.
nnpredictions <- predict(nnmodel, newdata = testData$x)
postResample(pred = nnpredictions, obs = testData$y)
##      RMSE  Rsquared       MAE 
## 1.9266030 0.8530679 1.4717585
varImp(nnmodel)
## loess r-squared variable importance
## 
##      Overall
## X4  100.0000
## X1   95.5047
## X2   89.6186
## X5   45.2170
## X3   29.9330
## X9    6.3299
## X10   5.5182
## X8    3.2527
## X6    0.8884
## X7    0.0000

SVM

svmmodel <- train(
  x = trainingData$x,
  y = trainingData$y,
  method = "svmRadial",
  preProcess = c("center","scale"),
  tuneLength = 10,
  trControl = trainControl(method = "cv")
        )
svmmodel
## Support Vector Machines with Radial Basis Function Kernel 
## 
## 200 samples
##  10 predictor
## 
## Pre-processing: centered (10), scaled (10) 
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ... 
## Resampling results across tuning parameters:
## 
##   C       RMSE      Rsquared   MAE     
##     0.25  2.505766  0.8044039  2.006183
##     0.50  2.260711  0.8183418  1.809843
##     1.00  2.078793  0.8394950  1.633594
##     2.00  1.969260  0.8531776  1.527600
##     4.00  1.899944  0.8578039  1.491375
##     8.00  1.879763  0.8586485  1.475681
##    16.00  1.877720  0.8585664  1.482805
##    32.00  1.877712  0.8585014  1.483396
##    64.00  1.877712  0.8585014  1.483396
##   128.00  1.877712  0.8585014  1.483396
## 
## Tuning parameter 'sigma' was held constant at a value of 0.05812698
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.05812698 and C = 32.
svmpredictions <- predict(svmmodel, newdata = testData$x)
postResample(pred = svmpredictions, obs = testData$y)
##      RMSE  Rsquared       MAE 
## 2.0634283 0.8273381 1.5679663
varImp(svmmodel)
## loess r-squared variable importance
## 
##      Overall
## X4  100.0000
## X1   95.5047
## X2   89.6186
## X5   45.2170
## X3   29.9330
## X9    6.3299
## X10   5.5182
## X8    3.2527
## X6    0.8884
## X7    0.0000
rbind(
  "mars" = postResample(pred = marspredictions, obs = testData$y),
  "svm" = postResample(pred = svmpredictions, obs = testData$y),
  "net" = postResample(pred = nnpredictions, obs = testData$y),
  "knn" = postResample(pred = knnPred, obs = testData$y)
)
##          RMSE  Rsquared      MAE
## mars 1.277999 0.9338365 1.014707
## svm  2.063428 0.8273381 1.567966
## net  1.926603 0.8530679 1.471759
## knn  3.204059 0.6819919 2.568346
  • The Mars model has the losest RMSE, the highest R2, and the lowest MAE.
  • X1-X5 are the top predictors in all models ran.
  • Analysis: I would chose the MARS model unless some other reason presented itself.

7.5

Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.

a.

Which nonlinear regression model gives the optimal resampling and test set performance?

library(AppliedPredictiveModeling)
data("ChemicalManufacturingProcess")
set.seed(42)

preP <- preProcess(ChemicalManufacturingProcess, 
                   method = c("BoxCox", "knnImpute", "center", "scale"))
df <- predict(preP, ChemicalManufacturingProcess)

df$Yield = ChemicalManufacturingProcess$Yield

trainRows <- createDataPartition(df$Yield, p = .80, list = FALSE)
df.train <- df[trainRows, ]
df.test <- df[-trainRows, ]
exclude <- which(colnames(df) == "Yield")
xtrain <- df.train[, -exclude]
ytrain <- df.train$Yield
xtest <- df.test[, -exclude]
ytest <- df.test$Yield

MARS

marsGrid <- expand.grid(.degree=1:2,
                        .nprune=2:10)

mars_model <- train(
  x = xtrain,
  y = ytrain,
  method = "earth",
  preProcess = c("center","scale"),
  tuneGrid = marsGrid,
  tuneLength = 10
        )

mars_model
## Multivariate Adaptive Regression Spline 
## 
## 144 samples
##  57 predictor
## 
## Pre-processing: centered (57), scaled (57) 
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 144, 144, 144, 144, 144, 144, ... 
## Resampling results across tuning parameters:
## 
##   degree  nprune  RMSE      Rsquared   MAE     
##   1        2      1.468432  0.3854713  1.162014
##   1        3      1.276332  0.5231733  1.021687
##   1        4      1.270135  0.5369475  1.011879
##   1        5      1.317958  0.5095408  1.045679
##   1        6      1.344973  0.4958556  1.061658
##   1        7      1.360999  0.4888739  1.072340
##   1        8      1.374832  0.4881251  1.074554
##   1        9      1.391794  0.4844014  1.077630
##   1       10      1.438006  0.4649259  1.102372
##   2        2      1.482258  0.3731996  1.170862
##   2        3      1.324510  0.4883376  1.053509
##   2        4      1.293718  0.5160405  1.023527
##   2        5      1.454376  0.4933105  1.049040
##   2        6      1.512474  0.4777561  1.064742
##   2        7      1.571612  0.4911945  1.067307
##   2        8      1.547459  0.4885791  1.060914
##   2        9      1.522435  0.4868360  1.066821
##   2       10      1.702510  0.4629505  1.128458
## 
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 4 and degree = 1.
marspredictions <- predict(mars_model, xtest)

mars <- postResample(pred = marspredictions, obs = ytest)

SVM

svmmodel <- train(
  x = xtrain,
  y = ytrain,
  method = "svmRadial",
  preProcess = c("center","scale"),
  tuneLength = 10,
  trControl = trainControl(method = "cv")
        )

svmmodel
## Support Vector Machines with Radial Basis Function Kernel 
## 
## 144 samples
##  57 predictor
## 
## Pre-processing: centered (57), scaled (57) 
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 129, 131, 131, 129, 128, 129, ... 
## Resampling results across tuning parameters:
## 
##   C       RMSE      Rsquared   MAE      
##     0.25  1.407769  0.4719019  1.1646254
##     0.50  1.294955  0.5184878  1.0522695
##     1.00  1.203443  0.5715253  0.9627867
##     2.00  1.158874  0.5854809  0.9348507
##     4.00  1.107940  0.6091838  0.8992096
##     8.00  1.092123  0.6197602  0.8895927
##    16.00  1.092123  0.6197602  0.8895927
##    32.00  1.092123  0.6197602  0.8895927
##    64.00  1.092123  0.6197602  0.8895927
##   128.00  1.092123  0.6197602  0.8895927
## 
## Tuning parameter 'sigma' was held constant at a value of 0.01693128
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.01693128 and C = 8.
svmpredictions <- predict(svmmodel, xtest)

svm <- postResample(pred = svmpredictions, obs = ytest)

Neural Network

nngrid <- expand.grid(size = c(1:10),
                        decay = c(0, 0.01, 0.1),
                        bag = FALSE)

ctrl <- trainControl(method = "cv")

nnmodel <- train(xtrain, ytrain,
                  method = "avNNet",
                  tuneGrid = nngrid,
                  trControl = ctrl,
                  preProcess = c("center", "scale"),
                  linout = TRUE,
                  trace = FALSE,
                  maxit = 500
                  )

nnmodel
## Model Averaged Neural Network 
## 
## 144 samples
##  57 predictor
## 
## Pre-processing: centered (57), scaled (57) 
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 131, 128, 129, 131, 130, 130, ... 
## Resampling results across tuning parameters:
## 
##   size  decay  RMSE      Rsquared   MAE     
##    1    0.00   1.508047  0.3932487  1.235517
##    1    0.01   1.581344  0.3786607  1.255111
##    1    0.10   1.814609  0.4338583  1.270631
##    2    0.00   1.488144  0.3542240  1.234525
##    2    0.01   1.452022  0.4742854  1.175461
##    2    0.10   1.798123  0.3986754  1.353560
##    3    0.00   1.689463  0.3893613  1.392070
##    3    0.01   1.684233  0.4197083  1.355959
##    3    0.10   2.095223  0.3297519  1.477792
##    4    0.00   1.730349  0.3717339  1.399099
##    4    0.01   1.912833  0.3696018  1.410385
##    4    0.10   2.081728  0.2845500  1.512914
##    5    0.00   1.838907  0.3409866  1.452365
##    5    0.01   1.934930  0.3574250  1.407448
##    5    0.10   2.121423  0.3393797  1.517933
##    6    0.00   2.056600  0.3537175  1.651309
##    6    0.01   1.713350  0.4099403  1.343987
##    6    0.10   2.027889  0.3497005  1.443503
##    7    0.00   3.199565  0.2074683  2.340726
##    7    0.01   1.585832  0.4063606  1.244766
##    7    0.10   2.021509  0.3654192  1.431197
##    8    0.00   3.972942  0.2364505  2.894159
##    8    0.01   1.789683  0.3645774  1.425752
##    8    0.10   2.002416  0.3891996  1.507188
##    9    0.00   5.155077  0.2612820  3.811597
##    9    0.01   1.842651  0.3867720  1.470110
##    9    0.10   1.874621  0.4170628  1.388579
##   10    0.00   5.711843  0.1305153  3.988211
##   10    0.01   2.565420  0.3305604  1.829478
##   10    0.10   1.674455  0.4020645  1.281144
## 
## Tuning parameter 'bag' was held constant at a value of FALSE
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were size = 2, decay = 0.01 and bag
##  = FALSE.
nnpredictions <- predict(nnmodel, xtest)

nn <- postResample(pred = nnpredictions, obs = ytest)

KNN

knnModel <- train(x = xtrain,
 y = ytrain,
 method = "knn",
 preProc = c("center", "scale"),
 tuneLength = 10)

knnModel
## k-Nearest Neighbors 
## 
## 144 samples
##  57 predictor
## 
## Pre-processing: centered (57), scaled (57) 
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 144, 144, 144, 144, 144, 144, ... 
## Resampling results across tuning parameters:
## 
##   k   RMSE      Rsquared   MAE     
##    5  1.494406  0.3492467  1.201961
##    7  1.477168  0.3614893  1.191511
##    9  1.470519  0.3690898  1.199675
##   11  1.463239  0.3787214  1.201348
##   13  1.462889  0.3810427  1.205321
##   15  1.464833  0.3855573  1.205486
##   17  1.469103  0.3848728  1.212392
##   19  1.475117  0.3813175  1.216769
##   21  1.484312  0.3771546  1.224719
##   23  1.485722  0.3818766  1.227935
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 13.
knnpredictions <- predict(knnModel, xtest)

knn <- postResample(pred = knnpredictions, obs = ytest)
rbind(
  "mars" = mars,
  "svm" = svm,
  "net" = nn,
  "knn" = knn
)
##          RMSE  Rsquared       MAE
## mars 1.205917 0.6862281 0.9506082
## svm  1.270269 0.6257162 0.9379920
## net  1.610104 0.4173795 1.3213569
## knn  1.307393 0.6760238 0.9760920
  • The MARS model performed better with the RMSE, however, its not going to be my choice.
  • The SVM model has a slightly higher RMSE but lower r2 and lower Mean Absolute error.

b.

Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?

varImp(svmmodel, 10)
## loess r-squared variable importance
## 
##   only 20 most important variables shown (out of 57)
## 
##                        Overall
## ManufacturingProcess32  100.00
## ManufacturingProcess13   94.69
## ManufacturingProcess09   90.80
## ManufacturingProcess17   88.88
## BiologicalMaterial06     84.11
## BiologicalMaterial03     80.13
## ManufacturingProcess36   74.71
## BiologicalMaterial12     73.94
## ManufacturingProcess06   70.12
## ManufacturingProcess11   62.92
## ManufacturingProcess31   56.87
## BiologicalMaterial02     51.24
## BiologicalMaterial11     48.43
## BiologicalMaterial09     45.48
## ManufacturingProcess30   42.23
## BiologicalMaterial08     40.10
## ManufacturingProcess33   39.16
## ManufacturingProcess29   38.87
## BiologicalMaterial04     38.44
## ManufacturingProcess25   37.15
varImp(mars_model, 10)
## earth variable importance
## 
##                        Overall
## ManufacturingProcess32     100
## ManufacturingProcess09       0
  • Manufacturing Processes dominate the top 10 predictors
  • Process 32 is the top predictor in both of the top models

c.

Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?

library(gridExtra)
## 
## Attaching package: 'gridExtra'
## The following object is masked from 'package:dplyr':
## 
##     combine
P1 <- ggplot(df.train, aes(ManufacturingProcess32, Yield))+
  geom_point()+
  geom_smooth(method = "lm", se=FALSE)

P2 <- ggplot(df.train, aes(ManufacturingProcess13, Yield))+
  geom_point()+
  geom_smooth(method = "lm", se=FALSE)

P3 <- ggplot(df.train, aes(ManufacturingProcess09, Yield))+
  geom_point()+
  geom_smooth(method = "lm", se=FALSE)

P4 <- ggplot(df.train, aes(ManufacturingProcess17, Yield))+
  geom_point()+
  geom_smooth(method = "lm", se=FALSE)

P5 <- ggplot(df.train, aes(BiologicalMaterial06, Yield))+
  geom_point()+
  geom_smooth(method = "lm", se=FALSE)

P6 <- ggplot(df.train, aes(BiologicalMaterial03, Yield))+
  geom_point()+
  geom_smooth(method = "lm", se=FALSE)

grid.arrange(P1, P2, P3, P4, P5, P6, nrow = 3)
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'

  • In the top 5 predictors, there appears to be a positive relationship on the odd predictors (1,3,5) and a negative relationship on the even predictors (2,4)