Non-Linear Regression
Michael Gankhuyag
April 7, 2020
7.2
Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data: y = 10 sin(πx1x2) + 20(x3 − 0.5)2 + 10x4 + 5x5 + N(0, σ2)
where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
library(caret)## Loading required package: lattice## Loading required package: ggplot2library(mlbench)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)
## or other methods.
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)Tune several models on these data. Which models appear to give the best performance? Does MARS select the informative predictors (those named X1–X5)?
KNN
knnModel <- train(x = trainingData$x, y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.466085 0.5121775 2.816838
## 7 3.349428 0.5452823 2.727410
## 9 3.264276 0.5785990 2.660026
## 11 3.214216 0.6024244 2.603767
## 13 3.196510 0.6176570 2.591935
## 15 3.184173 0.6305506 2.577482
## 17 3.183130 0.6425367 2.567787
## 19 3.198752 0.6483184 2.592683
## 21 3.188993 0.6611428 2.588787
## 23 3.200458 0.6638353 2.604529
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.knnPred <- predict(knnModel, newdata = testData$x)
## The function 'postResample' can be used to get the test set
## perforamnce values
postResample(pred = knnPred, obs = testData$y)## RMSE Rsquared MAE
## 3.2040595 0.6819919 2.5683461KNN RMSE IS 3.1751
MARS
library(Formula)
library(plotmo)## Loading required package: plotrix## Loading required package: TeachingDemoslibrary(plotrix)
library(TeachingDemos)
library(earth)
marsFit<- earth(trainingData$x, trainingData$y)
marsFit## Selected 12 of 18 terms, and 6 of 10 predictors
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 11 (additive model)
## GCV 2.540556 RSS 397.9654 GRSq 0.8968524 RSq 0.9183982# Define the candidate models to test
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
# Fix the seed so that the results can be reproduced
set.seed(100)
marsTuned <- train(trainingData$x, trainingData$y,
method = "earth",
# Explicitly declare the candidate models to test
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))marsTuned## Multivariate Adaptive Regression Spline
##
## 200 samples
## 10 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## degree nprune RMSE Rsquared MAE
## 1 2 4.327937 0.2544880 3.600474
## 1 3 3.572450 0.4912720 2.895811
## 1 4 2.596841 0.7183600 2.106341
## 1 5 2.370161 0.7659777 1.918669
## 1 6 2.276141 0.7881481 1.810001
## 1 7 1.766728 0.8751831 1.390215
## 1 8 1.780946 0.8723243 1.401345
## 1 9 1.665091 0.8819775 1.325515
## 1 10 1.663804 0.8821283 1.327657
## 1 11 1.657738 0.8822967 1.331730
## 1 12 1.653784 0.8827903 1.331504
## 1 13 1.648496 0.8823663 1.316407
## 1 14 1.639073 0.8841742 1.312833
## 1 15 1.639073 0.8841742 1.312833
## 1 16 1.639073 0.8841742 1.312833
## 1 17 1.639073 0.8841742 1.312833
## 1 18 1.639073 0.8841742 1.312833
## 1 19 1.639073 0.8841742 1.312833
## 1 20 1.639073 0.8841742 1.312833
## 1 21 1.639073 0.8841742 1.312833
## 1 22 1.639073 0.8841742 1.312833
## 1 23 1.639073 0.8841742 1.312833
## 1 24 1.639073 0.8841742 1.312833
## 1 25 1.639073 0.8841742 1.312833
## 1 26 1.639073 0.8841742 1.312833
## 1 27 1.639073 0.8841742 1.312833
## 1 28 1.639073 0.8841742 1.312833
## 1 29 1.639073 0.8841742 1.312833
## 1 30 1.639073 0.8841742 1.312833
## 1 31 1.639073 0.8841742 1.312833
## 1 32 1.639073 0.8841742 1.312833
## 1 33 1.639073 0.8841742 1.312833
## 1 34 1.639073 0.8841742 1.312833
## 1 35 1.639073 0.8841742 1.312833
## 1 36 1.639073 0.8841742 1.312833
## 1 37 1.639073 0.8841742 1.312833
## 1 38 1.639073 0.8841742 1.312833
## 2 2 4.327937 0.2544880 3.600474
## 2 3 3.572450 0.4912720 2.895811
## 2 4 2.661826 0.7070510 2.173471
## 2 5 2.404015 0.7578971 1.975387
## 2 6 2.243927 0.7914805 1.783072
## 2 7 1.856336 0.8605482 1.435682
## 2 8 1.754607 0.8763186 1.396841
## 2 9 1.603578 0.8938666 1.261361
## 2 10 1.492421 0.9084998 1.168700
## 2 11 1.317350 0.9292504 1.033926
## 2 12 1.304327 0.9320133 1.019108
## 2 13 1.277510 0.9323681 1.002927
## 2 14 1.269626 0.9350024 1.003346
## 2 15 1.266217 0.9359400 1.013893
## 2 16 1.268470 0.9354868 1.011414
## 2 17 1.268470 0.9354868 1.011414
## 2 18 1.268470 0.9354868 1.011414
## 2 19 1.268470 0.9354868 1.011414
## 2 20 1.268470 0.9354868 1.011414
## 2 21 1.268470 0.9354868 1.011414
## 2 22 1.268470 0.9354868 1.011414
## 2 23 1.268470 0.9354868 1.011414
## 2 24 1.268470 0.9354868 1.011414
## 2 25 1.268470 0.9354868 1.011414
## 2 26 1.268470 0.9354868 1.011414
## 2 27 1.268470 0.9354868 1.011414
## 2 28 1.268470 0.9354868 1.011414
## 2 29 1.268470 0.9354868 1.011414
## 2 30 1.268470 0.9354868 1.011414
## 2 31 1.268470 0.9354868 1.011414
## 2 32 1.268470 0.9354868 1.011414
## 2 33 1.268470 0.9354868 1.011414
## 2 34 1.268470 0.9354868 1.011414
## 2 35 1.268470 0.9354868 1.011414
## 2 36 1.268470 0.9354868 1.011414
## 2 37 1.268470 0.9354868 1.011414
## 2 38 1.268470 0.9354868 1.011414
##
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 15 and degree = 2.varImp(marsTuned)## earth variable importance
##
## Overall
## X1 100.00
## X4 85.14
## X2 69.24
## X5 49.31
## X3 40.00
## X9 0.00
## X6 0.00
## X8 0.00
## X7 0.00
## X10 0.00Support Vector Machines
library("kernlab")##
## Attaching package: 'kernlab'## The following object is masked from 'package:ggplot2':
##
## alphasvmRTuned <- train(trainingData$x, y=trainingData$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = trainControl(method = "cv"))
svmRTuned## Support Vector Machines with Radial Basis Function Kernel
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 2.536604 0.7865906 2.035796
## 0.50 2.262783 0.8033031 1.800168
## 1.00 2.087501 0.8225671 1.636606
## 2.00 1.973976 0.8359125 1.540666
## 4.00 1.890698 0.8494340 1.489189
## 8.00 1.837269 0.8573143 1.465238
## 16.00 1.830462 0.8587729 1.459150
## 32.00 1.830462 0.8587729 1.459150
## 64.00 1.830462 0.8587729 1.459150
## 128.00 1.830462 0.8587729 1.459150
## 256.00 1.830462 0.8587729 1.459150
## 512.00 1.830462 0.8587729 1.459150
## 1024.00 1.830462 0.8587729 1.459150
## 2048.00 1.830462 0.8587729 1.459150
##
## Tuning parameter 'sigma' was held constant at a value of 0.06450665
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.06450665 and C = 16.Neural Networks
nnetAvg <- train(trainingData$x, trainingData$y,
method = "avNNet",
tuneGrid = expand.grid(.decay = c(0,.01,.1),
.size=c(1:10),
.bag = FALSE),
trControl = trainControl(method = "cv", number = 10),
preProc = c("center", "scale"),
linout = TRUE,
## Reduce the amount of printed output
trace = FALSE,
##Expand the number of iterations to find parameter estimates
maxit = 500,
## and the number of parameters used by the model
MaxNWts = 10 * (ncol(trainingData$x) + 1) + 10 + 1)## Warning: executing %dopar% sequentially: no parallel backend registerednnetAvg## Model Averaged Neural Network
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## decay size RMSE Rsquared MAE
## 0.00 1 2.431864 0.7772129 1.922115
## 0.00 2 2.544101 0.7597992 2.002097
## 0.00 3 2.045626 0.8445616 1.665309
## 0.00 4 1.905381 0.8652435 1.551288
## 0.00 5 2.211015 0.8244115 1.754847
## 0.00 6 3.291727 0.7044813 2.232157
## 0.00 7 3.174662 0.7176137 2.236733
## 0.00 8 4.081524 0.5714388 2.650963
## 0.00 9 4.830687 0.5119711 2.722465
## 0.00 10 3.730703 0.6434741 2.383633
## 0.01 1 2.434671 0.7799828 1.904438
## 0.01 2 2.553838 0.7617523 2.007348
## 0.01 3 2.141779 0.8354917 1.675131
## 0.01 4 2.101448 0.8374858 1.684044
## 0.01 5 2.088530 0.8402610 1.616977
## 0.01 6 2.289970 0.8046163 1.789660
## 0.01 7 2.259714 0.8112816 1.769390
## 0.01 8 2.361376 0.7994090 1.910964
## 0.01 9 2.445780 0.7716093 1.984591
## 0.01 10 2.405197 0.7932173 1.885702
## 0.10 1 2.441609 0.7792296 1.907351
## 0.10 2 2.498457 0.7696395 1.935508
## 0.10 3 2.159366 0.8270132 1.718050
## 0.10 4 2.104034 0.8366465 1.704954
## 0.10 5 2.120954 0.8304251 1.659209
## 0.10 6 2.180896 0.8330111 1.751107
## 0.10 7 2.279417 0.8109326 1.834483
## 0.10 8 2.298283 0.8123087 1.842203
## 0.10 9 2.248915 0.8141416 1.775279
## 0.10 10 2.413067 0.7840677 1.871093
##
## Tuning parameter 'bag' was held constant at a value of FALSE
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were size = 4, decay = 0 and bag = FALSE.Model RMSE Rsquared MAE Neural Network 1.924841 0.8444967 1.528610 SVM 1.859167 0.8690215 1.497942 MARS 1.252080 0.9375471 0.984975 KNN 3.261973 0.6267032 2.616956
Comparing all the Models, MARS Model appears to give the best performance. Out of all the models, MARS had the lowest RMSE and the highest Rsquared score. MARs does select informative predictors.
7.5
7.5. Exercise 6.3 describes data for a chemical manufacturing process. Usethe same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
Which nonlinear regression model gives the optimal resampling and test set performance?
Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?