DATA 624 HW 8
Marley Myrianthopoulos
2024-11-10
library(AppliedPredictiveModeling)
library(caret)## Loading required package: ggplot2## Loading required package: lattice7.2
Friedman (1991) introduced several benchmark data sets create by
simulation. One of these simulations used the following nonlinear
equation to create data: \(22
y=10sin(πx1x2)+20(x3−0.5) +10x4+5x5+N(0,σ)\) where the \(x\) values are random variables uniformly
distributed between [0, 1] (there are also 5 other non-informative
variables also created in the simulation). The package
mlbench contains a function called
mlbench.friedman1 that simulates these data:
# Code provided by the textbook
library(mlbench)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)
## or other methods.
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)Tune several models on these data. For example:
# Code provided by the textbook
library(caret)
knnModel <- train(x = trainingData$x,
y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.466085 0.5121775 2.816838
## 7 3.349428 0.5452823 2.727410
## 9 3.264276 0.5785990 2.660026
## 11 3.214216 0.6024244 2.603767
## 13 3.196510 0.6176570 2.591935
## 15 3.184173 0.6305506 2.577482
## 17 3.183130 0.6425367 2.567787
## 19 3.198752 0.6483184 2.592683
## 21 3.188993 0.6611428 2.588787
## 23 3.200458 0.6638353 2.604529
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.knnPred <- predict(knnModel, newdata = testData$x)
## The function 'postResample' can be used to get the test set
## perforamnce values
postResample(pred = knnPred, obs = testData$y)## RMSE Rsquared MAE
## 3.2040595 0.6819919 2.5683461SVM
svmModel <- train(x = trainingData$x,
y = trainingData$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 10)
svmModel## Support Vector Machines with Radial Basis Function Kernel
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## C RMSE Rsquared MAE
## 0.25 2.545335 0.7804647 2.015121
## 0.50 2.319786 0.7965148 1.830009
## 1.00 2.188349 0.8119636 1.726027
## 2.00 2.103655 0.8241314 1.655842
## 4.00 2.066889 0.8294300 1.631059
## 8.00 2.052680 0.8313928 1.623553
## 16.00 2.049874 0.8318301 1.621839
## 32.00 2.049874 0.8318301 1.621839
## 64.00 2.049874 0.8318301 1.621839
## 128.00 2.049874 0.8318301 1.621839
##
## Tuning parameter 'sigma' was held constant at a value of 0.06802164
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were sigma = 0.06802164 and C = 16.MARS
marsModel <- train(x = trainingData$x,
y = trainingData$y,
method = "earth",
preProc = c("center", "scale"),
tuneLength = 10)## Loading required package: earth## Warning: package 'earth' was built under R version 4.3.3## Loading required package: Formula## Loading required package: plotmo## Warning: package 'plotmo' was built under R version 4.3.3## Loading required package: plotrixmarsModel## Multivariate Adaptive Regression Spline
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## nprune RMSE Rsquared MAE
## 2 4.447386 0.2254125 3.620675
## 3 3.790305 0.4344625 3.058704
## 4 2.801182 0.6884819 2.233531
## 6 2.493135 0.7492201 1.986528
## 7 2.089713 0.8239588 1.645996
## 9 1.816053 0.8673608 1.420333
## 10 1.819611 0.8674028 1.417343
## 12 1.832487 0.8651613 1.426371
## 13 1.845943 0.8632112 1.436005
## 15 1.854557 0.8617322 1.452920
##
## Tuning parameter 'degree' was held constant at a value of 1
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were nprune = 9 and degree = 1.7.2 (cont)
Which models appear to give the best performance? Does MARS select the informative predictors (those named X1–X5)?
KNN Performance
knnPred <- predict(knnModel, newdata = testData$x)
postResample(pred = knnPred, obs = testData$y)## RMSE Rsquared MAE
## 3.2040595 0.6819919 2.5683461SVM Performance
svmPred <- predict(svmModel, newdata = testData$x)
postResample(pred = svmPred, obs = testData$y)## RMSE Rsquared MAE
## 2.0864652 0.8236735 1.5854649MARS Performance
marsPred <- predict(marsModel, newdata = testData$x)
postResample(pred = marsPred, obs = testData$y)## RMSE Rsquared MAE
## 1.7901760 0.8705315 1.3712537Summary of Findings
Of these three models, the MARS model has the lowest RMSE and the largest \(R^2\), so it appears that this model had the best performance.
varImp(marsModel)## earth variable importance
##
## Overall
## X1 100.00
## X4 82.92
## X2 64.47
## X5 40.67
## X3 28.65
## X6 0.00The top 5 variables by variable importance are X1-X5, as desired.
7.5
Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
# Code provided by the textbook
library(AppliedPredictiveModeling)
data(ChemicalManufacturingProcess)
# Impute missing values using knn
library(impute)
chemicals_impute <- impute.knn(as.matrix(ChemicalManufacturingProcess), rng.seed = 31415)
chemicals_impute <- as.data.frame(chemicals_impute$data)
# Create a test/train split
training_rows_chemicals <- createDataPartition(chemicals_impute$Yield, p = 0.8, list = FALSE)
train_predictors_chemicals <- chemicals_impute[training_rows_chemicals,-1]
train_yield <- chemicals_impute[training_rows_chemicals, 1]
test_predictors_chemicals <- chemicals_impute[-training_rows_chemicals,-1]
test_yield <- chemicals_impute[-training_rows_chemicals, 1]KNN:
chemicals_knn <- train(x = train_predictors_chemicals,
y = train_yield,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knn_predict <- predict(chemicals_knn, test_predictors_chemicals)
yield_knn <- data.frame(obs = test_yield, pred = knn_predict)
print(defaultSummary(yield_knn))## RMSE Rsquared MAE
## 1.2892006 0.3525554 1.0741071MARS:
suppressWarnings(chemicals_mars <- train(train_predictors_chemicals,
train_yield,
method = 'earth',
preProc = c('center', 'scale')))
mars_predict <- predict(chemicals_mars, test_predictors_chemicals)
yield_mars <- data.frame(obs = test_yield, pred = mars_predict)
yield_mars$pred <- yield_mars$y
yield_mars <- yield_mars[,-2]
print(defaultSummary(yield_mars))## RMSE Rsquared MAE
## 1.2303707 0.5130859 1.0384975Support Vector Machine:
suppressWarnings(chemicals_svm <- train(train_predictors_chemicals,
train_yield,
method = 'svmRadial',
preProc = c('center', 'scale')))
svm_predict <- predict(chemicals_svm, test_predictors_chemicals)
yield_svm <- data.frame(obs = test_yield, pred = svm_predict)
print(defaultSummary(yield_svm))## RMSE Rsquared MAE
## 1.1103923 0.5152694 0.88979767.5(a)
Which nonlinear regression model gives the optimal resampling and test set performance?
Of the three models I trained, the SVM model had the lowest RMSE and the largest \(R^2\) value.
7.5(b)
Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
varImp(chemicals_svm, scale = FALSE)## loess r-squared variable importance
##
## only 20 most important variables shown (out of 57)
##
## Overall
## ManufacturingProcess13 0.3468
## ManufacturingProcess32 0.3410
## ManufacturingProcess17 0.3081
## BiologicalMaterial06 0.3072
## ManufacturingProcess31 0.3033
## BiologicalMaterial03 0.2858
## ManufacturingProcess09 0.2857
## BiologicalMaterial12 0.2815
## ManufacturingProcess36 0.2796
## ManufacturingProcess06 0.2516
## BiologicalMaterial02 0.2306
## ManufacturingProcess30 0.2073
## BiologicalMaterial11 0.1992
## ManufacturingProcess11 0.1855
## ManufacturingProcess29 0.1841
## BiologicalMaterial08 0.1825
## BiologicalMaterial09 0.1794
## BiologicalMaterial04 0.1774
## ManufacturingProcess33 0.1720
## BiologicalMaterial01 0.1416The top ten important predictors from the linear model (HW 7) were Manufacturing Processes 32, 36, 13, 09, 17, 06, and 33, as well as Biological Materials 02, 06, and 08.
The top ten important predictors from this SVM model were Manufacturing Processes 13, 32, 17, 31, 09, 36, and 06, as well as Biological Materials 06, 03, and 12.
Appearing on both lists were MP32, MP36, MP13, MP09, MP17, MP06, and BM06. Of these, the predictors that were ranked higher (closer to most important) on the list of important predictors) in the new SVM model were MP13, MP17, and BM06, while the remaining predictors that appear on both lists were ranked lower (closer to least important) in the SVM model.
Appearing on only the linear model list were MP33, BM02, and BM08.
Appearing on only the SVM list were MP31, BM03, and BM12.
At least in the top 10, the balance of Manufacturing Process predictors and Biological Material predictors was the same, there were 7 Manufacturing Process predictors and 3 Biological Material predictors in the top 10 most important predictors in both models. However, in the SVM model there are two Biological Material predictors in the top 6 most important predictors, whereas in the linear model there were none (all of the top 6 most important predictors in the linear model were Manufacturing Process predictors). Additionally, the top 20 most important predictors in the linear model included 12 Manufacturing Process predictors and 8 Biological Material predictors, while the new SVM model’s top 20 most important predictors include 11 Manufacturing Process predictors and 9 Biological Material predictors. Overall, the importance of the Biological Material predictors is slightly elevated in the new SVM model compared to the linear model.
7.5(c)
Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
ggplot(chemicals_impute, aes(x = ManufacturingProcess31, y = Yield)) + geom_point()
ggplot(chemicals_impute, aes(x = BiologicalMaterial03, y = Yield)) + geom_point()
ggplot(chemicals_impute, aes(x = BiologicalMaterial12, y = Yield)) + geom_point()
Each of these plots indicate that care needs to be taken not to over-do it with any of the predictor variables. In all three plots, the largest Yield is achieved before the right end of the graph, with points further right (indicating more of the particular biological material or a larger role for the manufacturing process component) resulting in a lower yield, sometimes drastically so.