DATA 624.HW8
Evan McLaughlin
2024-11-08
In Kuhn and Johnson do problems 7.2 and 7.5. There are only two but they consist of many parts. Please submit a link to your Rpubs and submit the .rmd file as well.
7.2 - Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data:
y=10sin(π1x1x2)+20(x3−0.5)2+10x4+5x5+N(0,σ2)
where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:
set.seed(200)
trainingData = mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x = data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)
## or other methods.
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData = mlbench.friedman1(5000, sd = 1)
testData$x = data.frame(testData$x)(a) Tune several models on these data.
For example:
knnModel <- train(x = trainingData$x,
y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.466085 0.5121775 2.816838
## 7 3.349428 0.5452823 2.727410
## 9 3.264276 0.5785990 2.660026
## 11 3.214216 0.6024244 2.603767
## 13 3.196510 0.6176570 2.591935
## 15 3.184173 0.6305506 2.577482
## 17 3.183130 0.6425367 2.567787
## 19 3.198752 0.6483184 2.592683
## 21 3.188993 0.6611428 2.588787
## 23 3.200458 0.6638353 2.604529
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.The final value is k=15, driven by RMSE value.
#MARS
mars_grid <-expand.grid(.degree = 1:2, .nprune = 2:38)
set.seed(100)
mars_fit <- earth(trainingData$x, trainingData$y)
mars_pred <- predict(mars_fit, newdata = testData$x)
postResample(pred = mars_pred, obs = testData$y)## RMSE Rsquared MAE
## 1.8136467 0.8677298 1.3911836#MARS Tuned
mars_grid <- expand.grid(.degree = 1:2, .nprune = 2:38)
set.seed(100)
mars_tuned <- train(trainingData$x, trainingData$y, method = "earth",
tuneGrid = mars_grid,
trControl = trainControl(method = "cv"))
marstunedpred <- predict(mars_tuned, newdata = testData$x)
postResample(pred = marstunedpred, obs = testData$y)## RMSE Rsquared MAE
## 1.1589948 0.9460418 0.9250230#SVM Tuned
svmRTuned <- train(trainingData$x, trainingData$y,
method = "svmRadial",
preProcess = c("center", "scale"),
tuneLength = 15,
trControl = trainControl(method = "cv"))
svmPred <- predict(svmRTuned, newdata = testData$x)
postResample(svmPred, testData$y)## RMSE Rsquared MAE
## 2.0741473 0.8255848 1.5755185#Neural Network
nnet1 <- avNNet(trainingData$x, trainingData$y,
size = 5,
decay = 0.01,
repeats = 5,
linout = TRUE,
trace = FALSE,
maxit = 500)## Warning: executing %dopar% sequentially: no parallel backend registerednnet_pred <- predict(nnet1, newdata = testData$x)
postResample(pred = nnet_pred, obs = testData$y)## RMSE Rsquared MAE
## 1.5690654 0.9011935 1.2110238(b) - Which models appear to give the best performance?Does MARS select the informative predictors (those named X1–X5)?
MARS Tuned enjoys the highest R-Squared value and appears to be performing at the highest level, as the other metrics across the models are similar. The MARS model does indeed select the informative predictors X1-X5.
7.5 - Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
# load, impute, split data
data(ChemicalManufacturingProcess)
chem_imputed_df <- preProcess(ChemicalManufacturingProcess, "knnImpute")
chem_imputed_full_df <- predict(chem_imputed_df, ChemicalManufacturingProcess)
# eliminate low freq
low_values <- nearZeroVar(chem_imputed_full_df)
imp_chem_df <- chem_imputed_full_df[,-low_values]
#split
split_chem <- createDataPartition(imp_chem_df$Yield , p=.8, list=F)
imp_train_chem <- imp_chem_df[split_chem,]
imp_test_chem <- imp_chem_df[-split_chem,](a) Which nonlinear regression model gives the optimal resampling and test set performance?
# MARS Tuned
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:38)
mars_tuned2 <- train(Yield~. ,
data = imp_train_chem,
method = "earth",
tuneGrid = marsGrid,
trControl = trainControl(method = "cv"))
mars_tuned2_pred <- predict(mars_tuned2, imp_test_chem)
postResample(mars_tuned2_pred, imp_test_chem$Yield)## RMSE Rsquared MAE
## 0.8087286 0.5580902 0.6532567# KNN
knn_model <- train(Yield~.,
data = imp_train_chem,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knn_pred <- predict(knn_model, imp_test_chem)
postResample(pred = knn_pred, obs = imp_test_chem$Yield)## RMSE Rsquared MAE
## 0.9221184 0.4929803 0.7102900# SVM Tuned
svm_tuned_chem <- train(Yield~. ,
data = imp_train_chem,
method = "svmRadial",
tuneLength = 15,
trControl = trainControl(method = "cv"))
svm_tuned_pred_chem <- predict(svm_tuned_chem, imp_test_chem)
postResample(svm_tuned_pred_chem, imp_test_chem$Yield)## RMSE Rsquared MAE
## 0.6731789 0.7597536 0.4974924# Tuned Neural Network
neur_net_grid <- expand.grid(.decay = c(0, 0.01, .1),
.size = c(1:10), .bag = FALSE)
#neur_net_tune <- train(Yield~.,
# data = imp_train_chem,
# method = "avNNet",
# tuneGrid = neur_net_grid,
# trControl = trainControl(method = "cv"),
# linout = TRUE,trace = FALSE,
# MaxNWts = 10 * (ncol(imp_train_chem) + 1) + 10 + 1,
# maxit = 500)
#neur_net_pred <- predict(neur_net_tune, imp_test_chem)
#postResample(predict(neur_net_tune, imp_test_chem), imp_test_chem$Yield)Neural Network results here, so I don’t have to run the model again.
RMSE Rsquared MAE 0.6977449 0.6734879 0.5590664
The tuned SVM Model provides the optimal RMSE and R squared values (RMSE:0.6653114, R-Squared: 0.7258880)
(b) Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
plot(varImp(svm_tuned_chem), top=10)
Of the top 10 predictros, 7 are ManufacturingProcess and 3 are
BiologicalMaterial. ManufacturingProcess predictors pretty clearly
demonstrate more influence in the model.
(c) Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
corr_chem <- imp_chem_df %>%
dplyr::select('Yield', 'ManufacturingProcess32','ManufacturingProcess13',
'BiologicalMaterial06','ManufacturingProcess09',
'BiologicalMaterial03','ManufacturingProcess17',
'BiologicalMaterial12','ManufacturingProcess36',
'ManufacturingProcess09','ManufacturingProcess31','ManufacturingProcess06')
corr_chem_plot <- cor(corr_chem)
corrplot.mixed(corr_chem_plot, tl.col = 'black', tl.pos = 'lt', tl.cex = .8, number.cex = 0.5,
upper = "number", lower="circle")
Judging by the above correlation matrix, the strongest positive
correlations are with ManufacturingProcess32 and ManufacturingProcess09
(0.61 and 0.50 respectively). The strongest negative correlations are
with ManufacturingProcess36 and ManufacturingProcess13. Biological
Material predictors from the previous question do not appear to drive
Yield in either direction.
Kuhn, Max; Johnson, Kjell. Applied Predictive Modeling (p. 171). Springer New York. Kindle Edition.
R Markdown
This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.
When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:
summary(cars)## speed dist
## Min. : 4.0 Min. : 2.00
## 1st Qu.:12.0 1st Qu.: 26.00
## Median :15.0 Median : 36.00
## Mean :15.4 Mean : 42.98
## 3rd Qu.:19.0 3rd Qu.: 56.00
## Max. :25.0 Max. :120.00Including Plots
You can also embed plots, for example:
Note that the echo = FALSE parameter was added to the
code chunk to prevent printing of the R code that generated the
plot.