DATA 624 Homework 8

Author

Jonnathan Zuna

7.2

Friedman (1991) introduced several benchmark data sets created by simulation. One of these simulations used the following nonlinear equation to create data: where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data:

# Loading required packages
library(AppliedPredictiveModeling)
library(caret)
library(mlbench)
library(earth)
library(kernlab)
library(nnet)
library(pls)
library(RANN)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
trainingData$x <- data.frame(trainingData$x)

featurePlot(trainingData$x, trainingData$y)

testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)

200 training points and a test set of 5,000 points to estimate the true error. X1 to X5 are used in the equation and X6 to X10 are noise.

Tune several models on these data. Which models appear to give the best performance? Does MARS select the informative predictors (those named X1 to X5)?

ctrl <- trainControl(method = "cv", number = 10)

set.seed(624)
knn_fit <- train(trainingData$x, trainingData$y,
                 method = "knn",
                 preProc = c("center", "scale"),
                 tuneLength = 10,
                 trControl = ctrl)

knn_fit$bestTune
  k
3 9
set.seed(624)
mars_fit <- train(trainingData$x, trainingData$y,
                  method = "earth",
                  tuneGrid = expand.grid(degree = 1:2, nprune = 2:38),
                  trControl = ctrl)

mars_fit$bestTune
   nprune degree
50     14      2
set.seed(624)
svm_fit <- train(trainingData$x, trainingData$y,
                 method = "svmRadial",
                 preProc = c("center", "scale"),
                 tuneLength = 14,
                 trControl = ctrl)

svm_fit$bestTune
       sigma C
5 0.05607744 4
set.seed(624)
nnet_fit <- train(trainingData$x, trainingData$y,
                  method = "avNNet",
                  preProc = c("center", "scale"),
                  tuneGrid = expand.grid(decay = c(0, 0.01, 0.1),
                                         size = 1:10, bag = FALSE),
                  linout = TRUE, trace = FALSE,
                  maxit = 500, MaxNWts = 500,
                  trControl = ctrl)
Warning: executing %dopar% sequentially: no parallel backend registered
nnet_fit$bestTune
  size decay   bag
4    4     0 FALSE

kNN, MARS, SVM and a neural net were tuned with 10 fold cross validation. The optimal settings are k = 9, a degree 2 MARS with 14 terms, an SVM with C = 4, and a neural net with 4 hidden units and no decay.

perf <- function(fit, name, x_new, y_new) {
  cv <- getTrainPerf(fit)
  te <- postResample(predict(fit, x_new), y_new)
  data.frame(Model = name,
             CV_RMSE = cv$TrainRMSE, CV_R2 = cv$TrainRsquared,
             Test_RMSE = te["RMSE"], Test_R2 = te["Rsquared"])
}

comp <- rbind(perf(knn_fit, "kNN", testData$x, testData$y),
              perf(mars_fit, "MARS", testData$x, testData$y),
              perf(svm_fit, "SVM", testData$x, testData$y),
              perf(nnet_fit, "Neural net", testData$x, testData$y))
rownames(comp) <- NULL

comp
       Model  CV_RMSE     CV_R2 Test_RMSE   Test_R2
1        kNN 3.028589 0.6824162  3.117232 0.6556622
2       MARS 1.279383 0.9390664  1.172263 0.9448890
3        SVM 1.847956 0.8721190  2.044656 0.8308598
4 Neural net 1.919085 0.8612943  2.008473 0.8404012

MARS is the best model, with a test RMSE of 1.17 and R squared of 0.945. The neural net (2.00, 0.841) and SVM (2.04, 0.831) come next, and kNN is the weakest (3.12, 0.656). MARS fits the equation well because it is built from hinge terms and the degree 2 model can capture the X1 and X2 interaction. kNN uses all 10 predictors in its distances, so the 5 noise predictors blur which points are truly close.

summary(mars_fit$finalModel)
Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
            nprune=14)

                                coefficients
(Intercept)                        21.905319
h(0.621722-X1)                    -15.726181
h(X1-0.621722)                      9.234027
h(0.601063-X2)                    -18.253527
h(X2-0.601063)                     10.448545
h(0.447442-X3)                      9.700589
h(X3-0.606015)                     12.674694
h(0.734892-X4)                     -9.863526
h(X4-0.734892)                     10.297964
h(0.850094-X5)                     -5.324175
h(0.218266-X1) * h(X2-0.601063)   -55.358726
h(X1-0.218266) * h(X2-0.601063)   -29.100250
h(X1-0.621722) * h(X2-0.295997)   -26.833129
h(0.649253-X1) * h(0.601063-X2)    27.120721

Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
Termination condition: Reached nk 21
Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
Number of terms at each degree of interaction: 1 9 4
GCV 1.62945    RSS 225.8601    GRSq 0.9338437    RSq 0.953688
varImp(mars_fit)
earth variable importance

   Overall
X1  100.00
X4   75.24
X2   48.74
X5   15.53
X3    0.00

Yes. The summary shows MARS used 5 of the 10 predictors, X1 to X5, and X6 to X10 are unused. X3 shows an importance of 0 only because caret rescales the scores so the lowest used predictor is 0, but it appears in two of the model terms.

7.5

Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and preprocessing steps as before and train several nonlinear regression models.

data(ChemicalManufacturingProcess)

cmp <- ChemicalManufacturingProcess
x <- cmp[, -1]
yield <- cmp$Yield

set.seed(624)
idx <- createDataPartition(yield, p = 0.8, list = FALSE)

x_train <- x[idx, ]
x_test <- x[-idx, ]
y_train <- yield[idx]
y_test <- yield[-idx]

pp <- c("knnImpute", "nzv", "center", "scale")

Same split as Homework 7, 144 batches for training and 32 for testing, with the same imputation and preprocessing.

set.seed(624)
pls_cmp <- train(x_train, y_train,
                 method = "pls",
                 tuneLength = 15,
                 preProc = pp,
                 trControl = ctrl)

The PLS model from 6.3 is refit so its top predictors can be compared in part b.

set.seed(624)
knn_cmp <- train(x_train, y_train,
                 method = "knn",
                 tuneLength = 10,
                 preProc = pp,
                 trControl = ctrl)

set.seed(624)
mars_cmp <- train(x_train, y_train,
                  method = "earth",
                  tuneGrid = expand.grid(degree = 1:2, nprune = 2:30),
                  preProc = pp,
                  trControl = ctrl)

set.seed(624)
svm_cmp <- train(x_train, y_train,
                 method = "svmRadial",
                 tuneLength = 14,
                 preProc = pp,
                 trControl = ctrl)

set.seed(624)
nnet_cmp <- train(x_train, y_train,
                  method = "avNNet",
                  tuneGrid = expand.grid(decay = c(0, 0.01, 0.1),
                                         size = 1:5, bag = FALSE),
                  linout = TRUE, trace = FALSE,
                  maxit = 500, MaxNWts = 1000,
                  preProc = pp,
                  trControl = ctrl)

a. Which nonlinear regression model gives the optimal resampling and test set performance?

comp_cmp <- rbind(perf(knn_cmp, "kNN", x_test, y_test),
                  perf(mars_cmp, "MARS", x_test, y_test),
                  perf(svm_cmp, "SVM", x_test, y_test),
                  perf(nnet_cmp, "Neural net", x_test, y_test),
                  perf(pls_cmp, "PLS", x_test, y_test))
rownames(comp_cmp) <- NULL

comp_cmp
       Model  CV_RMSE     CV_R2 Test_RMSE   Test_R2
1        kNN 1.315749 0.5090465  1.402164 0.6129980
2       MARS 1.193540 0.5808758  1.282667 0.6705045
3        SVM 1.167241 0.5916503  1.146870 0.7474025
4 Neural net 1.403694 0.4615988  1.322341 0.6094226
5        PLS 1.368345 0.5231989  1.201579 0.6756127

The SVM is the optimal nonlinear model. It has the lowest resampled RMSE (1.167) and the best test results (RMSE 1.147, R squared 0.747). MARS has a slightly higher resampled R squared (0.597 vs 0.592), but its test RMSE is worse (1.288). The SVM also beats the PLS model from 6.3 on both resampled and test RMSE.

b. Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?

imp_svm <- varImp(svm_cmp)
top_svm <- rownames(imp_svm$importance)[order(-imp_svm$importance$Overall)][1:10]

imp_svm
loess r-squared variable importance

  only 20 most important variables shown (out of 57)

                       Overall
ManufacturingProcess32  100.00
BiologicalMaterial06     89.32
ManufacturingProcess36   79.78
BiologicalMaterial03     77.48
ManufacturingProcess09   73.90
ManufacturingProcess13   73.23
ManufacturingProcess31   70.84
BiologicalMaterial02     66.91
ManufacturingProcess11   65.19
BiologicalMaterial12     64.94
ManufacturingProcess06   61.04
ManufacturingProcess17   53.06
ManufacturingProcess29   50.91
BiologicalMaterial11     49.10
BiologicalMaterial04     48.27
ManufacturingProcess33   47.62
ManufacturingProcess30   47.38
BiologicalMaterial01     40.69
BiologicalMaterial08     38.18
ManufacturingProcess25   34.07
table(substr(top_svm, 1, 10))

Biological Manufactur 
         4          6 

6 of the top 10 are process predictors and 4 are biological, so the process predictors still lead, but less than in PLS (7 of 10). caret has no built in importance for SVMs, so it ranks each predictor by how well a loess curve of yield on that predictor alone fits.

imp_pls <- varImp(pls_cmp)
top_pls <- rownames(imp_pls$importance)[order(-imp_pls$importance$Overall)][1:10]

intersect(top_svm, top_pls)
[1] "ManufacturingProcess32" "BiologicalMaterial06"   "ManufacturingProcess36"
[4] "BiologicalMaterial03"   "ManufacturingProcess09" "ManufacturingProcess13"
[7] "BiologicalMaterial02"   "ManufacturingProcess11"
setdiff(top_svm, top_pls)
[1] "ManufacturingProcess31" "BiologicalMaterial12"  
setdiff(top_pls, top_svm)
[1] "ManufacturingProcess17" "ManufacturingProcess06"

8 of the top 10 are shared with PLS, and ManufacturingProcess32 is first in both. BiologicalMaterial06 moves up to second in the SVM list. ManufacturingProcess31 and BiologicalMaterial12 are only in the SVM list, while ManufacturingProcess17 and 06 are only in the PLS list.

c. Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?

unique_svm <- setdiff(top_svm, top_pls)

summary(cmp[, unique_svm])
 ManufacturingProcess31 BiologicalMaterial12
 Min.   : 0.00          Min.   :18.35       
 1st Qu.:70.10          1st Qu.:19.73       
 Median :70.80          Median :20.12       
 Mean   :70.18          Mean   :20.20       
 3rd Qu.:71.40          3rd Qu.:20.75       
 Max.   :72.50          Max.   :22.21       
 NA's   :5                                  

ManufacturingProcess31 has one batch at 0 while its first quartile is 70.1, so that batch is likely a recording error. I leave it out of the plot so it does not stretch the axis.

keep <- which(cmp$ManufacturingProcess31 > 0)

round(cor(cmp[keep, unique_svm], cmp$Yield[keep], use = "complete.obs"), 3)
                         [,1]
ManufacturingProcess31 -0.366
BiologicalMaterial12    0.372
featurePlot(x = cmp[keep, unique_svm], y = cmp$Yield[keep],
            plot = "scatter", type = c("p", "smooth"),
            layout = c(2, 1))

ManufacturingProcess31 has a negative relationship with yield (-0.37), and the curve shows yield dropping once it goes above about 68. It is a process predictor, so keeping it lower is worth testing. BiologicalMaterial12 (0.37) rises with yield up to about 21 and then dips slightly, a curved shape a linear model like PLS cannot fully use. It cannot be changed, but it could help screen raw material lots. These are correlations, so any change should be tested before assuming it improves yield.