Response 7.2
Replicating the data
We replicate the code provided by the authors to generate the Friedman simulated dataset for training and test dataset. The training data uses 200 observations while the test data uses 5000 observations to achieve a high degree of precision.
set.seed(200)
trainingData = mlbench.friedman1( 200, sd = 1 )
## We convert the 'x' data from a matrix to a dataframe.
## One reason is that this will give the columns names
trainingData$x = data.frame( trainingData$x )The featurePlot below shows the scatterplot between the predictors and the response variable.
Our solution tunes a kNN model, a MARS model and an SVM model on the above training dataset. We conclude that the MARS model has the best test performance and includes the informative variables only.
KNN Model Results
We replicate the knn tuning process by the authors to confirm our results match the stated numbers. We find that they are close but not identical. This suggests that the underlying algorithms have changed even though we have uses the same random seeds and code invocation.
options(digits = 3)
set.seed(1020)
knnModel = train( x= trainingData$x, y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10)
knnModel## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 200, 200, 200, 200, 200, 200, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.49 0.500 2.82
## 7 3.34 0.546 2.71
## 9 3.28 0.572 2.66
## 11 3.22 0.600 2.62
## 13 3.21 0.614 2.60
## 15 3.19 0.634 2.58
## 17 3.19 0.644 2.58
## 19 3.21 0.642 2.61
## 21 3.23 0.647 2.63
## 23 3.23 0.656 2.64
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 17.The model test performance shows an \(R^2\) of 68.2% below.
## RMSE Rsquared MAE
## 3.204 0.682 2.568MARS Model Results
marsGrid = expand.grid( .degree = 1:2, .nprune = 2:20)
repeats = 4
ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = repeats, selectionFunction = "best", verboseIter = FALSE)
set.seed(1000)
marsTuned = caret::train(x = trainingData$x,
y = trainingData$y ,
method = "earth" ,
metric = "RMSE" ,
tuneGrid = marsGrid ,
preProc = c("center", "scale") ,
trControl = ctrl )The model hinge function and coefficients are shown below. The best model has 11 terms, degree 2 and \(R^2\) of 94.6%.
## Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
## nprune=16)
##
## coefficients
## (Intercept) 22.04
## h(0.507267-X1) -4.26
## h(X1-0.507267) 2.66
## h(0.325504-X2) -5.26
## h(-0.216741-X3) 2.91
## h(X3- -0.216741) 1.66
## h(0.953812-X4) -2.81
## h(X4-0.953812) 2.83
## h(1.17878-X5) -1.50
## h(-0.951872-X1) * h(X2-0.325504) -3.62
## h(X1- -0.951872) * h(X2-0.325504) -1.51
## h(X1-0.507267) * h(X2- -0.798188) -2.20
## h(0.606835-X1) * h(0.325504-X2) 1.98
## h(0.325504-X2) * h(X3-0.795427) 2.14
## h(X2-0.325504) * h(X3- -0.917499) 1.34
## h(X2-0.325504) * h(-0.917499-X3) 4.48
##
## Selected 16 of 21 terms, and 5 of 10 predictors (nprune=16)
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 8 7
## GCV 1.64 RSS 214 GRSq 0.933 RSq 0.956
## RMSE Rsquared MAE
## 1.279 0.934 1.009The test performance shows the model is very accurate at 93.4% \(R^2\) with RMSE 1.28.
If we look at the variable importance plots, we see that X1, X4, X2, X5, X3 are the top variables.
Moreover, the MARS model clearly selects the useful predictors \(X1, X2, X3, X4, X5\) to include in the model.
SVM Model Results
We run the support vector machine model using a radial basis function on the training data. After scaling and centering the data, we use the oneSE rule to choose the model based on RMSE and use repeated cross validation to generate the performance.
svmGrid = expand.grid( C = c( 1.0, 5, 10, 15, 30, 100 ), sigma = c( 0.005, 0.01, 0.015, 0.02, 0.04, 0.10 ))
repeats = 4
ctrl <- trainControl(method = "cv", number = 10, selectionFunction = "oneSE")
set.seed(1000)
svmTuned = caret::train(x = trainingData$x ,
y = trainingData$y,
method = "svmRadial" ,
metric = "RMSE" ,
tuneGrid = svmGrid,
preProc = c("center", "scale") ,
trControl = ctrl )
svmTuned## Support Vector Machines with Radial Basis Function Kernel
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## C sigma RMSE Rsquared MAE
## 1 0.005 2.63 0.767 2.14
## 1 0.010 2.43 0.778 1.96
## 1 0.015 2.35 0.787 1.88
## 1 0.020 2.32 0.793 1.85
## 1 0.040 2.17 0.818 1.70
## 1 0.100 2.17 0.829 1.70
## 5 0.005 2.38 0.779 1.91
## 5 0.010 2.26 0.800 1.78
## 5 0.015 2.11 0.824 1.65
## 5 0.020 1.99 0.844 1.55
## 5 0.040 1.88 0.864 1.45
## 5 0.100 2.05 0.845 1.62
## 10 0.005 2.35 0.785 1.87
## 10 0.010 2.12 0.821 1.66
## 10 0.015 1.94 0.852 1.51
## 10 0.020 1.88 0.862 1.47
## 10 0.040 1.89 0.865 1.46
## 10 0.100 2.05 0.845 1.62
## 15 0.005 2.30 0.793 1.83
## 15 0.010 2.02 0.839 1.57
## 15 0.015 1.89 0.861 1.47
## 15 0.020 1.86 0.866 1.45
## 15 0.040 1.93 0.863 1.50
## 15 0.100 2.05 0.845 1.62
## 30 0.005 2.17 0.813 1.69
## 30 0.010 1.89 0.862 1.47
## 30 0.015 1.86 0.868 1.45
## 30 0.020 1.83 0.873 1.43
## 30 0.040 1.91 0.867 1.51
## 30 0.100 2.05 0.845 1.62
## 100 0.005 1.90 0.860 1.47
## 100 0.010 1.84 0.872 1.45
## 100 0.015 1.88 0.868 1.45
## 100 0.020 1.99 0.857 1.54
## 100 0.040 1.90 0.868 1.51
## 100 0.100 2.05 0.845 1.62
##
## RMSE was used to select the optimal model using the one SE rule.
## The final values used for the model were sigma = 0.04 and C = 5.## RMSE Rsquared MAE
## 2.031 0.833 1.529The SVM model produces a respectable performance on test data of \(R^2\) equal to 83.3% and RMSE 2.03. It includes a number of spurious variables in the variable importance plot including X9, X10, X8, X6, X7.
Therefore while the predictive accuracy is decent, the variable importance is misleading. The last 5 predictors in the plot are all independent of the response.
Discussion
The table below summarized the test performance of the 3 models tested. We observe that MARS RMSE and \(R^2\) both beat those of kNN and SVM and conclude that MARS has the best test performance. Because MARS is able to do variable selection, it selects only informative predictors whereas SVM and kNN include uninformative predictors.
Test Performance for both models
| models | RMSE | RSquared |
|---|---|---|
| kNN | 3.20 | 0.682 |
| MARS | 1.28 | 0.934 |
| SVM - Radial Basis Function | 2.03 | 0.833 |
