Chap 6
Bruce Kim
October 19, 2016
Excercises
First, to turn on the parallel computing options for the speed.
num_core <- detectCores() - 1
cl <- makeCluster(num_core)
registerDoParallel(cl)Prob. 1
Put it short, according to the thoeory of IR, the unique structure of molecular absorbs the different spectrum. So, since the fat absorbs distinct spectrum given the theory, one can predict how much fat is on the food based on the predictive model, which will save the time and cost.
library(caret)
data(tecator)The matrix absorp contains the 100 absorbance values for the 215 samples, while matrix endpoints contains the percent of moisture, fat, and protein in columns 1–3, respectively.
- In this example the predictors are the measurements at the individual frequencies. Because the frequencies lie in a systematic order (850–1,050nm), the predictors have a high degree of correlation. Hence, the data lie in a smaller dimension than the total number of predictors (215). Use PCA to determine the effective dimension of these data. What is the effective dimension?
suppressMessages(library(pls))
pca.fit <- prcomp(absorp,center = TRUE,scale = TRUE)
s <- summary(pca.fit)
plot(1:100, s$importance[3,], type = "b")
Given the plot, the first principal components explanes most of the variance in the data. And after 5 componants, the variance explaned reached 100%. It means there is high corlinearity among variables.
- Split the data into a training and a test set, pre-process the data, and build each variety of models described in this chapter. For those models with tuning parameters, what are the optimal values of the tuning parameter(s)?
Before spliting the data, the data, predictors and the outcome should be merged.
data <- data.frame(absorp,endpoints[,2])
colnames(data) <- c(names(data[,1:100]), "Y")
set.seed(5647)
iTrain <- createDataPartition(y = data$Y, p = 0.75,
list = FALSE)
Train <- data[iTrain, ]
Test <- data[-iTrain, ]- Regularization method. Ridge, Lasso, Elastic net(0<alpha<1)
set.seed(13579)
ctrl <- trainControl(method = "repeatedcv", repeats = 3)
ridge.fit <- train(Y ~., data = data, method = "ridge", trControl = ctrl, preProc = c("center", "scale"), tuneLength = 20) ## Loading required package: elasticnet## Loading required package: lars## Loaded lars 1.2lasso.fit <- train(Y ~., data = data, method = "lasso", trControl = ctrl, preProc = c("center", "scale"), tuneLength = 20)
enet.fit <- train(Y ~., data = data, method = "enet", trControl = ctrl, preProc = c("center", "scale"), tuneLength = 20)
enet.fit$results## lambda fraction RMSE Rsquared RMSESD RsquaredSD
## 1 0.0000000000 0.05 2.344134 0.9682226 0.8031108 0.02053655
## 21 0.0001000000 0.05 8.837021 0.5942251 1.1140901 0.14332272
## 41 0.0001467799 0.05 9.201782 0.5449249 1.1454305 0.15472416
## 61 0.0002154435 0.05 9.447467 0.5099816 1.1621132 0.16131197
## 81 0.0003162278 0.05 9.615203 0.4855047 1.1728834 0.16496769
## 101 0.0004641589 0.05 9.739371 0.4671418 1.1808070 0.16709307
## 121 0.0006812921 0.05 9.846836 0.4511520 1.1890826 0.16837667
## 141 0.0010000000 0.05 9.955054 0.4348800 1.1978882 0.16933833
## 161 0.0014677993 0.05 10.069388 0.4175310 1.2083325 0.17003138
## 181 0.0021544347 0.05 10.185949 0.3997783 1.2195781 0.17031194
## 201 0.0031622777 0.05 10.303157 0.3818581 1.2314912 0.17021549
## 221 0.0046415888 0.05 10.416051 0.3645931 1.2435136 0.16971159
## 241 0.0068129207 0.05 10.515599 0.3493723 1.2549200 0.16889742
## 261 0.0100000000 0.05 10.601171 0.3362377 1.2656547 0.16796496
## 281 0.0146779927 0.05 10.664991 0.3263381 1.2743795 0.16703976
## 301 0.0215443469 0.05 10.711031 0.3190314 1.2821306 0.16627135
## 321 0.0316227766 0.05 10.746319 0.3131470 1.2898971 0.16557529
## 341 0.0464158883 0.05 10.781756 0.3071099 1.2997144 0.16485457
## 361 0.0681292069 0.05 10.823569 0.3001135 1.3135571 0.16401715
## 381 0.1000000000 0.05 10.863998 0.2920999 1.3184908 0.16255690
## 2 0.0000000000 0.10 2.459091 0.9642642 0.9925689 0.02576505
## 22 0.0001000000 0.10 6.623385 0.8018433 0.8900187 0.07842942
## 42 0.0001467799 0.10 7.229326 0.7598549 0.9606063 0.09422312
## 62 0.0002154435 0.10 7.660288 0.7231678 0.9992116 0.10756822
## 82 0.0003162278 0.10 7.963667 0.6937826 1.0221124 0.11772091
## 102 0.0004641589 0.10 8.192710 0.6696537 1.0374613 0.12547099
## 122 0.0006812921 0.10 8.393496 0.6472086 1.0520670 0.13202036
## 142 0.0010000000 0.10 8.596169 0.6231725 1.0676388 0.13849235
## 162 0.0014677993 0.10 8.810355 0.5963488 1.0866569 0.14521178
## 182 0.0021544347 0.10 9.031409 0.5672939 1.1070508 0.15166268
## 202 0.0031622777 0.10 9.256137 0.5363684 1.1290304 0.15764461
## 222 0.0046415888 0.10 9.476741 0.5048064 1.1520416 0.16269619
## 242 0.0068129207 0.10 9.671164 0.4761896 1.1728641 0.16615716
## 262 0.0100000000 0.10 9.836279 0.4512604 1.1917050 0.16836206
## 282 0.0146779927 0.10 9.962381 0.4317064 1.2065059 0.16942502
## 302 0.0215443469 0.10 10.052990 0.4171651 1.2187059 0.16990420
## 322 0.0316227766 0.10 10.118613 0.4059841 1.2299111 0.17005133
## 342 0.0464158883 0.10 10.189939 0.3934630 1.2453389 0.17024612
## 362 0.0681292069 0.10 10.272881 0.3787739 1.2638244 0.17004554
## 382 0.1000000000 0.10 10.367878 0.3622607 1.2924478 0.16985524
## 3 0.0000000000 0.15 2.616453 0.9570857 1.2675348 0.03998687
## 23 0.0001000000 0.15 4.932830 0.8892714 0.6620146 0.04171295
## 43 0.0001467799 0.15 5.640538 0.8570454 0.7684666 0.05709820
## 63 0.0002154435 0.15 6.162813 0.8275504 0.8382093 0.07017247
## 83 0.0003162278 0.15 6.516989 0.8069232 0.8825771 0.07747369
## 103 0.0004641589 0.15 6.800112 0.7895103 0.9129609 0.08427088
## 123 0.0006812921 0.15 7.062797 0.7716196 0.9330177 0.09088666
## 143 0.0010000000 0.15 7.334851 0.7508759 0.9530434 0.09824922
## 163 0.0014677993 0.15 7.628429 0.7257961 0.9781005 0.10701966
## 183 0.0021544347 0.15 7.938645 0.6960916 1.0057823 0.11690592
## 203 0.0031622777 0.15 8.260786 0.6616639 1.0362799 0.12763863
## 223 0.0046415888 0.15 8.575746 0.6247512 1.0668668 0.13802117
## 243 0.0068129207 0.15 8.857827 0.5890391 1.0952461 0.14682355
## 263 0.0100000000 0.15 9.098460 0.5565457 1.1210379 0.15373456
## 283 0.0146779927 0.15 9.279628 0.5307415 1.1411422 0.15830064
## 303 0.0215443469 0.15 9.410230 0.5111188 1.1573768 0.16133610
## 323 0.0316227766 0.15 9.513941 0.4943592 1.1742341 0.16366454
## 343 0.0464158883 0.15 9.619318 0.4764513 1.1926475 0.16576078
## 363 0.0681292069 0.15 9.736598 0.4559701 1.2176217 0.16786943
## 383 0.1000000000 0.15 9.881629 0.4307268 1.2578762 0.17041639
## 4 0.0000000000 0.20 2.772207 0.9491550 1.5304220 0.05748840
## 24 0.0001000000 0.20 3.794368 0.9276611 0.4928213 0.02538250
## 44 0.0001467799 0.20 4.392058 0.9087546 0.5798343 0.03322510
## 64 0.0002154435 0.20 4.932822 0.8884278 0.6576434 0.04251843
## 84 0.0003162278 0.20 5.339173 0.8706625 0.7161263 0.05079496
## 104 0.0004641589 0.20 5.662226 0.8548161 0.7605929 0.05837654
## 124 0.0006812921 0.20 5.956723 0.8388123 0.8001752 0.06577927
## 144 0.0010000000 0.20 6.257961 0.8214075 0.8367182 0.07248646
## 164 0.0014677993 0.20 6.579314 0.8029711 0.8791261 0.07894974
## 184 0.0021544347 0.20 6.946805 0.7793190 0.9165655 0.08781269
## 204 0.0031622777 0.20 7.338035 0.7498799 0.9517995 0.09845795
## 224 0.0046415888 0.20 7.728816 0.7155472 0.9876312 0.11025521
## 244 0.0068129207 0.20 8.081643 0.6801355 1.0218389 0.12167353
## 264 0.0100000000 0.20 8.386475 0.6459271 1.0536205 0.13178822
## 284 0.0146779927 0.20 8.617681 0.6175837 1.0783225 0.13928353
## 304 0.0215443469 0.20 8.796887 0.5936306 1.1008308 0.14525208
## 324 0.0316227766 0.20 8.931003 0.5740334 1.1196006 0.14961464
## 344 0.0464158883 0.20 9.063303 0.5532972 1.1414346 0.15394202
## 364 0.0681292069 0.20 9.223102 0.5271722 1.1761548 0.15933506
## 384 0.1000000000 0.20 9.416203 0.4947894 1.2237730 0.16542140
## 5 0.0000000000 0.25 2.832587 0.9463020 1.5927066 0.06368109
## 25 0.0001000000 0.25 3.390530 0.9413877 0.4872714 0.02128739
## 45 0.0001467799 0.25 3.654677 0.9320660 0.4749564 0.02393407
## 65 0.0002154435 0.25 4.020081 0.9204718 0.5157841 0.02863808
## 85 0.0003162278 0.25 4.380807 0.9085482 0.5671661 0.03382434
## 105 0.0004641589 0.25 4.701906 0.8969011 0.6123155 0.03889077
## 125 0.0006812921 0.25 5.009720 0.8846712 0.6574578 0.04425313
## 145 0.0010000000 0.25 5.339474 0.8703367 0.7045917 0.05072822
## 165 0.0014677993 0.25 5.709623 0.8522661 0.7591849 0.05911470
## 185 0.0021544347 0.25 6.111296 0.8297460 0.8152785 0.06909105
## 205 0.0031622777 0.25 6.516928 0.8060235 0.8685679 0.07760138
## 225 0.0046415888 0.25 6.941698 0.7790252 0.9158812 0.08764654
## 245 0.0068129207 0.25 7.348489 0.7483050 0.9533743 0.09859228
## 265 0.0100000000 0.25 7.707985 0.7164762 0.9899560 0.10946170
## 285 0.0146779927 0.25 7.995209 0.6873857 1.0209003 0.11881884
## 305 0.0215443469 0.25 8.209347 0.6630540 1.0451487 0.12622751
## 325 0.0316227766 0.25 8.367266 0.6430178 1.0665016 0.13198287
## 345 0.0464158883 0.25 8.532068 0.6202585 1.0917145 0.13826867
## 365 0.0681292069 0.25 8.732541 0.5903587 1.1364334 0.14683738
## 385 0.1000000000 0.25 8.970204 0.5530511 1.1895567 0.15645215
## 6 0.0000000000 0.30 2.825663 0.9471176 1.5208044 0.05946166
## 26 0.0001000000 0.30 3.228322 0.9485836 0.5254938 0.01978043
## 46 0.0001467799 0.30 3.381651 0.9418634 0.4934683 0.02124096
## 66 0.0002154435 0.30 3.556592 0.9353232 0.4731738 0.02314666
## 86 0.0003162278 0.30 3.772601 0.9280794 0.4776704 0.02563791
## 106 0.0004641589 0.30 4.010104 0.9203470 0.5015194 0.02871142
## 126 0.0006812921 0.30 4.267499 0.9118690 0.5407413 0.03233125
## 146 0.0010000000 0.30 4.573633 0.9012251 0.5865842 0.03679066
## 166 0.0014677993 0.30 4.937294 0.8874028 0.6429515 0.04281419
## 186 0.0021544347 0.30 5.346314 0.8699936 0.7039146 0.05057718
## 206 0.0031622777 0.30 5.797891 0.8476034 0.7702454 0.06080367
## 226 0.0046415888 0.30 6.267863 0.8202459 0.8340621 0.07231963
## 246 0.0068129207 0.30 6.686205 0.7952820 0.8879771 0.08103998
## 266 0.0100000000 0.30 7.084709 0.7679595 0.9319870 0.09110684
## 286 0.0146779927 0.30 7.407937 0.7416382 0.9648254 0.10025299
## 306 0.0215443469 0.30 7.648122 0.7191741 0.9915138 0.10788095
## 326 0.0316227766 0.30 7.834694 0.6993439 1.0149642 0.11432840
## 346 0.0464158883 0.30 8.027870 0.6763019 1.0462894 0.12202365
## 366 0.0681292069 0.30 8.267163 0.6446125 1.0964802 0.13275845
## 386 0.1000000000 0.30 8.540194 0.6051484 1.1569560 0.14514753
## 7 0.0000000000 0.35 2.796022 0.9486409 1.4555241 0.05496523
## 27 0.0001000000 0.35 3.130739 0.9525562 0.5511051 0.01832928
## 47 0.0001467799 0.35 3.243229 0.9479644 0.5225125 0.01999597
## 67 0.0002154435 0.35 3.364661 0.9426179 0.4973034 0.02115932
## 87 0.0003162278 0.35 3.481054 0.9380139 0.4800578 0.02238028
## 107 0.0004641589 0.35 3.607498 0.9333822 0.4703599 0.02399885
## 127 0.0006812921 0.35 3.769017 0.9278872 0.4750442 0.02596959
## 147 0.0010000000 0.35 3.995175 0.9204891 0.4967328 0.02868088
## 167 0.0014677993 0.35 4.307845 0.9102080 0.5442216 0.03287325
## 187 0.0021544347 0.35 4.697794 0.8964884 0.6052810 0.03852307
## 207 0.0031622777 0.35 5.152169 0.8784677 0.6761527 0.04645603
## 227 0.0046415888 0.35 5.642162 0.8556486 0.7468244 0.05656832
## 247 0.0068129207 0.35 6.115902 0.8292531 0.8136490 0.06814423
## 267 0.0100000000 0.35 6.531900 0.8037467 0.8685332 0.07756420
## 287 0.0146779927 0.35 6.862811 0.7821076 0.9113025 0.08527743
## 307 0.0215443469 0.35 7.128010 0.7621170 0.9400729 0.09235874
## 327 0.0316227766 0.35 7.331089 0.7442621 0.9667124 0.09868698
## 347 0.0464158883 0.35 7.557614 0.7213937 1.0011746 0.10699675
## 367 0.0681292069 0.35 7.824551 0.6903224 1.0572014 0.11881852
## 387 0.1000000000 0.35 8.131794 0.6502162 1.1251784 0.13298400
## 8 0.0000000000 0.40 2.782475 0.9499429 1.3867005 0.05019967
## 28 0.0001000000 0.40 3.077479 0.9544435 0.5701236 0.01699226
## 48 0.0001467799 0.40 3.152932 0.9516991 0.5454879 0.01877942
## 68 0.0002154435 0.40 3.242092 0.9479925 0.5212550 0.02008068
## 88 0.0003162278 0.40 3.337505 0.9438251 0.4996035 0.02103342
## 108 0.0004641589 0.40 3.426916 0.9399853 0.4837913 0.02184015
## 128 0.0006812921 0.40 3.528031 0.9360384 0.4732260 0.02303383
## 148 0.0010000000 0.40 3.667076 0.9309982 0.4671924 0.02476326
## 168 0.0014677993 0.40 3.884119 0.9237384 0.4808280 0.02728824
## 188 0.0021544347 0.40 4.199346 0.9134550 0.5247950 0.03130967
## 208 0.0031622777 0.40 4.612162 0.8992883 0.5911639 0.03704401
## 228 0.0046415888 0.40 5.091736 0.8808390 0.6646999 0.04494505
## 248 0.0068129207 0.40 5.576171 0.8587868 0.7351205 0.05469155
## 268 0.0100000000 0.40 6.032206 0.8339152 0.8006124 0.06540593
## 288 0.0146779927 0.40 6.395831 0.8110200 0.8465066 0.07416906
## 308 0.0215443469 0.40 6.657203 0.7940486 0.8835854 0.08009162
## 328 0.0316227766 0.40 6.880354 0.7775704 0.9183475 0.08619742
## 348 0.0464158883 0.40 7.116598 0.7574205 0.9573850 0.09403136
## 368 0.0681292069 0.40 7.409755 0.7277889 1.0183672 0.10612224
## 388 0.1000000000 0.40 7.751734 0.6877628 1.0940921 0.12130686
## 9 0.0000000000 0.45 2.744403 0.9517053 1.3137449 0.04649432
## 29 0.0001000000 0.45 3.042430 0.9555976 0.5849291 0.01627444
## 49 0.0001467799 0.45 3.100738 0.9536328 0.5660725 0.01765033
## 69 0.0002154435 0.45 3.166798 0.9511715 0.5437212 0.01912164
## 89 0.0003162278 0.45 3.235720 0.9482644 0.5231386 0.02009345
## 109 0.0004641589 0.45 3.311637 0.9449820 0.5038245 0.02083375
## 129 0.0006812921 0.45 3.393758 0.9413352 0.4884475 0.02165089
## 149 0.0010000000 0.45 3.494985 0.9371472 0.4755914 0.02269281
## 169 0.0014677993 0.45 3.634545 0.9318625 0.4681882 0.02444812
## 189 0.0021544347 0.45 3.851870 0.9244104 0.4773124 0.02708633
## 209 0.0031622777 0.45 4.187500 0.9134477 0.5223781 0.03112248
## 229 0.0046415888 0.45 4.623574 0.8985993 0.5912723 0.03698966
## 249 0.0068129207 0.45 5.096723 0.8803733 0.6638462 0.04472703
## 269 0.0100000000 0.45 5.558360 0.8593149 0.7307366 0.05383080
## 289 0.0146779927 0.45 5.938377 0.8388543 0.7820619 0.06230877
## 309 0.0215443469 0.45 6.244489 0.8198113 0.8260765 0.07000569
## 329 0.0316227766 0.45 6.486507 0.8029909 0.8597337 0.07623853
## 349 0.0464158883 0.45 6.732986 0.7841940 0.9054345 0.08368699
## 369 0.0681292069 0.45 7.049806 0.7557366 0.9740699 0.09571370
## 389 0.1000000000 0.45 7.408692 0.7178541 1.0576771 0.11099172
## 10 0.0000000000 0.50 2.720564 0.9527919 1.2651366 0.04392192
## 30 0.0001000000 0.50 3.011166 0.9565326 0.5917371 0.01558625
## 50 0.0001467799 0.50 3.066289 0.9548115 0.5799428 0.01686997
## 70 0.0002154435 0.50 3.116275 0.9530681 0.5649744 0.01818617
## 90 0.0003162278 0.50 3.173564 0.9508988 0.5435299 0.01922105
## 110 0.0004641589 0.50 3.232472 0.9484256 0.5233591 0.01999734
## 130 0.0006812921 0.50 3.301821 0.9454164 0.5057422 0.02080713
## 150 0.0010000000 0.50 3.389851 0.9415369 0.4884950 0.02168530
## 170 0.0014677993 0.50 3.501801 0.9367162 0.4760573 0.02289672
## 190 0.0021544347 0.50 3.655620 0.9307867 0.4672221 0.02474521
## 210 0.0031622777 0.50 3.890761 0.9227093 0.4814796 0.02769918
## 230 0.0046415888 0.50 4.246415 0.9110930 0.5294484 0.03185108
## 250 0.0068129207 0.50 4.677222 0.8962967 0.5980080 0.03767883
## 270 0.0100000000 0.50 5.127345 0.8787511 0.6659339 0.04491731
## 290 0.0146779927 0.50 5.526475 0.8603283 0.7207043 0.05251040
## 310 0.0215443469 0.50 5.844409 0.8433115 0.7647083 0.05951140
## 330 0.0316227766 0.50 6.103453 0.8274872 0.8044930 0.06597694
## 350 0.0464158883 0.50 6.389022 0.8071570 0.8591526 0.07451781
## 370 0.0681292069 0.50 6.736051 0.7784512 0.9287985 0.08673025
## 390 0.1000000000 0.50 7.114761 0.7417272 1.0188507 0.10209565
## 11 0.0000000000 0.55 2.709506 0.9537714 1.2005293 0.04016340
## 31 0.0001000000 0.55 2.988034 0.9571581 0.5955181 0.01502596
## 51 0.0001467799 0.55 3.035507 0.9557774 0.5872400 0.01621679
## 71 0.0002154435 0.55 3.083084 0.9542136 0.5762470 0.01738760
## 91 0.0003162278 0.55 3.125667 0.9526970 0.5618168 0.01845982
## 111 0.0004641589 0.55 3.177247 0.9507153 0.5425637 0.01935109
## 131 0.0006812921 0.55 3.235978 0.9482825 0.5229588 0.02013817
## 151 0.0010000000 0.55 3.311767 0.9450193 0.5036901 0.02098497
## 171 0.0014677993 0.55 3.411531 0.9406192 0.4877421 0.02208374
## 191 0.0021544347 0.55 3.537148 0.9351231 0.4757600 0.02350799
## 211 0.0031622777 0.55 3.716996 0.9282202 0.4705456 0.02570856
## 231 0.0046415888 0.55 3.988393 0.9189861 0.4920423 0.02898608
## 251 0.0068129207 0.55 4.352140 0.9070576 0.5452673 0.03323586
## 271 0.0100000000 0.55 4.770725 0.8923758 0.6092227 0.03887331
## 291 0.0146779927 0.55 5.156037 0.8768232 0.6627459 0.04506112
## 311 0.0215443469 0.55 5.481154 0.8616167 0.7100864 0.05130247
## 331 0.0316227766 0.55 5.754676 0.8467314 0.7543591 0.05746169
## 351 0.0464158883 0.55 6.073913 0.8264442 0.8141522 0.06625359
## 371 0.0681292069 0.55 6.431774 0.7998560 0.8927403 0.07833474
## 391 0.1000000000 0.55 6.827418 0.7646294 0.9893449 0.09409844
## 12 0.0000000000 0.60 2.727836 0.9535778 1.1633644 0.03805586
## 32 0.0001000000 0.60 2.972875 0.9575437 0.5984875 0.01462419
## 52 0.0001467799 0.60 3.010931 0.9565011 0.5907797 0.01563567
## 72 0.0002154435 0.60 3.055105 0.9551325 0.5835386 0.01676547
## 92 0.0003162278 0.60 3.094945 0.9537906 0.5731894 0.01770588
## 112 0.0004641589 0.60 3.132914 0.9523912 0.5597277 0.01864126
## 132 0.0006812921 0.60 3.184854 0.9503711 0.5412637 0.01955223
## 152 0.0010000000 0.60 3.253346 0.9475533 0.5201730 0.02041630
## 172 0.0014677993 0.60 3.344872 0.9436220 0.4997397 0.02141577
## 192 0.0021544347 0.60 3.462546 0.9383896 0.4840636 0.02277177
## 212 0.0031622777 0.60 3.609359 0.9320347 0.4755353 0.02455505
## 232 0.0046415888 0.60 3.815094 0.9243149 0.4773764 0.02715856
## 252 0.0068129207 0.60 4.108082 0.9144039 0.5087864 0.03062861
## 272 0.0100000000 0.60 4.473658 0.9022903 0.5640088 0.03503744
## 292 0.0146779927 0.60 4.839184 0.8889014 0.6160924 0.04010987
## 312 0.0215443469 0.60 5.160314 0.8753374 0.6622640 0.04538076
## 332 0.0316227766 0.60 5.460081 0.8606145 0.7117870 0.05133093
## 352 0.0464158883 0.60 5.782104 0.8423086 0.7771649 0.05968331
## 372 0.0681292069 0.60 6.146473 0.8177940 0.8628644 0.07137966
## 392 0.1000000000 0.60 6.564059 0.7837430 0.9637449 0.08728962
## 13 0.0000000000 0.65 2.750376 0.9531250 1.1477768 0.03718889
## 33 0.0001000000 0.65 2.960818 0.9578558 0.6018585 0.01434482
## 53 0.0001467799 0.65 2.993555 0.9569831 0.5940646 0.01515896
## 73 0.0002154435 0.65 3.031455 0.9558660 0.5870019 0.01618405
## 93 0.0003162278 0.65 3.069254 0.9546370 0.5797862 0.01712746
## 113 0.0004641589 0.65 3.106122 0.9533517 0.5700413 0.01799234
## 133 0.0006812921 0.65 3.145669 0.9518525 0.5561941 0.01890894
## 153 0.0010000000 0.65 3.205723 0.9494638 0.5354664 0.01988336
## 173 0.0014677993 0.65 3.290132 0.9459569 0.5137462 0.02091966
## 193 0.0021544347 0.65 3.398929 0.9412126 0.4940304 0.02216090
## 213 0.0031622777 0.65 3.537622 0.9349855 0.4830173 0.02390509
## 233 0.0046415888 0.65 3.710440 0.9277781 0.4785431 0.02610540
## 253 0.0068129207 0.65 3.941856 0.9192926 0.4910390 0.02897081
## 273 0.0100000000 0.65 4.247067 0.9090115 0.5302676 0.03256470
## 293 0.0146779927 0.65 4.580800 0.8973945 0.5768473 0.03674213
## 313 0.0215443469 0.65 4.902416 0.8847231 0.6261084 0.04150636
## 333 0.0316227766 0.65 5.195940 0.8715467 0.6777380 0.04700610
## 353 0.0464158883 0.65 5.514060 0.8552044 0.7498895 0.05471208
## 373 0.0681292069 0.65 5.889400 0.8321321 0.8386014 0.06600581
## 393 0.1000000000 0.65 6.320521 0.7998099 0.9423947 0.08161296
## 14 0.0000000000 0.70 2.767642 0.9529309 1.1298464 0.03625774
## 34 0.0001000000 0.70 2.949543 0.9581454 0.6041994 0.01412413
## 54 0.0001467799 0.70 2.980706 0.9573204 0.5975476 0.01480959
## 74 0.0002154435 0.70 3.013120 0.9563943 0.5902352 0.01569740
## 94 0.0003162278 0.70 3.047634 0.9553160 0.5833670 0.01660240
## 114 0.0004641589 0.70 3.082129 0.9541537 0.5757153 0.01747180
## 134 0.0006812921 0.70 3.120017 0.9527838 0.5653583 0.01836412
## 154 0.0010000000 0.70 3.167836 0.9509203 0.5490297 0.01934291
## 174 0.0014677993 0.70 3.241737 0.9479155 0.5271881 0.02043492
## 194 0.0021544347 0.70 3.344115 0.9435579 0.5054435 0.02164968
## 214 0.0031622777 0.70 3.475232 0.9377398 0.4907093 0.02326511
## 234 0.0046415888 0.70 3.633335 0.9306014 0.4873462 0.02546164
## 254 0.0068129207 0.70 3.831165 0.9227018 0.4906658 0.02808056
## 274 0.0100000000 0.70 4.090340 0.9134267 0.5134927 0.03120457
## 294 0.0146779927 0.70 4.385459 0.9031107 0.5528143 0.03484581
## 314 0.0215443469 0.70 4.681580 0.8919627 0.6005981 0.03910964
## 334 0.0316227766 0.70 4.962509 0.8802453 0.6567143 0.04408993
## 354 0.0464158883 0.70 5.280079 0.8651078 0.7305454 0.05131768
## 374 0.0681292069 0.70 5.659480 0.8435987 0.8201077 0.06200611
## 394 0.1000000000 0.70 6.095456 0.8132758 0.9258113 0.07703840
## 15 0.0000000000 0.75 2.785717 0.9527284 1.1142329 0.03543867
## 35 0.0001000000 0.75 2.938620 0.9584160 0.6052777 0.01392362
## 55 0.0001467799 0.75 2.969988 0.9575949 0.6003016 0.01454281
## 75 0.0002154435 0.75 2.998467 0.9568053 0.5938272 0.01531287
## 95 0.0003162278 0.75 3.029491 0.9558661 0.5867711 0.01614901
## 115 0.0004641589 0.75 3.061367 0.9548232 0.5792945 0.01698939
## 135 0.0006812921 0.75 3.097837 0.9535463 0.5701643 0.01788501
## 155 0.0010000000 0.75 3.143073 0.9518541 0.5580204 0.01887031
## 175 0.0014677993 0.75 3.204371 0.9494001 0.5397671 0.01996654
## 195 0.0021544347 0.75 3.296523 0.9455400 0.5173128 0.02119111
## 215 0.0031622777 0.75 3.420909 0.9400983 0.4991623 0.02270329
## 235 0.0046415888 0.75 3.573917 0.9331672 0.4921203 0.02479789
## 255 0.0068129207 0.75 3.747207 0.9255378 0.4973844 0.02733546
## 275 0.0100000000 0.75 3.967650 0.9170781 0.5122069 0.03034431
## 295 0.0146779927 0.75 4.225128 0.9077819 0.5436887 0.03369099
## 315 0.0215443469 0.75 4.495799 0.8976561 0.5895369 0.03764346
## 335 0.0316227766 0.75 4.766944 0.8866932 0.6456138 0.04235615
## 355 0.0464158883 0.75 5.079349 0.8726084 0.7189312 0.04914689
## 375 0.0681292069 0.75 5.452552 0.8528049 0.8084624 0.05917736
## 395 0.1000000000 0.75 5.893334 0.8242072 0.9143073 0.07357942
## 16 0.0000000000 0.80 2.809063 0.9523498 1.0995174 0.03472032
## 36 0.0001000000 0.80 2.929297 0.9586293 0.6054008 0.01376020
## 56 0.0001467799 0.80 2.960224 0.9578431 0.6024406 0.01433472
## 76 0.0002154435 0.80 2.986692 0.9571201 0.5967381 0.01500302
## 96 0.0003162278 0.80 3.014418 0.9563089 0.5899431 0.01576999
## 116 0.0004641589 0.80 3.044520 0.9553550 0.5826451 0.01657506
## 136 0.0006812921 0.80 3.078876 0.9541800 0.5737701 0.01745670
## 156 0.0010000000 0.80 3.121906 0.9526060 0.5621626 0.01843884
## 176 0.0014677993 0.80 3.179883 0.9503506 0.5472261 0.01953535
## 196 0.0021544347 0.80 3.259929 0.9470305 0.5278560 0.02077058
## 216 0.0031622777 0.80 3.375753 0.9420200 0.5076651 0.02224273
## 236 0.0046415888 0.80 3.522102 0.9354331 0.4973181 0.02421307
## 256 0.0068129207 0.80 3.689349 0.9277618 0.5018694 0.02683057
## 276 0.0100000000 0.80 3.880645 0.9198270 0.5158936 0.02967321
## 296 0.0146779927 0.80 4.104527 0.9112458 0.5428048 0.03293914
## 316 0.0215443469 0.80 4.352841 0.9017487 0.5864785 0.03679268
## 336 0.0316227766 0.80 4.608802 0.8914400 0.6422551 0.04140382
## 356 0.0464158883 0.80 4.906888 0.8784046 0.7148065 0.04786527
## 376 0.0681292069 0.80 5.275345 0.8598411 0.8029228 0.05737812
## 396 0.1000000000 0.80 5.718678 0.8327427 0.9082369 0.07116279
## 17 0.0000000000 0.85 2.850435 0.9513116 1.0893981 0.03453144
## 37 0.0001000000 0.85 2.921989 0.9587984 0.6059551 0.01364299
## 57 0.0001467799 0.85 2.951443 0.9580634 0.6041183 0.01416166
## 77 0.0002154435 0.85 2.977567 0.9573578 0.5992988 0.01476101
## 97 0.0003162278 0.85 3.002619 0.9566434 0.5931229 0.01544988
## 117 0.0004641589 0.85 3.030855 0.9557708 0.5858513 0.01621920
## 137 0.0006812921 0.85 3.063099 0.9546918 0.5771165 0.01707921
## 157 0.0010000000 0.85 3.103708 0.9532341 0.5657113 0.01805904
## 177 0.0014677993 0.85 3.159147 0.9511111 0.5512643 0.01916056
## 197 0.0021544347 0.85 3.234375 0.9480460 0.5339465 0.02038810
## 217 0.0031622777 0.85 3.336662 0.9436496 0.5162841 0.02185041
## 237 0.0046415888 0.85 3.476260 0.9374252 0.5030467 0.02371095
## 257 0.0068129207 0.85 3.638924 0.9299451 0.5051349 0.02624263
## 277 0.0100000000 0.85 3.818413 0.9218301 0.5217764 0.02926180
## 297 0.0146779927 0.85 4.018705 0.9137710 0.5465974 0.03246279
## 317 0.0215443469 0.85 4.241514 0.9048501 0.5886169 0.03631807
## 337 0.0316227766 0.85 4.480622 0.8950062 0.6449615 0.04093237
## 357 0.0464158883 0.85 4.768537 0.8825731 0.7165814 0.04723661
## 377 0.0681292069 0.85 5.129005 0.8650443 0.8035460 0.05636825
## 397 0.1000000000 0.85 5.569345 0.8393740 0.9066101 0.06952906
## 18 0.0000000000 0.90 2.893456 0.9502389 1.0774021 0.03431220
## 38 0.0001000000 0.90 2.915726 0.9589511 0.6073822 0.01356600
## 58 0.0001467799 0.90 2.943796 0.9582543 0.6052784 0.01400763
## 78 0.0002154435 0.90 2.969815 0.9575606 0.6016949 0.01456511
## 98 0.0003162278 0.90 2.993944 0.9568745 0.5959446 0.01518656
## 118 0.0004641589 0.90 3.019959 0.9560898 0.5889883 0.01591452
## 138 0.0006812921 0.90 3.050261 0.9550959 0.5802221 0.01675755
## 158 0.0010000000 0.90 3.088551 0.9537431 0.5689131 0.01772267
## 178 0.0014677993 0.90 3.141009 0.9517596 0.5546243 0.01882833
## 198 0.0021544347 0.90 3.212335 0.9488850 0.5375635 0.02004611
## 218 0.0031622777 0.90 3.309983 0.9447486 0.5208557 0.02147377
## 238 0.0046415888 0.90 3.436848 0.9391178 0.5094325 0.02330465
## 258 0.0068129207 0.90 3.592901 0.9319617 0.5088174 0.02570982
## 278 0.0100000000 0.90 3.768008 0.9238357 0.5248630 0.02881032
## 298 0.0146779927 0.90 3.951678 0.9157662 0.5528715 0.03212700
## 318 0.0215443469 0.90 4.158416 0.9071809 0.5937369 0.03602104
## 338 0.0316227766 0.90 4.386297 0.8974760 0.6506826 0.04077054
## 358 0.0464158883 0.90 4.661775 0.8854736 0.7230983 0.04705153
## 378 0.0681292069 0.90 5.009706 0.8688644 0.8084306 0.05588854
## 398 0.1000000000 0.90 5.439393 0.8446538 0.9084765 0.06845655
## 19 0.0000000000 0.95 2.938527 0.9490345 1.0734114 0.03432539
## 39 0.0001000000 0.95 2.910081 0.9590875 0.6087226 0.01350243
## 59 0.0001467799 0.95 2.937593 0.9584062 0.6065261 0.01388842
## 79 0.0002154435 0.95 2.963538 0.9577215 0.6036758 0.01440372
## 99 0.0003162278 0.95 2.987560 0.9570439 0.5986309 0.01498489
## 119 0.0004641589 0.95 3.011934 0.9563135 0.5918332 0.01566999
## 139 0.0006812921 0.95 3.040092 0.9554051 0.5830756 0.01648193
## 159 0.0010000000 0.95 3.075908 0.9541568 0.5718643 0.01743163
## 179 0.0014677993 0.95 3.125062 0.9523162 0.5576787 0.01852772
## 199 0.0021544347 0.95 3.193652 0.9495859 0.5407567 0.01975298
## 219 0.0031622777 0.95 3.288369 0.9456134 0.5237989 0.02116980
## 239 0.0046415888 0.95 3.410371 0.9402473 0.5122048 0.02293790
## 259 0.0068129207 0.95 3.557231 0.9335277 0.5125499 0.02530089
## 279 0.0100000000 0.95 3.728671 0.9255113 0.5272058 0.02837538
## 299 0.0146779927 0.95 3.908072 0.9171699 0.5571468 0.03194878
## 319 0.0215443469 0.95 4.099747 0.9087420 0.6003344 0.03590509
## 339 0.0316227766 0.95 4.317658 0.8991786 0.6586562 0.04078887
## 359 0.0464158883 0.95 4.579106 0.8875814 0.7315437 0.04709127
## 379 0.0681292069 0.95 4.914994 0.8716077 0.8156550 0.05574779
## 399 0.1000000000 0.95 5.332883 0.8486147 0.9125403 0.06779616
## 20 0.0000000000 1.00 2.991078 0.9474166 1.0789961 0.03489684
## 40 0.0001000000 1.00 2.905240 0.9592033 0.6094522 0.01345035
## 60 0.0001467799 1.00 2.932911 0.9585235 0.6080986 0.01380785
## 80 0.0002154435 1.00 2.958637 0.9578454 0.6054093 0.01427516
## 100 0.0003162278 1.00 2.982641 0.9571733 0.6009291 0.01482961
## 120 0.0004641589 1.00 3.006357 0.9564671 0.5944285 0.01548124
## 140 0.0006812921 1.00 3.032628 0.9556268 0.5857474 0.01625936
## 160 0.0010000000 1.00 3.065882 0.9544779 0.5745734 0.01718941
## 180 0.0014677993 1.00 3.112136 0.9527628 0.5604631 0.01827370
## 200 0.0021544347 1.00 3.178268 0.9501574 0.5436133 0.01950266
## 220 0.0031622777 1.00 3.270113 0.9463388 0.5264208 0.02091588
## 240 0.0046415888 1.00 3.389907 0.9411075 0.5142815 0.02266796
## 260 0.0068129207 1.00 3.534729 0.9345113 0.5136816 0.02498591
## 280 0.0100000000 1.00 3.697734 0.9268632 0.5287137 0.02799999
## 300 0.0146779927 1.00 3.872542 0.9185757 0.5598961 0.03166089
## 320 0.0215443469 1.00 4.059057 0.9098471 0.6062003 0.03590489
## 340 0.0316227766 1.00 4.267700 0.9003366 0.6666900 0.04091320
## 360 0.0464158883 1.00 4.519198 0.8889338 0.7399037 0.04724299
## 380 0.0681292069 1.00 4.838141 0.8736653 0.8239112 0.05575853
## 400 0.1000000000 1.00 5.241996 0.8517626 0.9185722 0.06741659Interesting to note is that the regularization path of two model, lasso, and elastic net is really same. Actually, it is no supprise, because, as I will show, the elastic net chooses the most efficient parameter as lambda 0, meaning Lasso.
par(mfrow = c(1,3))
plot.enet(ridge.fit$finalModel,xvar = "fraction", use.color = T)
plot.enet(lasso.fit$finalModel,xvar = "fraction", use.color = T)
plot.enet(enet.fit$finalModel,xvar = "fraction", use.color = T)
The chart indicates each best tune the model choose. The best tune for ridge, lambda, is 0.0004. One could infer that the ridge takes the small panelty. And the fraction value of Lasso is 0.1. Fraction means the ratio of distance value between max Beta and the regularized beta. Lastly, the parameters of enet shows the lambda is 0, and the fraction is 0.05. One can infer that although the regularization path for enet is almost same as the Lasso, The performance could be differ because of the difference in fraction size.
mat <- matrix(NA, nrow = 2, ncol = 3)
mat <- data.frame(mat, row.names = c("lambda", "fraction"))
colnames(mat) <- c("Ridge", "Lasso", "Enet")
mat[1,1] <- ridge.fit$bestTune[1]
mat[2,2] <- lasso.fit$bestTune[1]
mat[1,3] <- enet.fit$bestTune[2]
mat[2,3] <- enet.fit$bestTune[1]
kable(mat)| Ridge | Lasso | Enet | |
|---|---|---|---|
| lambda | 0 | NA | 0.00 |
| fraction | NA | 0.1 | 0.05 |
Here is the RMSE plot on the training set. As we see, the RMSE on enet is the lowest compared to other model - lambda is 0. And the RMSE on the best tune of Lasso and Ridge is almost same.
suppressMessages(library(gridExtra))
g1 <- ggplot(ridge.fit)
g2 <- ggplot(lasso.fit)
g3 <- ggplot(enet.fit)
suppressMessages(grid.arrange(g1,g2,g3, ncol = 3))## Warning: The shape palette can deal with a maximum of 6 discrete values
## because more than 6 becomes difficult to discriminate; you have
## 20. Consider specifying shapes manually if you must have them.
## Warning: The shape palette can deal with a maximum of 6 discrete values
## because more than 6 becomes difficult to discriminate; you have
## 20. Consider specifying shapes manually if you must have them.## Warning: Removed 280 rows containing missing values (geom_point).## Warning: The shape palette can deal with a maximum of 6 discrete values
## because more than 6 becomes difficult to discriminate; you have
## 20. Consider specifying shapes manually if you must have them.
The chart shows the RMSE values on the each model in the training set. Lasso shows the lowest RMSE on the test set, followed by Elastic net, then Ridge. But the RMSE between Lasso and Elastic net shows not much difference, about 0.2.
pred.ridge <- predict(ridge.fit, Test)
pred.lasso <- predict(lasso.fit, Test)
pred.enet <- predict(enet.fit, Test)
Ridge.RMSE <- RMSE(pred.ridge, Test$Y)
Lasso.RMSE <- RMSE(pred.lasso, Test$Y)
Enet.RMSE <- RMSE(pred.enet, Test$Y)
c <- cbind(Ridge.RMSE,Lasso.RMSE,Enet.RMSE); rownames(c) <- "RMSE"
kable(round(c,digits = 3))| Ridge.RMSE | Lasso.RMSE | Enet.RMSE | |
|---|---|---|---|
| RMSE | 0.782 | 1.221 | 1.445 |
Here is the sampled RMSE in the cross validation data. The RMSE comes from each best model.
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | NA’s | |
|---|---|---|---|---|---|---|---|
| Lasso | 1.437 | 1.697 | 2.133 | 2.265 | 2.611 | 4.370 | 0 |
| ElasticNet | 1.405 | 1.812 | 2.202 | 2.344 | 2.662 | 5.054 | 0 |
The table indicates the difference of RMSE in each model is significant. The lower diagonal in the table means the p-value against the null hypothesis that two model’s MSE is different. Given the p-value, one can see that there is no significant difference between Lasso, and Elastic net.
diff <- diff(resample)
sum.diff <- summary(diff); sum.tb <- sum.diff$table
kable(sum.tb$RMSE)| Lasso | ElasticNet | |
|---|---|---|
| Lasso | -0.07887 | |
| ElasticNet | 0.6769 |
Since the model is similar, following the Ocum’s rule, Lasso is preferable. Then the next question is which predictor shows the most influence on the test set in Lasso?
ncol(Test)## [1] 101Test.mat <- model.matrix(Y~.,data = Test)
pred.coef <- predict.enet(lasso.fit$finalModel,newx = Test.mat,s = 0.1,mode = "fraction", type = "coef")
coef <- pred.coef$coefficients; coefn0 <- coef[coef != 0]
dcoef <- data.frame(coefn0, row.names = c(1:length(coefn0)))
colnames(dcoef) <- "coefficients"
dcoef$Var <- names(coefn0)
suppressMessages(library(dplyr))
dcoef <- dcoef %>%
select(Var,coefficients) %>%
arrange(desc(abs(coefficients)))
kable(dcoef[1:10,])| Var | coefficients |
|---|---|
| X45 | 4349.119 |
| X34 | -4255.593 |
| X22 | 3215.517 |
| X11 | 2492.542 |
| X32 | 2403.624 |
| X1 | 2383.820 |
| X2 | -2340.160 |
| X42 | -2261.163 |
| X37 | 2212.958 |
| X46 | -2194.587 |
As you see in the chart, X45, and X34 shows the highest absolute value above 4,000. and followed by X22, X11, and others.