Read carefully before jumping in. Answer all questions and include all R code in the Markdown file. Make sure the Markdown properly knits to html. Convert the html to pdf via ‘print to pdf’ and submit both the pdf AND the markdown file. MAKE SURE YOU INCLUDE YOUR NAME.

Read and submit a signed and dated statement of deletion along with the above files.


1. (64 points).
A new study has been conducted to assess the feasibility of developing a screening test for individuals diagnosed with cancer X and surgically treated for it. The ultimate goal is to be able to offer a targeted preventive treatment to patients at high risk of recurrence, while sparing patients at low risk of recurrence the life-altering side effects of such preventive treatment.
The csv file dataX.csv contains the gene expression profiles of 450 patients as well as their 5-year recurrence indicator (first column; 1=recurred, 0=not recurred). The 600 gene expression probes in the dataset were selected from a much larger set of approximately 30,000 probes based on their variance (highly variable probes are often considered more likely to be relevant for the outcome.)
  1. (40 points) Develop a model to predict cancer recurrence within 5 years of surgical treatment based on the gene expression features. Consider at least two machine learning methods and select the best using an approrpiate approach and performance metric. Keep in mind that in addition to good predictivity a model based on a small subset of gene expression features would be preferable (but not absolutely essential) because it would lower the cost of the screening test. You should only consider methods that allow you to compute and ROC curve as this is required in part b. Explain your choice of models, the selection approach, and the performance metric used. Report the performance of your final model and the subset of expression features it uses (if not all).
## set-up
suppressMessages(library(mlr))

setwd("D:/Google Drive/_PM591 Machine learning/Final")
dataX = read.csv("dataX.csv")
summarizeColumns(dataX)
##      name    type na          mean      disp        median       mad        min
## 1       y integer  0  2.533333e-01 0.4354042  0.0000000000 0.0000000  0.0000000
## 2     x.1 numeric  0 -1.024880e-02 1.0648761  0.0080306220 0.9940388 -3.6941223
## 3     x.2 numeric  0  5.052697e-02 0.9905398  0.0238084620 1.0133202 -2.6705030
## 4     x.3 numeric  0  2.350380e-02 0.9979562 -0.0023812210 0.9660026 -3.0590047
## 5     x.4 numeric  0  4.432098e-02 0.9452583  0.0194043675 0.9776007 -2.7725429
## 6     x.5 numeric  0 -1.548568e-02 0.9727023 -0.0376711965 0.9046820 -2.7395521
## 7     x.6 numeric  0 -1.347107e-02 1.0186489 -0.0171437920 0.9826188 -2.8142795
## 8     x.7 numeric  0  4.311855e-02 0.9750570 -0.0234497235 0.8594349 -2.5507112
## 9     x.8 numeric  0 -1.497521e-04 1.0060952 -0.1164556980 0.8184661 -2.3693022
## 10    x.9 numeric  0  1.043830e-01 1.0813616 -0.0400445290 0.8326303 -2.1181678
## 11   x.10 numeric  0  7.935131e-03 1.0502555 -0.0727995145 0.8793137 -2.5697546
## 12   x.11 numeric  0 -2.177522e-02 0.9979958 -0.1102878905 0.8878141 -3.4635304
## 13   x.12 numeric  0  1.651152e-02 0.9749069 -0.0603437215 0.9185201 -2.6940760
## 14   x.13 numeric  0 -2.415446e-02 1.0063381 -0.1127584085 1.1332638 -2.1027858
## 15   x.14 numeric  0  1.420733e-02 1.0320759 -0.1397368505 0.9760178 -2.1996266
## 16   x.15 numeric  0 -3.363125e-02 0.9615862 -0.2335726985 0.8014710 -1.9740496
## 17   x.16 numeric  0  2.639865e-02 1.0158933  0.0466033455 0.9915577 -3.3647320
## 18   x.17 numeric  0  2.411427e-02 1.0384891 -0.0955501695 0.8042157 -2.5522314
## 19   x.18 numeric  0  1.719077e-02 0.9998939  0.0056943715 0.9263222 -3.0460775
## 20   x.19 numeric  0 -5.082274e-03 1.0145848  0.0337929625 0.9439356 -4.8333820
## 21   x.20 numeric  0 -1.131494e-02 1.0168089 -0.1546255770 0.2578097 -1.0131637
## 22   x.21 numeric  0  4.513051e-02 1.0388232 -0.0357314415 1.0045293 -2.4898120
## 23   x.22 numeric  0 -1.401639e-02 0.9172197 -0.0255090290 0.8449364 -2.6334660
## 24   x.23 numeric  0  1.553456e-02 1.0050497  0.0619054450 1.0017218 -3.2153048
## 25   x.24 numeric  0 -4.089796e-02 0.9833877  0.0329343960 0.8996659 -3.4046351
## 26   x.25 numeric  0 -1.969236e-03 1.0154298  0.0118161770 0.9771813 -2.4992585
## 27   x.26 numeric  0 -1.421607e-02 0.9283525 -0.0583130330 0.9540398 -2.6519225
## 28   x.27 numeric  0 -1.095911e-03 0.9893718 -0.0479195750 0.9827973 -3.3491013
## 29   x.28 numeric  0 -6.890745e-04 1.0191280 -0.0295048025 0.9969333 -2.7573337
## 30   x.29 numeric  0  3.751652e-02 1.0033769  0.2177157460 0.9467713 -2.4580320
## 31   x.30 numeric  0 -2.601649e-02 1.0059088 -0.0875517265 1.0696795 -2.6147975
## 32   x.31 numeric  0  1.559476e-02 1.0097740 -0.0234943490 0.9902280 -2.4554883
## 33   x.32 numeric  0  4.181868e-02 0.9954975 -0.2359313930 0.4773801 -1.1890937
## 34   x.33 numeric  0 -3.876380e-02 0.9366234 -0.1252861025 0.9402043 -2.3314472
## 35   x.34 numeric  0  4.153272e-02 0.9900674 -0.2041362305 0.7136204 -1.8236279
## 36   x.35 numeric  0  2.167965e-02 1.1109897 -0.0963361795 0.7294124 -2.2893843
## 37   x.36 numeric  0 -6.229502e-03 1.0216411 -0.0077028825 0.9373644 -3.3504331
## 38   x.37 numeric  0 -9.649055e-03 0.9861602 -0.1393164845 0.2857929 -0.8861091
## 39   x.38 numeric  0 -6.754005e-03 0.9763182  0.0001477255 0.8377315 -2.5977452
## 40   x.39 numeric  0  2.181091e-02 0.9937656 -0.0188127815 1.0068933 -2.8999885
## 41   x.40 numeric  0 -5.620731e-03 1.0255332 -0.0253561095 0.9150772 -3.1883049
## 42   x.41 numeric  0  3.557856e-02 0.9608617  0.0218481810 0.9365894 -2.5190165
## 43   x.42 numeric  0  4.241266e-03 0.9823503  0.0508449445 0.9323030 -3.3112662
## 44   x.43 numeric  0 -2.989576e-02 0.9663352 -0.0422141400 0.9016852 -3.6133918
## 45   x.44 numeric  0  1.211972e-03 0.9928332 -0.0061126695 0.9714698 -2.7818912
## 46   x.45 numeric  0  2.347782e-02 0.9955027  0.0157025130 0.8658248 -2.7731610
## 47   x.46 numeric  0  3.551436e-02 0.9515248  0.0028221505 0.9104334 -2.0669322
## 48   x.47 numeric  0 -4.849920e-02 0.9308718 -0.3297511660 0.6804646 -1.3002327
## 49   x.48 numeric  0 -1.929923e-03 0.9776165 -0.1331716795 0.9555992 -2.1435041
## 50   x.49 numeric  0  3.341611e-02 0.9675479  0.0109662955 0.9963767 -3.0908419
## 51   x.50 numeric  0  5.362540e-02 1.1849448 -0.0107973015 0.7265635 -3.8064031
## 52   x.51 numeric  0  2.661586e-02 0.9708759  0.0562812165 0.9211074 -3.1528876
## 53   x.52 numeric  0 -5.873102e-02 1.0601466 -0.1573485060 0.9941260 -2.4500235
## 54   x.53 numeric  0  3.474241e-02 0.9822394 -0.0387024420 0.9672311 -2.9819857
## 55   x.54 numeric  0 -4.579851e-02 1.0040064  0.0135693705 0.9532363 -3.1672833
## 56   x.55 numeric  0 -1.526510e-02 0.9666235 -0.1465550745 0.8411231 -2.1269600
## 57   x.56 numeric  0 -2.567351e-02 0.9986462 -0.0617761340 0.9410455 -3.1441066
## 58   x.57 numeric  0 -5.100930e-02 1.0015105 -0.0667054145 1.0441054 -3.0651987
## 59   x.58 numeric  0  2.981709e-03 0.9433116 -0.0656932330 0.8921344 -2.7956432
## 60   x.59 numeric  0 -5.449410e-02 0.9697629 -0.1982475520 0.8017886 -2.1560580
## 61   x.60 numeric  0  6.041801e-03 1.0118706 -0.0158484290 0.9673488 -3.2138300
## 62   x.61 numeric  0 -1.406830e-02 0.8443216 -0.0603202370 0.6188641 -1.9477758
## 63   x.62 numeric  0 -1.992543e-03 0.9601038 -0.0025242180 0.9399450 -2.9099729
## 64   x.63 numeric  0 -5.950805e-02 0.9779494 -0.1298627050 1.0076815 -2.9089472
## 65   x.64 numeric  0 -2.794617e-02 1.0254594 -0.0096923295 0.9664489 -4.3950872
## 66   x.65 numeric  0 -2.779568e-02 0.9874815 -0.0223493130 0.9735749 -2.9746592
## 67   x.66 numeric  0 -4.500588e-03 1.0431966 -0.0472852950 0.9583674 -2.1007425
## 68   x.67 numeric  0  3.568166e-02 0.9500414  0.0351375790 0.9041319 -2.7206227
## 69   x.68 numeric  0 -6.414825e-02 1.0336442 -0.0787266330 1.0056179 -3.6819615
## 70   x.69 numeric  0 -3.789149e-02 0.9932003 -0.0618838225 0.9454007 -2.9766829
## 71   x.70 numeric  0 -5.398067e-03 0.9874879  0.0577356410 0.9483958 -3.6939668
## 72   x.71 numeric  0 -8.048867e-03 0.9671759 -0.0841179395 0.9579281 -2.5622681
## 73   x.72 numeric  0 -4.344213e-04 1.0454658 -0.0262616965 0.9820078 -3.1796921
## 74   x.73 numeric  0 -4.062321e-02 0.9449649 -0.0920200240 0.8914131 -2.6467566
## 75   x.74 numeric  0 -1.990616e-02 0.9607625 -0.0564070180 0.8675906 -2.9096096
## 76   x.75 numeric  0  2.570727e-02 0.9836409 -0.0351377110 0.9929083 -2.4697513
## 77   x.76 numeric  0 -1.131288e-02 1.0263640 -0.0407999070 1.0683111 -2.8053274
## 78   x.77 numeric  0 -8.280495e-03 0.9840483 -0.0417080110 0.9832478 -2.8748853
## 79   x.78 numeric  0  3.109553e-02 0.9997129 -0.0991060565 0.7268028 -2.5061313
## 80   x.79 numeric  0 -3.637262e-02 1.0272531 -0.0077227355 0.9261007 -3.8997127
## 81   x.80 numeric  0  4.521755e-02 1.0900693 -0.2864923415 0.2668918 -0.8941760
## 82   x.81 numeric  0 -2.909372e-03 0.9994980 -0.0431333250 1.0004871 -2.6231871
## 83   x.82 numeric  0 -1.837178e-02 0.9891541  0.0156221485 1.0153998 -3.3610478
## 84   x.83 numeric  0  4.043303e-02 0.9822753  0.1001085540 0.9464838 -2.7070774
## 85   x.84 numeric  0 -5.920678e-02 1.0096556 -0.0221655595 0.9397000 -3.8145633
## 86   x.85 numeric  0  1.151595e-02 0.9898174 -0.0435484620 0.8908361 -2.2662694
## 87   x.86 numeric  0  9.934384e-03 1.0023294 -0.0361134715 0.9939461 -2.4995788
## 88   x.87 numeric  0  1.802882e-04 0.8248721 -0.1068701185 0.5675420 -1.5103254
## 89   x.88 numeric  0 -1.303535e-02 0.9807562 -0.0389361690 1.0171529 -3.9348946
## 90   x.89 numeric  0 -2.548646e-02 1.0020018 -0.1546438015 0.8531654 -2.0734171
## 91   x.90 numeric  0 -2.880810e-02 0.9996353 -0.0172720235 0.8768648 -2.8450578
## 92   x.91 numeric  0 -2.031723e-02 0.9763805 -0.0250497115 0.9507388 -2.4373873
## 93   x.92 numeric  0  4.289073e-02 1.0056015  0.0008311965 0.9812616 -2.6199907
## 94   x.93 numeric  0 -4.127844e-02 0.9714099 -0.0030627650 0.9023748 -3.6859134
## 95   x.94 numeric  0 -9.915449e-02 0.9564644 -0.0869575330 0.9420245 -2.8108734
## 96   x.95 numeric  0 -9.924198e-03 0.9802883 -0.1244438035 0.9294325 -2.8073946
## 97   x.96 numeric  0  2.244691e-02 0.9873048  0.0201280385 0.9314573 -2.8580812
## 98   x.97 numeric  0  1.376639e-02 0.9903350 -0.1782534580 0.7415566 -1.4746687
## 99   x.98 numeric  0 -4.857316e-02 1.0195882 -0.1068141980 1.0592673 -2.7358905
## 100  x.99 numeric  0  3.731843e-02 1.0173086 -0.1598241620 0.9028997 -1.7584131
## 101 x.100 numeric  0 -4.971312e-02 0.9758254 -0.0730920605 0.9867384 -2.4992633
## 102 x.101 numeric  0  8.916323e-03 1.0262887 -0.0297139570 0.9294830 -3.6845885
## 103 x.102 numeric  0  1.509050e-03 1.1573040 -0.0676545430 0.4588990 -3.2959126
## 104 x.103 numeric  0 -5.382208e-02 1.0283644 -0.0574490425 0.9181034 -3.5469319
## 105 x.104 numeric  0  4.635659e-02 1.0387169 -0.1093288550 1.0252073 -1.9174622
## 106 x.105 numeric  0 -7.409891e-03 0.9719003 -0.0059626485 0.9221154 -2.7980176
## 107 x.106 numeric  0  3.110522e-02 1.0162398 -0.1191683750 0.9085915 -2.1172927
## 108 x.107 numeric  0 -1.625718e-03 0.9877878 -0.0663288265 0.9381757 -2.5366464
## 109 x.108 numeric  0  4.274066e-02 1.0377427 -0.0452173390 0.8751987 -2.6599291
## 110 x.109 numeric  0 -4.258430e-02 0.9616551 -0.1797134750 0.7264697 -2.8628633
## 111 x.110 numeric  0  4.342404e-02 1.0024740 -0.0242251775 0.9491951 -3.2642541
## 112 x.111 numeric  0 -2.851341e-02 1.0114383 -0.0665203260 0.8587147 -2.5317930
## 113 x.112 numeric  0 -4.305274e-03 0.9699224 -0.0840509955 0.8937476 -2.7307673
## 114 x.113 numeric  0 -3.305167e-02 0.9972696 -0.0419599130 0.9948663 -2.5407225
## 115 x.114 numeric  0  3.972089e-02 1.0010262 -0.0669660745 0.7278121 -2.3840722
## 116 x.115 numeric  0 -2.259349e-02 1.0324772 -0.0788857575 0.9601269 -3.6444636
## 117 x.116 numeric  0 -1.767802e-02 0.9924357 -0.0100628130 0.9204576 -3.0378071
## 118 x.117 numeric  0  1.281790e-03 0.9796801 -0.0085415050 1.0140975 -2.6299346
## 119 x.118 numeric  0 -2.176424e-02 1.0208104 -0.0494801955 1.0198142 -3.2348092
## 120 x.119 numeric  0 -1.078031e-02 0.9724116 -0.0890604315 0.9383423 -2.7369414
## 121 x.120 numeric  0 -3.692889e-02 1.0171017 -0.0014120825 0.9209354 -3.4198208
## 122 x.121 numeric  0  7.632087e-02 0.9850670  0.0432631235 0.9461251 -3.3906014
## 123 x.122 numeric  0  1.062866e-02 0.9540795  0.0466776245 0.8689912 -3.0557451
## 124 x.123 numeric  0 -8.193734e-03 0.9694755 -0.1566553535 0.7808332 -2.2679162
## 125 x.124 numeric  0 -4.152163e-02 1.0177685 -0.1190649150 1.0900533 -2.5248082
## 126 x.125 numeric  0  5.455663e-02 1.0019716  0.0063278455 0.9362776 -2.3067014
## 127 x.126 numeric  0  4.559152e-02 0.9846998  0.0589226390 0.8942516 -3.4215080
## 128 x.127 numeric  0 -2.021980e-02 0.9974664 -0.0019601095 0.9937769 -2.7200227
## 129 x.128 numeric  0 -2.297843e-02 0.9663233 -0.0392104205 0.9151472 -3.1324848
## 130 x.129 numeric  0 -1.822546e-02 0.9866949 -0.0347119635 0.9620755 -3.3312355
## 131 x.130 numeric  0 -7.678958e-03 0.9813352 -0.0081175055 0.9437169 -3.0126118
## 132 x.131 numeric  0  4.245811e-03 0.9656934 -0.1772349720 0.8678803 -1.9385237
## 133 x.132 numeric  0 -1.369096e-03 1.0063933  0.0224665890 0.9589092 -3.0338644
## 134 x.133 numeric  0  1.482543e-02 1.0390484 -0.0124474990 0.9512414 -2.9443219
## 135 x.134 numeric  0 -8.784546e-03 0.9548117 -0.0640698960 0.9775860 -2.6773216
## 136 x.135 numeric  0 -3.486564e-02 0.9589554 -0.0634051295 0.9061108 -3.1187385
## 137 x.136 numeric  0 -6.498966e-02 0.9861655 -0.0389626380 1.0454972 -2.8596313
## 138 x.137 numeric  0  2.740248e-02 0.9889309  0.0268627025 1.0271659 -2.8796757
## 139 x.138 numeric  0  3.968180e-02 1.0291299 -0.1464049630 0.6749076 -2.2094809
## 140 x.139 numeric  0 -5.337930e-02 1.0664139  0.0522091650 1.1467456 -3.3739128
## 141 x.140 numeric  0  2.222830e-02 0.9853480 -0.1634248995 0.7995447 -1.8011501
## 142 x.141 numeric  0 -4.405304e-04 0.9931808 -0.0775589355 0.9040504 -2.2408235
## 143 x.142 numeric  0 -1.683547e-02 1.0372531  0.0297541715 0.9662128 -2.6889397
## 144 x.143 numeric  0  7.431461e-04 0.9394768  0.0012504820 0.8642855 -2.7962354
## 145 x.144 numeric  0 -2.229632e-02 0.9566990 -0.0794893490 1.0523903 -2.5978809
## 146 x.145 numeric  0  1.095655e-01 1.0392590 -0.0088024395 0.8034507 -2.1000529
## 147 x.146 numeric  0 -4.235307e-02 0.9331325 -0.0803474460 0.9136416 -2.5716113
## 148 x.147 numeric  0  4.594239e-02 1.0140985 -0.1463070535 0.9294368 -2.0863799
## 149 x.148 numeric  0 -8.229536e-03 1.0240686 -0.0638168450 1.0000563 -2.9388696
## 150 x.149 numeric  0  4.548621e-02 1.0296794 -0.1714430455 0.8628795 -1.7454969
## 151 x.150 numeric  0 -3.506317e-02 0.9655271 -0.1376468700 0.8186874 -2.9265124
## 152 x.151 numeric  0 -2.126758e-02 1.0443267 -0.1052971155 1.0672890 -2.0705804
## 153 x.152 numeric  0 -5.556155e-04 0.9809811 -0.0857686030 0.9888445 -3.8328603
## 154 x.153 numeric  0  5.701331e-02 1.0334848 -0.0861679030 0.8805456 -2.6052549
## 155 x.154 numeric  0  1.770370e-02 1.0205782 -0.0029169495 0.9725715 -2.5330050
## 156 x.155 numeric  0 -8.467016e-06 0.9439520 -0.0109934105 0.8713868 -3.0638827
## 157 x.156 numeric  0 -4.854711e-02 0.9933597 -0.1649202020 0.9741670 -2.2585084
## 158 x.157 numeric  0 -9.948831e-03 0.9947231 -0.1158082745 0.8556460 -1.7555590
## 159 x.158 numeric  0  2.684259e-02 0.9841746 -0.0081104580 0.9470456 -3.1310003
## 160 x.159 numeric  0  9.192058e-02 0.9903810  0.0841364950 0.9669095 -2.5372630
## 161 x.160 numeric  0  6.648531e-02 1.0095282 -0.0605884135 1.0187299 -1.7941481
## 162 x.161 numeric  0  7.888606e-04 1.0571604 -0.1231171275 0.8119609 -2.6598592
## 163 x.162 numeric  0  8.570243e-03 0.9697069 -0.0186760200 0.9426199 -2.8392832
## 164 x.163 numeric  0 -4.789788e-02 1.0058281  0.0177151200 1.0777460 -1.8652884
## 165 x.164 numeric  0 -7.253745e-02 0.9295572 -0.0781395825 0.8367536 -2.8540110
## 166 x.165 numeric  0 -1.652773e-03 1.0045030 -0.0218414790 0.9211836 -3.6082769
## 167 x.166 numeric  0 -1.675599e-02 0.9480325 -0.1123442610 0.8218080 -2.6934640
## 168 x.167 numeric  0 -4.031837e-02 1.0124429 -0.0371396145 1.0365678 -2.4610495
## 169 x.168 numeric  0 -8.427913e-02 0.9927935 -0.0637398885 0.9717916 -3.9394829
## 170 x.169 numeric  0  3.087969e-03 0.9855320 -0.0592019110 0.8939594 -2.9821834
## 171 x.170 numeric  0 -1.884322e-02 1.0086735  0.0117031215 0.9647287 -2.8891495
## 172 x.171 numeric  0  1.145501e-02 0.9633535 -0.0756642405 0.8979326 -2.6101455
## 173 x.172 numeric  0  8.087929e-02 0.9722177  0.0805238830 0.9260243 -2.3232105
## 174 x.173 numeric  0  2.941296e-02 1.0031324  0.0080698800 1.0063030 -3.3015236
## 175 x.174 numeric  0  1.711691e-02 0.9877321  0.0034835340 1.0502123 -2.3678433
## 176 x.175 numeric  0 -1.893954e-02 1.0209781 -0.0049889425 0.9603911 -4.2829750
## 177 x.176 numeric  0  6.495729e-02 0.9778161  0.1830708265 0.8991326 -3.5681295
## 178 x.177 numeric  0  3.751010e-02 1.0775427 -0.0356678610 0.7611716 -2.4454308
## 179 x.178 numeric  0 -1.422031e-02 0.9610008 -0.0876968405 0.8909987 -3.6326792
## 180 x.179 numeric  0 -3.216965e-02 0.9922158 -0.0335935425 1.0027126 -3.2450201
## 181 x.180 numeric  0  5.891680e-02 1.0740878 -0.1409008910 0.7006239 -1.9628191
## 182 x.181 numeric  0 -1.359045e-02 1.0088408 -0.3275089495 0.7133611 -1.5124515
## 183 x.182 numeric  0  3.425545e-02 1.0229401  0.0040415710 0.9596815 -3.4218868
## 184 x.183 numeric  0  4.206231e-02 1.0004830 -0.0110723775 0.9320651 -3.1371667
## 185 x.184 numeric  0  1.189937e-02 0.9812840 -0.0628711515 0.8729796 -2.1514364
## 186 x.185 numeric  0 -6.680064e-02 0.9364651 -0.2349599845 0.7746327 -2.0846838
## 187 x.186 numeric  0  1.468415e-02 0.9886278 -0.0587753385 0.8953674 -2.4333664
## 188 x.187 numeric  0  4.141096e-02 1.1778272 -0.0879476975 0.6419682 -2.2167848
## 189 x.188 numeric  0  3.969418e-03 0.9799169 -0.0024212390 0.8480469 -2.8715989
## 190 x.189 numeric  0 -7.236091e-03 0.9831703 -0.0697339265 0.9006167 -2.4730204
## 191 x.190 numeric  0  5.017366e-02 1.0151546  0.0534530915 0.9228375 -2.7569555
## 192 x.191 numeric  0  7.856426e-03 0.9875581 -0.0211339170 0.9788796 -3.0821710
## 193 x.192 numeric  0 -2.472495e-02 1.0144094  0.0885320090 1.0060009 -2.9457762
## 194 x.193 numeric  0 -6.450726e-03 1.0503116 -0.0817285270 0.7419520 -2.2923954
## 195 x.194 numeric  0  2.342548e-02 1.0167213  0.0257332700 1.0604837 -2.4701776
## 196 x.195 numeric  0  2.105864e-02 0.9878600  0.0131253145 0.9115352 -2.5614464
## 197 x.196 numeric  0 -5.094621e-02 0.9725503 -0.0163239645 0.9816072 -2.8845665
## 198 x.197 numeric  0 -4.765846e-02 0.9022794 -0.0704940845 0.7729097 -2.8357401
## 199 x.198 numeric  0  4.870768e-02 1.0717709 -0.1119547620 0.7460653 -2.7980127
## 200 x.199 numeric  0  5.190293e-02 1.0280419  0.0009757475 0.9876286 -2.8206058
## 201 x.200 numeric  0 -3.768627e-02 0.9752159 -0.0475350350 0.9533911 -3.1496173
## 202 x.201 numeric  0 -3.520748e-02 0.9669816  0.0062590750 0.9083334 -3.2998386
## 203 x.202 numeric  0 -6.617257e-03 1.0103204 -0.1097392260 0.9663376 -2.5371318
## 204 x.203 numeric  0  2.513698e-03 0.9687931 -0.0288684090 1.0207996 -2.6511585
## 205 x.204 numeric  0  3.020108e-02 1.0093146 -0.0159048290 0.9968578 -3.0972131
## 206 x.205 numeric  0 -1.203868e-02 1.0067642 -0.0273529565 0.9387657 -3.0786889
## 207 x.206 numeric  0  2.913500e-02 1.0020100  0.0192316005 0.9492632 -2.4714542
## 208 x.207 numeric  0  1.283315e-02 0.9815085 -0.0035860985 0.9746697 -3.3402460
## 209 x.208 numeric  0 -2.700708e-03 0.9929217 -0.4510133895 0.4363510 -1.2197082
## 210 x.209 numeric  0  5.762292e-03 1.0202053 -0.1900135325 0.6436719 -2.4082662
## 211 x.210 numeric  0  3.303715e-02 0.9952640  0.0021145265 0.9292379 -2.4807260
## 212 x.211 numeric  0  3.342453e-02 1.0040741  0.0255045450 0.8823485 -2.7981735
## 213 x.212 numeric  0 -3.881089e-02 0.9595527 -0.2261056295 0.7925505 -2.2733557
## 214 x.213 numeric  0  3.860697e-02 0.9940182  0.0248186400 0.9513373 -2.8733613
## 215 x.214 numeric  0 -1.029588e-02 0.9703664 -0.1130381305 0.9117842 -2.0558463
## 216 x.215 numeric  0  3.841035e-02 0.9974614 -0.1522392190 0.8510804 -2.0211939
## 217 x.216 numeric  0 -3.180436e-02 1.0015393 -0.0580940340 0.9188953 -2.5634476
## 218 x.217 numeric  0 -1.287502e-02 0.9132367 -0.0143039405 0.9086722 -2.4894723
## 219 x.218 numeric  0  5.463195e-02 1.0091801  0.1093521465 0.9543228 -3.0707932
## 220 x.219 numeric  0 -3.929600e-02 0.9938765  0.1000822085 1.1274874 -1.9659200
## 221 x.220 numeric  0  5.249920e-04 1.0008420 -0.0014871685 1.0234774 -2.9902279
## 222 x.221 numeric  0 -3.227311e-02 0.9866803 -0.2314727180 0.9683714 -1.6475311
## 223 x.222 numeric  0 -6.569472e-02 0.9426976 -0.1676840120 0.8246215 -2.6291035
## 224 x.223 numeric  0 -1.709275e-02 0.9978720  0.0005551870 0.9336896 -2.5631473
## 225 x.224 numeric  0 -2.272616e-02 1.0050543 -0.1657655205 0.8997539 -2.5555929
## 226 x.225 numeric  0  7.737633e-02 1.0281274  0.0649036845 0.9651993 -2.7211932
## 227 x.226 numeric  0  5.281977e-02 0.9888372  0.0363700695 0.9650899 -3.1702598
## 228 x.227 numeric  0 -5.585186e-02 1.0180327  0.0147024760 1.0057336 -2.9056007
## 229 x.228 numeric  0  5.201048e-03 0.9818950  0.0218639020 0.9450619 -2.5942394
## 230 x.229 numeric  0 -1.698695e-02 0.9885500 -0.0724692005 0.8465634 -3.0530931
## 231 x.230 numeric  0  4.405414e-03 0.9837696 -0.1207421780 0.9927073 -2.5514641
## 232 x.231 numeric  0 -6.861139e-02 0.9812234 -0.1091413225 1.1009046 -2.3533334
## 233 x.232 numeric  0 -1.250068e-02 1.0106252 -0.1711686390 0.8836270 -2.8497341
## 234 x.233 numeric  0  2.166615e-03 1.0811326 -0.1023361965 0.6555537 -2.1133373
## 235 x.234 numeric  0 -3.086440e-02 0.9692560 -0.0672527135 0.8611589 -2.0412512
## 236 x.235 numeric  0  2.711256e-02 0.9803984 -0.0284811990 0.8883779 -2.7080999
## 237 x.236 numeric  0 -6.562471e-03 1.0293610  0.0389142670 0.9726675 -3.1071758
## 238 x.237 numeric  0 -4.153344e-02 1.0238127 -0.0917704715 0.8363190 -3.8056437
## 239 x.238 numeric  0 -3.403842e-02 1.0249753 -0.0361772160 1.0229221 -4.6849677
## 240 x.239 numeric  0  1.832064e-02 0.9626112  0.0069805040 0.9385739 -2.8130919
## 241 x.240 numeric  0 -1.936219e-02 1.0389958  0.0039915575 0.9123925 -3.4838049
## 242 x.241 numeric  0  1.593361e-02 0.9862846  0.0770711105 1.0266205 -3.8504390
## 243 x.242 numeric  0 -3.072416e-02 0.9963710  0.0153866930 0.9317099 -3.4239442
## 244 x.243 numeric  0  8.524691e-02 0.9972159 -0.0072130245 0.9178839 -2.5166262
## 245 x.244 numeric  0 -3.045226e-02 0.9210255 -0.1177618095 0.6771971 -2.3592993
## 246 x.245 numeric  0 -3.392120e-02 1.0038648 -0.0364888710 0.9997639 -3.0966133
## 247 x.246 numeric  0 -6.000782e-02 0.8987465 -0.1184093970 0.8339684 -2.7094776
## 248 x.247 numeric  0  3.141877e-02 1.0020845  0.0423581750 1.0375082 -3.3316052
## 249 x.248 numeric  0  2.086642e-02 1.0305558  0.0113172940 1.0191515 -3.3447914
## 250 x.249 numeric  0 -9.888478e-03 0.9919227 -0.0389475655 0.9757269 -2.8313887
## 251 x.250 numeric  0  3.876998e-02 1.0445203  0.0313606170 0.9832050 -3.0854953
## 252 x.251 numeric  0  4.930231e-02 1.0457838 -0.0939240850 1.0081292 -2.2253539
## 253 x.252 numeric  0  5.802674e-02 0.9539001  0.0611132335 1.0020007 -2.5006618
## 254 x.253 numeric  0 -7.102932e-03 0.9945565  0.0303861200 1.0554572 -2.7812584
## 255 x.254 numeric  0  1.487101e-03 1.0166228  0.0011308795 0.9592225 -3.6196834
## 256 x.255 numeric  0 -1.467396e-02 0.9999929 -0.0205040000 0.8520632 -2.9639559
## 257 x.256 numeric  0  1.983212e-02 1.0326425 -0.1210034185 0.3739978 -1.3398858
## 258 x.257 numeric  0  4.170171e-02 1.0112458 -0.1617007940 0.9006063 -2.0759371
## 259 x.258 numeric  0  1.454943e-02 0.9354707 -0.0202506475 0.9045289 -2.8005142
## 260 x.259 numeric  0  1.019609e-02 0.9401291  0.0103756460 0.9039453 -3.2109939
## 261 x.260 numeric  0  2.295609e-02 0.9721143 -0.0178292430 0.8997944 -2.6793429
## 262 x.261 numeric  0  6.540808e-02 1.0848435 -0.1420203700 0.6515392 -2.0011516
## 263 x.262 numeric  0  4.439845e-02 0.9728811  0.0382205955 0.9591444 -2.9445164
## 264 x.263 numeric  0  4.025975e-02 1.1133190 -0.0453701470 0.8693194 -2.7337131
## 265 x.264 numeric  0  1.150286e-02 0.9637538 -0.0303556140 0.9515204 -3.1647574
## 266 x.265 numeric  0 -2.262264e-02 0.9887793 -0.0686562370 0.9344294 -2.4205028
## 267 x.266 numeric  0 -4.918069e-02 1.0292966 -0.1020714825 0.9874786 -2.6475572
## 268 x.267 numeric  0 -1.560668e-02 0.9815584 -0.0116881870 0.9329049 -2.7645482
## 269 x.268 numeric  0 -7.003004e-02 0.9931834 -0.1176193445 0.9962037 -2.6099820
## 270 x.269 numeric  0 -4.029223e-02 1.0235360 -0.0722461135 1.0090712 -2.8891423
## 271 x.270 numeric  0 -2.180154e-02 1.0172747 -0.0100961075 0.9422415 -2.8900182
## 272 x.271 numeric  0 -9.033658e-02 0.9840468 -0.2164634260 0.8772265 -2.7672259
## 273 x.272 numeric  0 -3.919760e-02 1.0012977 -0.0537531980 1.0173760 -3.2702310
## 274 x.273 numeric  0  2.138585e-02 1.1241185 -0.1225021155 0.8363269 -2.2791679
## 275 x.274 numeric  0  6.729168e-02 1.1669944 -0.1192653135 0.6198838 -1.7078334
## 276 x.275 numeric  0  2.296333e-02 1.0669716 -0.1345200435 0.5789488 -1.7645757
## 277 x.276 numeric  0  6.903576e-02 1.0157511  0.0526182810 1.0009380 -3.5365081
## 278 x.277 numeric  0 -1.237098e-02 1.0619433 -0.0905574355 0.8609847 -2.9606975
## 279 x.278 numeric  0  7.656865e-03 0.9920562 -0.1201293675 0.7931107 -2.3797285
## 280 x.279 numeric  0  2.338274e-02 1.0301032 -0.0118447840 0.9269998 -3.0902691
## 281 x.280 numeric  0  4.007737e-02 1.0185871  0.0294754425 0.9698494 -2.4299223
## 282 x.281 numeric  0  2.997782e-02 1.0276983 -0.1571701760 0.7949199 -2.9180420
## 283 x.282 numeric  0 -9.296433e-03 1.0081828 -0.0840312275 0.8997324 -2.0955681
## 284 x.283 numeric  0 -1.839850e-02 0.8909414 -0.0047653200 0.8079399 -3.4248807
## 285 x.284 numeric  0 -2.699289e-03 1.0022492 -0.4303869695 0.3792975 -0.9693039
## 286 x.285 numeric  0  1.737184e-02 1.0351221 -0.1653285115 0.9822200 -2.5719937
## 287 x.286 numeric  0 -6.916675e-02 0.9746855 -0.1929069000 1.0166750 -2.1733997
## 288 x.287 numeric  0  3.552411e-02 1.0207022  0.0852237660 1.0087019 -4.0175873
## 289 x.288 numeric  0  1.075000e-01 1.0842804 -0.0835433320 1.1040531 -1.5906060
## 290 x.289 numeric  0  4.119347e-02 1.0190117 -0.1133542375 1.1267600 -2.2865794
## 291 x.290 numeric  0 -4.278087e-02 1.0207322 -0.0898780175 0.9926064 -2.7405669
## 292 x.291 numeric  0  2.359139e-03 0.9965188 -0.0022886720 0.9840402 -3.1383741
## 293 x.292 numeric  0 -4.285909e-03 0.9921366 -0.0360298215 0.8621742 -3.1930688
## 294 x.293 numeric  0 -1.827661e-02 1.0135014 -0.0262891135 1.0281349 -2.8123478
## 295 x.294 numeric  0  4.327779e-02 0.9728283  0.0352250395 0.9139673 -2.7021558
## 296 x.295 numeric  0  4.968256e-04 1.0549976 -0.0355479195 0.9650499 -3.7444311
## 297 x.296 numeric  0  1.625004e-03 0.9907840 -0.0284678350 0.9693265 -4.1827700
## 298 x.297 numeric  0  4.376792e-02 0.9831502 -0.0719601240 0.8893395 -2.5712009
## 299 x.298 numeric  0 -3.437473e-02 0.9534203 -0.0991255495 0.9733040 -2.9404632
## 300 x.299 numeric  0 -2.387864e-02 0.9851112 -0.0262019595 0.9180065 -2.8560453
## 301 x.300 numeric  0 -5.765587e-02 1.0080245 -0.1236498295 0.9299847 -2.6970627
## 302 x.301 numeric  0  4.677280e-02 1.0025027 -0.0319814760 0.9176982 -3.0628180
## 303 x.302 numeric  0  2.842535e-02 0.9726939 -0.0435411870 0.9494280 -2.2847667
## 304 x.303 numeric  0  4.050028e-02 1.1136626 -0.2340934255 0.3034393 -1.1567978
## 305 x.304 numeric  0  3.745206e-02 0.9767523 -0.0605165870 0.9769591 -2.2825849
## 306 x.305 numeric  0 -3.054821e-02 1.0150870 -0.0316349485 0.9620421 -3.7206718
## 307 x.306 numeric  0  1.369666e-02 1.0039302 -0.0193122030 0.9488959 -2.7760788
## 308 x.307 numeric  0 -7.806054e-02 0.9723978 -0.1984478310 0.9590979 -2.2541536
## 309 x.308 numeric  0  1.077440e-01 1.0824404 -0.0836054565 0.8333465 -2.0316864
## 310 x.309 numeric  0 -5.904904e-04 1.0726396 -0.0220827465 0.9129530 -2.8684550
## 311 x.310 numeric  0  4.190946e-02 1.0113589  0.0017275540 0.8720786 -2.5778752
## 312 x.311 numeric  0  2.432150e-02 0.9861142 -0.0259302665 1.0519539 -2.7164093
## 313 x.312 numeric  0 -1.447898e-02 0.9866795 -0.0267201295 1.0685712 -3.3980383
## 314 x.313 numeric  0 -1.810439e-02 0.9719870  0.0253608100 0.9084974 -3.9743208
## 315 x.314 numeric  0 -1.642767e-04 0.9809090 -0.0305018360 0.8890257 -4.2157795
## 316 x.315 numeric  0 -2.180831e-02 0.9997635 -0.0737445175 0.9386251 -3.1000730
## 317 x.316 numeric  0 -3.783978e-02 1.0098626 -0.1264798895 0.8993355 -3.1956498
## 318 x.317 numeric  0  1.612626e-02 0.9534271  0.0594848390 0.9075856 -3.3096468
## 319 x.318 numeric  0  4.312810e-02 1.0441423  0.0622277650 0.9085820 -3.6504512
## 320 x.319 numeric  0  5.211965e-02 1.0988530 -0.1868691665 0.7277208 -1.7551166
## 321 x.320 numeric  0 -4.031151e-03 1.0435027 -0.0083576725 1.0632688 -3.1869021
## 322 x.321 numeric  0 -2.105443e-02 1.0421670  0.0308202265 0.9770474 -3.3711501
## 323 x.322 numeric  0  3.618256e-02 1.0142662 -0.0204293805 0.9053571 -2.6302819
## 324 x.323 numeric  0  5.219854e-02 1.0418280 -0.1244585775 0.8008164 -2.1609889
## 325 x.324 numeric  0 -6.834171e-02 0.9460973 -0.0839629205 0.9388735 -2.5995904
## 326 x.325 numeric  0 -7.508893e-03 1.0063584 -0.0440985440 1.0040963 -2.6351842
## 327 x.326 numeric  0 -1.252373e-02 0.9980535 -0.0679461790 0.9882901 -2.2538374
## 328 x.327 numeric  0  1.466059e-03 0.9842515 -0.0598595065 0.8989135 -3.2223759
## 329 x.328 numeric  0  3.294955e-02 1.0199525 -0.0555384905 1.0296495 -1.7934485
## 330 x.329 numeric  0  4.430769e-02 1.0232156 -0.0834230605 0.8801590 -2.1520306
## 331 x.330 numeric  0  1.221705e-02 0.9929725 -0.0835100320 0.9935628 -2.7629460
## 332 x.331 numeric  0 -3.591643e-02 0.9802632 -0.0874718525 0.9453055 -2.7433094
## 333 x.332 numeric  0  1.072552e-02 0.9619413 -0.3535175570 0.3465223 -0.9765794
## 334 x.333 numeric  0  4.265166e-02 0.9685268  0.0554382080 0.9766033 -3.0183940
## 335 x.334 numeric  0  2.628111e-02 0.9842452 -0.0615791835 0.9817843 -2.3769941
## 336 x.335 numeric  0  1.829405e-02 1.0193292 -0.0225583405 0.9575419 -2.7558915
## 337 x.336 numeric  0 -2.668463e-02 1.0018931 -0.0269896150 0.9496973 -2.6863301
## 338 x.337 numeric  0 -7.002910e-02 0.9992696 -0.1275964835 0.9468223 -3.0253989
## 339 x.338 numeric  0  5.753366e-03 1.0267053  0.0641226865 1.0065264 -2.8989993
## 340 x.339 numeric  0 -1.152762e-02 0.9862891 -0.0682275575 0.9718812 -2.7114053
## 341 x.340 numeric  0 -1.759205e-02 1.0143320  0.0406820630 0.9908177 -2.8955170
## 342 x.341 numeric  0 -2.162457e-02 0.8848454 -0.1896909620 0.7277534 -2.1220347
## 343 x.342 numeric  0 -3.686437e-02 0.9867153 -0.0906514565 0.9455037 -2.4296334
## 344 x.343 numeric  0  9.069326e-04 1.0450485 -0.1677712710 0.8710119 -1.7625092
## 345 x.344 numeric  0  6.447162e-03 0.9935432 -0.0372005905 0.9492630 -2.5942370
## 346 x.345 numeric  0  2.176013e-02 1.0257596 -0.0058398845 1.0240225 -3.8358085
## 347 x.346 numeric  0 -1.985829e-02 1.0374324  0.0422246625 0.9650122 -2.6400342
## 348 x.347 numeric  0  1.654450e-02 1.0688543  0.0224669270 1.0950893 -2.8233009
## 349 x.348 numeric  0 -1.134292e-02 1.0262587 -0.0723573910 0.9440312 -2.4889424
## 350 x.349 numeric  0  1.317738e-02 0.9857421 -0.1325587360 0.8628879 -2.3356226
## 351 x.350 numeric  0  5.917633e-02 0.9952809  0.0531081435 0.9083168 -2.7327261
## 352 x.351 numeric  0  4.565014e-02 1.0525108 -0.0483051260 0.9143667 -1.8773885
## 353 x.352 numeric  0  2.512879e-02 1.0263917 -0.0044048090 1.0065085 -2.8770633
## 354 x.353 numeric  0 -2.116026e-03 1.0197898 -0.1348514445 0.9782223 -2.0408120
## 355 x.354 numeric  0  2.045153e-02 0.9815174  0.1375670370 0.7567248 -4.8821093
## 356 x.355 numeric  0  2.656056e-02 0.9944470 -0.1484313830 0.5982679 -1.4587474
## 357 x.356 numeric  0 -2.087171e-02 0.9477244 -0.0075469330 0.8570636 -3.1591879
## 358 x.357 numeric  0 -9.247617e-03 0.9640570 -0.0139335320 0.9654437 -2.8520737
## 359 x.358 numeric  0 -5.154125e-02 1.0101654 -0.0950682985 0.9762125 -3.0715842
## 360 x.359 numeric  0  3.942353e-02 1.0082772  0.0066020205 1.0094358 -2.7667177
## 361 x.360 numeric  0  2.777086e-02 0.9954374 -0.0354895795 1.0090216 -2.4464121
## 362 x.361 numeric  0  1.903546e-02 1.0088679  0.0125995500 0.9617066 -2.8600509
## 363 x.362 numeric  0 -7.268807e-02 0.9184036 -0.1537508405 0.8068770 -2.2998287
## 364 x.363 numeric  0 -1.884617e-03 1.0111297 -0.0604549865 0.9116667 -3.1912012
## 365 x.364 numeric  0 -2.621206e-02 0.9543842 -0.0728434860 0.8848265 -2.2811077
## 366 x.365 numeric  0  5.615693e-02 1.0981617 -0.1711389545 0.7045391 -1.9044438
## 367 x.366 numeric  0 -3.501811e-02 0.9940801 -0.0267401920 0.8809584 -6.1095607
## 368 x.367 numeric  0  4.332500e-03 1.0152907  0.0552587685 0.9748968 -2.9588199
## 369 x.368 numeric  0  2.381647e-02 1.0280234 -0.0288751525 1.0114075 -3.4388196
## 370 x.369 numeric  0  8.111407e-02 1.0212833 -0.0151600135 0.9212118 -2.4231132
## 371 x.370 numeric  0  1.471196e-02 1.0383984 -0.0987681200 0.7608982 -2.7523638
## 372 x.371 numeric  0  2.159594e-02 1.0358430  0.0476069950 0.9722658 -3.3694911
## 373 x.372 numeric  0  3.378630e-04 0.9977576 -0.0816115020 1.0609771 -2.0659757
## 374 x.373 numeric  0 -5.302103e-02 0.9809282 -0.0362972900 1.0156108 -2.8674355
## 375 x.374 numeric  0 -3.131927e-02 1.0057218 -0.0415039855 0.9104321 -2.6834568
## 376 x.375 numeric  0 -1.101332e-02 1.0194596 -0.0973525230 0.8210284 -2.0581053
## 377 x.376 numeric  0  5.602803e-03 1.0134018  0.0620416515 1.0173181 -3.0811883
## 378 x.377 numeric  0  1.698156e-02 1.0297438  0.2132601405 0.9257855 -4.4185454
## 379 x.378 numeric  0  7.982605e-03 0.9920231  0.0221250035 0.9193323 -3.3021218
## 380 x.379 numeric  0  4.698486e-02 0.9656268  0.0070110415 0.9543222 -2.3986738
## 381 x.380 numeric  0  3.133921e-02 0.9612590  0.0461062870 1.0383754 -2.7085831
## 382 x.381 numeric  0 -3.684629e-02 0.9607830 -0.0621349420 0.8658643 -2.9293820
## 383 x.382 numeric  0  3.880270e-03 0.9854553 -0.1640665765 0.8215161 -2.0074029
## 384 x.383 numeric  0  1.842774e-02 1.0280665 -0.1894160915 0.7474158 -2.2093460
## 385 x.384 numeric  0 -4.713999e-02 0.9967957 -0.0095817480 0.9871795 -3.6151933
## 386 x.385 numeric  0 -5.316294e-02 0.9504155 -0.1322050470 0.8759271 -2.4517404
## 387 x.386 numeric  0  1.886034e-02 1.0292027 -0.1297769260 0.8604287 -2.3710073
## 388 x.387 numeric  0  4.035894e-02 1.0772912 -0.1910028900 0.7518219 -1.6765539
## 389 x.388 numeric  0 -5.950911e-03 1.0209567 -0.1048988480 0.9614443 -2.4846943
## 390 x.389 numeric  0  3.090074e-02 1.0065386 -0.0057392225 0.8310866 -2.8697958
## 391 x.390 numeric  0 -3.449678e-02 0.9953417 -0.0962653400 1.0689122 -2.5774248
## 392 x.391 numeric  0  5.080238e-03 0.9860824 -0.0559280185 0.9466379 -2.5824967
## 393 x.392 numeric  0 -5.822049e-02 0.9638268 -0.0604702130 0.9411225 -3.0484857
## 394 x.393 numeric  0 -1.169054e-02 1.0141111  0.0331944880 0.9370559 -4.1078174
## 395 x.394 numeric  0  6.449890e-04 0.8882512 -0.1365336005 0.6677194 -2.2039643
## 396 x.395 numeric  0 -1.270905e-02 1.0121287  0.0330222545 0.9896230 -3.2339340
## 397 x.396 numeric  0  3.426650e-03 0.9615047  0.0105631195 0.9372206 -3.1612715
## 398 x.397 numeric  0 -2.920780e-02 1.0214909 -0.0559492820 0.9969460 -2.5527832
## 399 x.398 numeric  0 -5.679890e-02 0.9588604 -0.0715339420 0.9563033 -2.6070418
## 400 x.399 numeric  0  3.953267e-02 1.0263336 -0.0401774930 0.9669470 -2.4902619
## 401 x.400 numeric  0  3.048106e-02 1.0307085 -0.0094667250 1.0013156 -3.0068284
## 402 x.401 numeric  0 -4.618892e-02 0.8806192 -0.0884315830 0.6542010 -3.5506889
## 403 x.402 numeric  0  5.881173e-02 1.0220339 -0.0639460475 0.6435454 -1.8801818
## 404 x.403 numeric  0  3.518025e-02 1.0241962 -0.0671642040 1.0009463 -3.1832757
## 405 x.404 numeric  0 -3.675764e-02 1.0111575 -0.0416329795 0.9557710 -3.2868568
## 406 x.405 numeric  0 -4.960005e-02 0.9189577 -0.0901217770 0.8961392 -2.9570526
## 407 x.406 numeric  0 -7.438925e-03 0.9985834  0.0369231330 0.9852690 -3.2044325
## 408 x.407 numeric  0  3.231146e-02 0.9747647  0.0584832625 0.8997196 -2.6032447
## 409 x.408 numeric  0 -6.755014e-02 1.0158769 -0.0871340295 1.0093750 -2.8482570
## 410 x.409 numeric  0  1.163085e-02 0.9812958  0.0275000135 0.9441865 -3.4426578
## 411 x.410 numeric  0  4.409728e-03 0.9772650  0.0506824005 0.9023894 -2.8225017
## 412 x.411 numeric  0 -2.330688e-02 0.9579817 -0.0888356235 0.8918867 -2.2573725
## 413 x.412 numeric  0  3.433691e-02 1.0078915  0.0161802695 0.9551575 -3.6060328
## 414 x.413 numeric  0  2.570604e-03 1.0138812 -0.0263111715 0.9564148 -3.9118647
## 415 x.414 numeric  0  4.008865e-02 0.9828635 -0.1019086490 0.9563906 -2.0770734
## 416 x.415 numeric  0 -5.540058e-02 0.9257731 -0.0940755030 0.8737666 -2.7224394
## 417 x.416 numeric  0  3.491287e-02 1.1036992 -0.2975209740 0.3739526 -1.2503522
## 418 x.417 numeric  0  2.793631e-02 1.0418609 -0.0894063175 1.0124172 -2.3509617
## 419 x.418 numeric  0  6.278385e-02 1.0529623 -0.1007486245 0.9279174 -2.4282071
## 420 x.419 numeric  0  3.887762e-03 0.9771584  0.0038246050 0.8679287 -3.6824124
## 421 x.420 numeric  0 -7.230582e-02 0.9862259 -0.1094643825 0.8516030 -2.6292908
## 422 x.421 numeric  0 -7.956800e-03 0.9906497 -0.0252159620 0.9597586 -3.2943026
## 423 x.422 numeric  0 -6.134022e-02 0.9756513 -0.1282848140 0.8711350 -2.7745843
## 424 x.423 numeric  0 -3.804411e-02 1.0080526 -0.0598724105 0.9429462 -3.7142042
## 425 x.424 numeric  0  2.934364e-02 0.9642397  0.0177878440 0.9351459 -2.5837192
## 426 x.425 numeric  0  3.116830e-02 1.0303365 -0.1295416460 0.9371069 -2.0588492
## 427 x.426 numeric  0  2.383222e-02 1.0138595 -0.3726964770 0.4245585 -1.0191666
## 428 x.427 numeric  0  3.842090e-03 0.9916506  0.0212918995 1.0159234 -4.2992259
## 429 x.428 numeric  0 -5.121989e-02 0.9904986 -0.0735804805 1.0053902 -2.9904454
## 430 x.429 numeric  0  5.386295e-03 0.9873577 -0.0541585510 0.8532756 -2.9180194
## 431 x.430 numeric  0  2.797601e-02 1.0032687  0.0239578285 0.9841804 -3.8910810
## 432 x.431 numeric  0  3.801752e-02 1.0028096 -0.0849039130 0.9682918 -2.1945900
## 433 x.432 numeric  0 -3.699061e-03 0.9987005 -0.0112522250 0.9685928 -2.7387927
## 434 x.433 numeric  0  2.568238e-02 0.9987094  0.0170869870 0.9920801 -3.1780045
## 435 x.434 numeric  0  3.522036e-03 1.0099078 -0.1435228500 0.9231727 -2.3414435
## 436 x.435 numeric  0  1.624299e-02 1.0286861 -0.0162311170 0.9047598 -2.7287871
## 437 x.436 numeric  0 -6.257447e-02 0.9973778 -0.0044256500 0.8699485 -3.1510916
## 438 x.437 numeric  0 -3.808150e-02 1.0207158 -0.0132942050 0.9609050 -3.9317813
## 439 x.438 numeric  0  1.604073e-02 1.0050722 -0.1091186250 0.9050793 -2.7237512
## 440 x.439 numeric  0 -4.142577e-02 1.0046322 -0.0940968910 0.9811293 -4.0527814
## 441 x.440 numeric  0  2.402460e-02 0.9992703  0.0083487530 0.9841535 -2.6660285
## 442 x.441 numeric  0  1.710012e-02 1.0331342 -0.0399204650 1.0202604 -3.5998262
## 443 x.442 numeric  0 -4.925663e-02 0.8265528 -0.1360254975 0.5698387 -2.3769021
## 444 x.443 numeric  0  3.838080e-02 0.9938868 -0.0121147660 1.0003104 -3.0749107
## 445 x.444 numeric  0  4.450180e-02 1.0007559  0.0113179985 0.9184467 -2.7976121
## 446 x.445 numeric  0  7.852162e-03 1.0253062 -0.1048305550 1.0668684 -2.3208102
## 447 x.446 numeric  0 -7.614263e-02 0.9698070 -0.0452594660 0.8377645 -2.7893851
## 448 x.447 numeric  0 -1.979125e-02 0.9665217 -0.0208498830 1.0443971 -2.5426316
## 449 x.448 numeric  0  5.536104e-02 1.0085963  0.0021450800 0.9127180 -2.5690047
## 450 x.449 numeric  0  3.378881e-02 1.0147000 -0.0350385525 0.9996334 -2.7542470
## 451 x.450 numeric  0  1.035802e-02 1.0643753 -0.2722817985 0.3796779 -1.1968215
## 452 x.451 numeric  0 -1.571789e-03 1.0010505 -0.1077366665 0.9210081 -2.4370794
## 453 x.452 numeric  0 -1.696554e-03 1.0298464 -0.0432702210 0.9452639 -3.0987727
## 454 x.453 numeric  0  6.761899e-02 1.0097448  0.0409243490 1.0023028 -2.8820355
## 455 x.454 numeric  0  5.045514e-03 0.9939156 -0.0704891070 0.9855442 -2.7505683
## 456 x.455 numeric  0  2.840649e-03 1.0310654 -0.0439294120 1.0373086 -3.0790440
## 457 x.456 numeric  0  5.738186e-02 0.9908595  0.0226012405 0.8657793 -2.7445679
## 458 x.457 numeric  0 -3.501360e-02 0.9915808 -0.2368877410 0.3388125 -1.1212127
## 459 x.458 numeric  0 -5.820102e-02 0.9822804 -0.0500199285 0.9939934 -3.1692344
## 460 x.459 numeric  0 -5.299015e-02 1.0003800 -0.1253925515 0.9883165 -2.2819969
## 461 x.460 numeric  0  4.957704e-02 1.0091877 -0.0023053090 0.9663904 -3.5527132
## 462 x.461 numeric  0 -2.215887e-02 1.0108129 -0.0045380360 0.9324369 -3.3980111
## 463 x.462 numeric  0 -1.012226e-03 0.9606053  0.0304744830 1.0317444 -2.6470516
## 464 x.463 numeric  0 -1.252951e-02 0.7220939 -0.0364677435 0.6128564 -2.4029045
## 465 x.464 numeric  0  4.007335e-02 0.9857628 -0.1729953515 0.8300729 -1.9385783
## 466 x.465 numeric  0  1.085959e-02 0.9818179  0.0319490090 0.9851982 -2.7506402
## 467 x.466 numeric  0 -3.386583e-02 0.9946108 -0.0232350665 0.9295614 -2.8321954
## 468 x.467 numeric  0  5.365750e-02 1.0369655 -0.0593632915 0.9593358 -2.3758358
## 469 x.468 numeric  0  8.586739e-02 1.0063355  0.0327812915 0.8698350 -2.6041602
## 470 x.469 numeric  0 -1.766402e-03 0.9925044 -0.0446463520 0.9737602 -3.2706779
## 471 x.470 numeric  0  2.149499e-02 1.0266205 -0.1018775360 0.8888043 -2.3164977
## 472 x.471 numeric  0  1.938774e-04 0.9951654 -0.0400013950 0.9589208 -2.8046321
## 473 x.472 numeric  0 -3.897455e-02 1.0145102 -0.0221662020 1.0196355 -2.6975324
## 474 x.473 numeric  0  9.920650e-03 0.9772441 -0.0229388865 0.8499546 -3.6846294
## 475 x.474 numeric  0 -2.683832e-02 0.9929346  0.0845322615 1.0586463 -2.9788712
## 476 x.475 numeric  0  8.657794e-03 1.0407914 -0.0780236110 1.0381703 -2.8503811
## 477 x.476 numeric  0 -2.337632e-02 0.9939931 -0.2533257025 0.8385705 -2.5074364
## 478 x.477 numeric  0 -4.503787e-02 0.9808908 -0.0129026220 0.9180277 -3.3886547
## 479 x.478 numeric  0 -1.606715e-02 0.9988598 -0.3251695580 0.6353398 -1.3230134
## 480 x.479 numeric  0 -4.960480e-02 0.9856048 -0.0274608615 0.9543013 -2.4377300
## 481 x.480 numeric  0  7.627394e-03 0.9705802 -0.0730547955 0.9276881 -2.4184847
## 482 x.481 numeric  0  4.727482e-02 1.0308312  0.0401428070 0.9278072 -3.0358561
## 483 x.482 numeric  0  1.389421e-02 0.9940734  0.0279956015 1.0329339 -2.6080056
## 484 x.483 numeric  0 -1.072881e-02 0.9975851 -0.0916020415 0.9154685 -3.3564189
## 485 x.484 numeric  0 -3.103942e-02 0.9222928 -0.0378476865 0.8590337 -2.4267212
## 486 x.485 numeric  0  3.436726e-03 0.9632010 -0.1232588485 0.9033034 -2.0754875
## 487 x.486 numeric  0  4.832013e-02 0.9943747 -0.0109737900 0.9866415 -3.3259920
## 488 x.487 numeric  0 -3.169159e-02 0.9556667 -0.1394058265 0.8268345 -2.8307995
## 489 x.488 numeric  0 -5.597923e-03 1.0408235 -0.1680122620 0.9397780 -2.4306654
## 490 x.489 numeric  0 -4.561513e-02 1.0257335  0.0865736170 0.9305737 -3.8093359
## 491 x.490 numeric  0  3.042056e-03 1.0128649 -0.0231120860 1.0729543 -2.7681579
## 492 x.491 numeric  0  1.657528e-03 0.9976403 -0.0274155315 0.9276870 -2.8961885
## 493 x.492 numeric  0  4.901646e-02 1.0108221 -0.2166854970 0.5720679 -1.4945842
## 494 x.493 numeric  0  1.527092e-05 1.0164616 -0.0736241235 0.9753198 -2.3564220
## 495 x.494 numeric  0 -1.765509e-02 0.9293742  0.0685332185 0.8682012 -3.6209342
## 496 x.495 numeric  0  1.806932e-03 1.0300000 -0.1123157445 0.9199288 -2.5851656
## 497 x.496 numeric  0 -3.458528e-02 1.0004515 -0.1259206935 0.9171895 -2.4509176
## 498 x.497 numeric  0  2.465591e-02 0.8949428 -0.0900790240 0.8182064 -2.4284752
## 499 x.498 numeric  0 -3.639432e-04 0.9962034 -0.3217730845 0.3337505 -1.2158877
## 500 x.499 numeric  0  4.248555e-03 0.9525363  0.0057290795 0.8610490 -3.9058976
## 501 x.500 numeric  0  7.651200e-02 1.0404454  0.0248690330 0.9425011 -3.5286728
## 502 x.501 numeric  0 -2.717770e-02 0.9877227 -0.0533475570 0.9455996 -5.0650813
## 503 x.502 numeric  0 -8.088132e-02 1.0142500 -0.1440462975 1.0133754 -2.9610435
## 504 x.503 numeric  0 -3.172617e-02 0.9833340 -0.0718599930 1.0069401 -3.1474374
## 505 x.504 numeric  0  5.825887e-02 1.0395936 -0.1024182730 0.9753800 -2.1809571
## 506 x.505 numeric  0 -2.694869e-02 0.9508940 -0.2423187660 0.6352597 -1.9394482
## 507 x.506 numeric  0  1.367559e-02 0.9653745 -0.0030736295 0.9173329 -3.2254518
## 508 x.507 numeric  0 -7.174293e-02 0.9867564 -0.1463769580 0.8717619 -3.0809992
## 509 x.508 numeric  0 -1.737391e-02 0.9316135 -0.0420563180 0.8908629 -2.4373833
## 510 x.509 numeric  0 -4.526441e-02 0.9247705 -0.1709746690 0.9030399 -2.3414619
## 511 x.510 numeric  0  2.576428e-02 0.9763687  0.0390568150 0.9979694 -3.0248012
## 512 x.511 numeric  0 -4.434930e-03 0.9985056 -0.0726717645 0.9462139 -2.4680287
## 513 x.512 numeric  0 -2.625645e-02 0.9946037 -0.0395847825 0.9948780 -2.6520612
## 514 x.513 numeric  0 -1.670506e-02 1.0192533 -0.0861976505 0.9525597 -2.9419271
## 515 x.514 numeric  0  3.694699e-03 1.0135321 -0.0161921475 0.9056765 -2.9291466
## 516 x.515 numeric  0 -5.550639e-03 0.9214668 -0.0981947390 0.6545396 -2.5752738
## 517 x.516 numeric  0 -2.913796e-02 1.0144081 -0.0177108240 0.9238239 -2.6674991
## 518 x.517 numeric  0 -2.718342e-02 0.9928029 -0.0482496905 0.9343411 -2.4193809
## 519 x.518 numeric  0  5.977731e-02 1.0803643 -0.0551932125 0.6814781 -2.3158074
## 520 x.519 numeric  0  3.029478e-02 0.9750468 -0.0306342095 0.9172413 -2.6293344
## 521 x.520 numeric  0  2.169988e-02 0.9777718 -0.0232764220 0.9706183 -2.8740262
## 522 x.521 numeric  0 -2.525885e-02 0.9747717 -0.1409986950 0.9706908 -2.5138912
## 523 x.522 numeric  0  2.823864e-02 1.0352564  0.1867038630 0.9916084 -4.6148623
## 524 x.523 numeric  0  3.342920e-02 1.0213541 -0.0527200795 0.9149725 -2.2902016
## 525 x.524 numeric  0  1.584299e-02 0.9954040 -0.0473542675 0.9496468 -3.8698324
## 526 x.525 numeric  0  3.844169e-02 0.9516834  0.0187243960 0.8096542 -2.3943359
## 527 x.526 numeric  0 -5.553611e-02 1.0230077 -0.1029044780 0.9950056 -3.0570625
## 528 x.527 numeric  0 -1.176498e-02 0.9619134 -0.1127436320 0.8834703 -2.5462596
## 529 x.528 numeric  0  2.654134e-02 0.9481370 -0.0447440765 0.8784890 -2.3035348
## 530 x.529 numeric  0  6.246538e-02 0.9912280  0.0176673440 0.9787154 -3.4629967
## 531 x.530 numeric  0  3.105393e-02 0.9395949 -0.0132662450 0.9196911 -2.9419417
## 532 x.531 numeric  0  2.279239e-02 1.0240703 -0.1134994465 0.9148641 -2.2340378
## 533 x.532 numeric  0 -7.906246e-03 0.9879031 -0.1168699145 0.9040340 -2.2524269
## 534 x.533 numeric  0  4.768565e-02 0.9764810  0.0621034530 1.0238450 -3.3788979
## 535 x.534 numeric  0  3.080238e-02 0.9785415  0.0473362625 0.8563543 -3.0743286
## 536 x.535 numeric  0 -3.964450e-03 0.9862690 -0.0381349005 0.8210445 -3.1029287
## 537 x.536 numeric  0  1.394683e-02 0.9999425 -0.0714337055 1.0327864 -2.5007362
## 538 x.537 numeric  0  4.547444e-03 1.0314780 -0.3355910885 0.2871356 -0.8030868
## 539 x.538 numeric  0  1.971869e-02 1.0237546 -0.0633948790 1.0396908 -2.6766651
## 540 x.539 numeric  0  4.081047e-02 0.9900892  0.0404701110 0.9495311 -3.0458033
## 541 x.540 numeric  0  5.220966e-03 1.0412931 -0.0762067390 0.9464620 -3.3470463
## 542 x.541 numeric  0 -5.042373e-02 0.9664468 -0.0723519845 0.9170605 -2.9149500
## 543 x.542 numeric  0 -4.439529e-02 0.9763367 -0.1103518875 0.9742301 -2.7770516
## 544 x.543 numeric  0 -1.005221e-02 1.0037034  0.0056892000 0.9630296 -2.7706537
## 545 x.544 numeric  0 -3.483312e-03 0.9936064 -0.0063215200 0.9330836 -2.6034497
## 546 x.545 numeric  0 -5.585820e-03 0.9834570 -0.0932551410 0.9454992 -2.2841855
## 547 x.546 numeric  0 -3.276352e-03 0.9687825 -0.0587713850 0.9472253 -2.6497651
## 548 x.547 numeric  0 -2.986611e-02 1.0539502 -0.0092311205 0.9902163 -3.2654962
## 549 x.548 numeric  0  1.323262e-02 0.9396607  0.0252039490 0.9140918 -3.0785389
## 550 x.549 numeric  0  2.102958e-03 0.9637489 -0.0206461370 0.8737029 -2.9879187
## 551 x.550 numeric  0  2.712234e-02 1.0461933 -0.0225204845 0.9699136 -2.5993738
## 552 x.551 numeric  0 -1.972744e-03 1.0028297 -0.0462510485 0.9377145 -2.7867885
## 553 x.552 numeric  0 -7.499342e-03 0.9926879 -0.2205101610 0.7420834 -1.9513300
## 554 x.553 numeric  0  6.327235e-02 0.9789106  0.1059948450 0.7548319 -3.5804900
## 555 x.554 numeric  0 -5.325708e-02 0.9469213 -0.2148056245 0.8547915 -2.4399774
## 556 x.555 numeric  0 -3.548092e-03 0.9729567 -0.0504279710 0.9245531 -2.5957785
## 557 x.556 numeric  0  1.598406e-02 0.9954086 -0.0132138180 0.9744731 -2.5413705
## 558 x.557 numeric  0  4.846238e-02 1.0455239 -0.3536939945 0.5548112 -1.0006441
## 559 x.558 numeric  0 -5.496734e-02 0.9855361 -0.0579579935 0.8963408 -3.4007698
## 560 x.559 numeric  0 -1.211785e-02 0.9548547 -0.1302293385 0.8294369 -2.0588751
## 561 x.560 numeric  0  4.191111e-02 1.0287144  0.0624725360 0.9617760 -2.9937422
## 562 x.561 numeric  0  3.794759e-02 0.9535755 -0.0548367975 0.7974113 -2.2945834
## 563 x.562 numeric  0  2.879300e-02 1.0399735 -0.2942988435 0.4661656 -1.1473049
## 564 x.563 numeric  0  2.188029e-02 0.9923347  0.0652794925 0.8691341 -3.4412281
## 565 x.564 numeric  0 -2.750065e-02 1.0142056 -0.0217566080 0.8505609 -2.7885618
## 566 x.565 numeric  0  7.415552e-03 1.0403997 -0.1909536410 1.0118576 -2.2670114
## 567 x.566 numeric  0  1.128481e-02 1.0338057 -0.0642650700 0.9762301 -2.7891608
## 568 x.567 numeric  0  3.647535e-02 1.0010784  0.0279456410 0.9153601 -2.6064916
## 569 x.568 numeric  0 -4.616104e-03 0.9920345 -0.0574129050 0.9275370 -2.9823847
## 570 x.569 numeric  0  1.056894e-02 1.0341973 -0.1467690920 0.8281262 -2.6585506
## 571 x.570 numeric  0 -2.076083e-02 0.9893009  0.0109970655 0.9999117 -2.5763157
## 572 x.571 numeric  0 -2.326988e-02 0.9879821 -0.0303166930 1.0039047 -2.7080670
## 573 x.572 numeric  0 -1.415086e-02 0.8149116 -0.0877609325 0.6080865 -3.0736796
## 574 x.573 numeric  0  2.762501e-03 0.9882133  0.0672651085 0.9783610 -2.4059022
## 575 x.574 numeric  0  1.873922e-02 1.0597570 -0.1467541010 0.8799700 -2.2431907
## 576 x.575 numeric  0 -3.891740e-02 1.0129371 -0.0922145630 0.9449730 -2.4217102
## 577 x.576 numeric  0  3.244784e-02 1.0205080  0.0302024975 0.9719645 -3.0165688
## 578 x.577 numeric  0  3.366913e-02 1.0425208 -0.1861247825 0.4621617 -1.4970633
## 579 x.578 numeric  0 -1.232610e-02 0.9632018 -0.0604221740 0.9450043 -2.3720707
## 580 x.579 numeric  0 -1.211114e-02 0.9728442 -0.0853132585 0.9815911 -1.9396686
## 581 x.580 numeric  0  5.937385e-02 0.9988214 -0.0058393920 0.9804308 -2.6472251
## 582 x.581 numeric  0 -5.777760e-03 1.0167444 -0.0790216115 0.8797593 -3.2529309
## 583 x.582 numeric  0  1.013980e-02 1.0070173 -0.0553174195 0.9191453 -3.0580380
## 584 x.583 numeric  0 -2.759429e-03 1.0115727 -0.0817031060 0.9886881 -2.5107651
## 585 x.584 numeric  0  1.050834e-02 1.1180114 -0.1172050625 0.8386977 -3.1071284
## 586 x.585 numeric  0 -5.707167e-02 0.9988654 -0.0836336565 0.8892425 -3.0136801
## 587 x.586 numeric  0 -1.907369e-03 1.0130185 -0.0673790670 0.9142323 -2.9123237
## 588 x.587 numeric  0 -1.604153e-02 0.9636224 -0.0301332265 1.0033436 -3.3330512
## 589 x.588 numeric  0 -2.192985e-02 1.0216606 -0.0029947655 1.1765994 -2.4731024
## 590 x.589 numeric  0 -1.807072e-02 0.9636136 -0.1107832490 0.8832316 -1.9724477
## 591 x.590 numeric  0  9.523741e-02 1.0027092  0.0814432700 0.9495949 -4.0962470
## 592 x.591 numeric  0 -5.973318e-02 0.9812291 -0.0516327605 1.0486355 -3.0890238
## 593 x.592 numeric  0  5.972640e-02 0.9920311  0.0484866435 1.0095878 -3.1208782
## 594 x.593 numeric  0  2.697319e-02 0.9655499 -0.0006373075 0.8494731 -3.3239076
## 595 x.594 numeric  0  6.420324e-03 1.0175798 -0.0592776730 0.9660567 -2.3972450
## 596 x.595 numeric  0 -4.495463e-03 1.0035979 -0.0503503480 0.8912811 -2.4087870
## 597 x.596 numeric  0 -5.219253e-02 0.7285698 -0.1103539940 0.6286406 -2.4886454
## 598 x.597 numeric  0 -5.627358e-03 0.9908854 -0.1331443150 0.9166230 -2.6994764
## 599 x.598 numeric  0 -1.443061e-02 0.8503276 -0.0232349405 0.7628863 -3.1168814
## 600 x.599 numeric  0  5.622844e-02 1.0228311 -0.5495346385 0.2024065 -0.8757221
## 601 x.600 numeric  0 -9.232946e-03 0.9840656 -0.1143282215 0.9748013 -2.5317329
##           max nlevs
## 1    1.000000     0
## 2    3.405338     0
## 3    3.053096     0
## 4    3.106502     0
## 5    3.817299     0
## 6    3.753474     0
## 7    4.075355     0
## 8    3.827203     0
## 9    5.356366     0
## 10   5.821427     0
## 11   6.315310     0
## 12   3.794283     0
## 13   2.927135     0
## 14   2.810559     0
## 15   4.047613     0
## 16   6.808566     0
## 17   4.006391     0
## 18   7.484983     0
## 19   4.546598     0
## 20   3.231152     0
## 21  13.602102     0
## 22   5.381414     0
## 23   3.160485     0
## 24   3.022282     0
## 25   4.468797     0
## 26   3.519138     0
## 27   3.278754     0
## 28   4.444245     0
## 29   3.115720     0
## 30   2.055149     0
## 31   2.539371     0
## 32   2.823619     0
## 33   6.136544     0
## 34   3.565081     0
## 35   4.960171     0
## 36   9.619010     0
## 37   3.017535     0
## 38  11.067376     0
## 39   2.643794     0
## 40   3.611898     0
## 41   5.816493     0
## 42   2.692862     0
## 43   2.694280     0
## 44   2.703413     0
## 45   3.479685     0
## 46   4.848923     0
## 47   3.821112     0
## 48   4.583715     0
## 49   3.487524     0
## 50   2.982982     0
## 51  16.424168     0
## 52   3.181193     0
## 53   3.634097     0
## 54   3.269815     0
## 55   2.661460     0
## 56   3.496550     0
## 57   3.359345     0
## 58   3.139911     0
## 59   5.062061     0
## 60   5.342889     0
## 61   3.058575     0
## 62   9.314068     0
## 63   3.076492     0
## 64   3.486124     0
## 65   3.046260     0
## 66   3.708231     0
## 67   9.600819     0
## 68   2.957707     0
## 69   2.981761     0
## 70   3.602941     0
## 71   3.651839     0
## 72   4.607624     0
## 73   3.200848     0
## 74   3.089077     0
## 75   3.200232     0
## 76   2.841355     0
## 77   3.064687     0
## 78   4.023834     0
## 79   7.593406     0
## 80   2.760779     0
## 81   8.197150     0
## 82   4.777185     0
## 83   3.305518     0
## 84   3.781723     0
## 85   2.492796     0
## 86   5.711016     0
## 87   3.179744     0
## 88   7.757577     0
## 89   3.033288     0
## 90   4.252139     0
## 91   3.641611     0
## 92   3.129524     0
## 93   5.189343     0
## 94   2.553863     0
## 95   2.612850     0
## 96   5.404766     0
## 97   3.215169     0
## 98   6.420463     0
## 99   4.097400     0
## 100  3.658643     0
## 101  3.538650     0
## 102  5.552321     0
## 103 20.410774     0
## 104  3.819905     0
## 105  4.012333     0
## 106  2.808321     0
## 107  4.402198     0
## 108  3.396774     0
## 109  6.165225     0
## 110  4.189148     0
## 111  3.878870     0
## 112  6.180580     0
## 113  3.689854     0
## 114  3.368937     0
## 115  4.885655     0
## 116  2.915792     0
## 117  3.008177     0
## 118  4.576165     0
## 119  2.761133     0
## 120  4.385635     0
## 121  3.754276     0
## 122  2.825079     0
## 123  3.332440     0
## 124  5.904586     0
## 125  3.047654     0
## 126  4.831880     0
## 127  3.499133     0
## 128  3.190583     0
## 129  5.014877     0
## 130  3.512933     0
## 131  3.260169     0
## 132  4.176364     0
## 133  3.419604     0
## 134  3.700144     0
## 135  3.083968     0
## 136  2.844407     0
## 137  2.639983     0
## 138  4.389908     0
## 139  9.050458     0
## 140  2.925877     0
## 141  5.328359     0
## 142  5.262108     0
## 143  7.433263     0
## 144  3.493674     0
## 145  3.001449     0
## 146  6.586595     0
## 147  3.585290     0
## 148  4.474224     0
## 149  3.488179     0
## 150  5.511919     0
## 151  4.196907     0
## 152  3.693028     0
## 153  3.572530     0
## 154  5.665315     0
## 155  4.403270     0
## 156  2.959360     0
## 157  4.255031     0
## 158  4.348712     0
## 159  2.962395     0
## 160  3.686679     0
## 161  3.780190     0
## 162  5.281297     0
## 163  4.535048     0
## 164  2.730726     0
## 165  4.100537     0
## 166  4.346881     0
## 167  4.811176     0
## 168  4.277483     0
## 169  3.143571     0
## 170  4.448999     0
## 171  3.641459     0
## 172  4.611558     0
## 173  3.714022     0
## 174  4.390323     0
## 175  3.649350     0
## 176  4.496485     0
## 177  2.924688     0
## 178 10.202456     0
## 179  3.078026     0
## 180  3.848769     0
## 181  7.273246     0
## 182  4.340827     0
## 183  3.467193     0
## 184  3.685328     0
## 185  4.516614     0
## 186  4.697344     0
## 187  5.956819     0
## 188 15.763174     0
## 189  5.767786     0
## 190  4.062353     0
## 191  3.366894     0
## 192  3.636413     0
## 193  2.446825     0
## 194  6.910530     0
## 195  3.087591     0
## 196  4.276420     0
## 197  3.354153     0
## 198  6.801751     0
## 199  6.772753     0
## 200  5.343823     0
## 201  3.891927     0
## 202  2.607814     0
## 203  3.626029     0
## 204  2.533342     0
## 205  3.524968     0
## 206  3.362662     0
## 207  3.641469     0
## 208  2.826085     0
## 209  4.696607     0
## 210  8.519056     0
## 211  3.646317     0
## 212  4.162576     0
## 213  3.749972     0
## 214  4.867942     0
## 215  5.446345     0
## 216  5.814850     0
## 217  3.835546     0
## 218  3.766231     0
## 219  4.325584     0
## 220  2.421515     0
## 221  3.367755     0
## 222  3.437862     0
## 223  3.630006     0
## 224  2.880769     0
## 225  3.601460     0
## 226  3.615434     0
## 227  3.226523     0
## 228  3.880334     0
## 229  3.661311     0
## 230  4.768772     0
## 231  3.895432     0
## 232  2.691090     0
## 233  4.584714     0
## 234 14.266098     0
## 235  3.605349     0
## 236  3.487931     0
## 237  5.612053     0
## 238  5.461723     0
## 239  2.639137     0
## 240  4.317352     0
## 241  5.300414     0
## 242  2.807430     0
## 243  3.346291     0
## 244  3.539974     0
## 245  5.126335     0
## 246  3.124227     0
## 247  3.743128     0
## 248  3.128662     0
## 249  3.794947     0
## 250  3.283536     0
## 251  3.601344     0
## 252  3.896391     0
## 253  3.660907     0
## 254  2.625088     0
## 255  3.034480     0
## 256  3.995533     0
## 257 10.612544     0
## 258  4.376119     0
## 259  3.200966     0
## 260  2.997815     0
## 261  3.500493     0
## 262  9.916976     0
## 263  4.220573     0
## 264 11.710975     0
## 265  4.675938     0
## 266  3.603238     0
## 267  3.575827     0
## 268  2.803941     0
## 269  2.646574     0
## 270  3.654371     0
## 271  3.181291     0
## 272  3.424248     0
## 273  2.889167     0
## 274  8.139790     0
## 275 12.160726     0
## 276  8.475681     0
## 277  3.308632     0
## 278 10.786254     0
## 279  6.444865     0
## 280  3.054061     0
## 281  5.304473     0
## 282  4.910612     0
## 283  4.978595     0
## 284  4.796107     0
## 285  4.275668     0
## 286  4.632694     0
## 287  3.056635     0
## 288  2.995447     0
## 289  6.729809     0
## 290  3.544322     0
## 291  3.210295     0
## 292  3.263034     0
## 293  4.269719     0
## 294  3.581069     0
## 295  4.241355     0
## 296  5.216367     0
## 297  2.875818     0
## 298  4.750493     0
## 299  2.899930     0
## 300  2.862854     0
## 301  6.105533     0
## 302  4.601167     0
## 303  6.519165     0
## 304  9.498959     0
## 305  3.420404     0
## 306  3.714265     0
## 307  4.288745     0
## 308  4.586735     0
## 309  5.631569     0
## 310  7.612364     0
## 311  6.073219     0
## 312  3.069542     0
## 313  2.803624     0
## 314  2.891021     0
## 315  4.122002     0
## 316  4.128224     0
## 317  4.574411     0
## 318  2.470161     0
## 319  4.266267     0
## 320  8.484855     0
## 321  2.773670     0
## 322  3.769945     0
## 323  5.708089     0
## 324  5.615717     0
## 325  3.110096     0
## 326  3.952297     0
## 327  3.872354     0
## 328  3.721465     0
## 329  4.278396     0
## 330  5.374263     0
## 331  3.618064     0
## 332  2.762954     0
## 333  4.503319     0
## 334  2.809100     0
## 335  2.888848     0
## 336  4.046251     0
## 337  3.103474     0
## 338  4.327998     0
## 339  3.610198     0
## 340  4.800890     0
## 341  2.846773     0
## 342  4.340313     0
## 343  4.748291     0
## 344  8.392635     0
## 345  3.954438     0
## 346  3.100373     0
## 347  5.060137     0
## 348  6.916204     0
## 349  3.747088     0
## 350  4.093639     0
## 351  4.027724     0
## 352  7.373306     0
## 353  3.271836     0
## 354  4.069728     0
## 355  2.467557     0
## 356  6.252751     0
## 357  2.841317     0
## 358  4.423664     0
## 359  2.762717     0
## 360  4.114760     0
## 361  3.100594     0
## 362  4.054644     0
## 363  3.193924     0
## 364  3.149393     0
## 365  3.435380     0
## 366  9.435914     0
## 367  4.991710     0
## 368  3.238784     0
## 369  3.690248     0
## 370  4.853519     0
## 371  9.103245     0
## 372  4.228186     0
## 373  4.032240     0
## 374  3.745975     0
## 375  4.310016     0
## 376  6.742122     0
## 377  2.442892     0
## 378  2.200035     0
## 379  3.001812     0
## 380  3.436195     0
## 381  3.171186     0
## 382  4.543279     0
## 383  4.777891     0
## 384  5.341861     0
## 385  2.590807     0
## 386  3.760341     0
## 387  3.985894     0
## 388  6.239272     0
## 389  4.161109     0
## 390  5.743783     0
## 391  3.496730     0
## 392  3.803599     0
## 393  3.741142     0
## 394  2.623360     0
## 395  4.626612     0
## 396  3.042423     0
## 397  2.825178     0
## 398  6.290733     0
## 399  3.020428     0
## 400  4.202898     0
## 401  5.363951     0
## 402  5.361361     0
## 403  8.712179     0
## 404  3.008398     0
## 405  3.525988     0
## 406  3.815690     0
## 407  2.508471     0
## 408  4.025797     0
## 409  3.893250     0
## 410  4.460170     0
## 411  3.657496     0
## 412  3.596769     0
## 413  4.993972     0
## 414  3.443358     0
## 415  4.013376     0
## 416  4.796335     0
## 417  6.731300     0
## 418  3.739003     0
## 419  8.292230     0
## 420  3.856942     0
## 421  4.436876     0
## 422  3.639994     0
## 423  3.484106     0
## 424  3.232065     0
## 425  4.474783     0
## 426  6.218200     0
## 427  5.964947     0
## 428  2.779896     0
## 429  2.947645     0
## 430  5.580927     0
## 431  3.013806     0
## 432  4.760198     0
## 433  3.677177     0
## 434  3.068925     0
## 435  5.183571     0
## 436  3.629043     0
## 437  2.760990     0
## 438  3.899418     0
## 439  4.148125     0
## 440  3.049045     0
## 441  2.991373     0
## 442  3.487334     0
## 443  9.748833     0
## 444  3.973526     0
## 445  4.426379     0
## 446  3.970593     0
## 447  3.375224     0
## 448  3.119912     0
## 449  5.939377     0
## 450  3.631623     0
## 451  8.734836     0
## 452  6.627963     0
## 453  4.737978     0
## 454  3.584597     0
## 455  4.112793     0
## 456  4.755982     0
## 457  3.656443     0
## 458  9.078022     0
## 459  2.587313     0
## 460  3.351627     0
## 461  4.786515     0
## 462  3.308294     0
## 463  2.654439     0
## 464  3.998787     0
## 465  4.275738     0
## 466  2.624025     0
## 467  3.980438     0
## 468  4.094985     0
## 469  5.261845     0
## 470  2.878162     0
## 471  5.216901     0
## 472  3.406075     0
## 473  4.311842     0
## 474  2.681645     0
## 475  2.554341     0
## 476  6.253616     0
## 477  3.798546     0
## 478  3.375225     0
## 479  5.554666     0
## 480  3.772804     0
## 481  4.052821     0
## 482  4.048065     0
## 483  3.974929     0
## 484  4.377572     0
## 485  4.329727     0
## 486  3.260012     0
## 487  4.421395     0
## 488  3.708354     0
## 489  5.098128     0
## 490  3.545829     0
## 491  2.686667     0
## 492  2.845629     0
## 493  7.472933     0
## 494  4.214023     0
## 495  2.230396     0
## 496  3.994369     0
## 497  4.481448     0
## 498  4.849163     0
## 499  5.609675     0
## 500  4.184415     0
## 501  6.802091     0
## 502  3.052985     0
## 503  3.434527     0
## 504  3.776598     0
## 505  3.526889     0
## 506  4.853180     0
## 507  3.666125     0
## 508  4.773823     0
## 509  3.644836     0
## 510  3.278291     0
## 511  2.711156     0
## 512  4.105982     0
## 513  3.321315     0
## 514  3.676696     0
## 515  4.186457     0
## 516  6.791565     0
## 517  6.459023     0
## 518  3.265109     0
## 519 13.402831     0
## 520  6.621832     0
## 521  3.281357     0
## 522  2.762842     0
## 523  2.087610     0
## 524  5.428772     0
## 525  3.692877     0
## 526  4.095244     0
## 527  4.752815     0
## 528  4.355448     0
## 529  4.710025     0
## 530  2.647944     0
## 531  3.483943     0
## 532  5.100551     0
## 533  3.829684     0
## 534  3.169745     0
## 535  5.329203     0
## 536  6.862561     0
## 537  3.352589     0
## 538  5.842709     0
## 539  3.560436     0
## 540  3.427147     0
## 541  5.859043     0
## 542  3.181024     0
## 543  2.478951     0
## 544  4.154685     0
## 545  3.544290     0
## 546  3.199069     0
## 547  4.414846     0
## 548  3.324828     0
## 549  3.861518     0
## 550  3.358707     0
## 551  4.280056     0
## 552  3.051253     0
## 553  5.202116     0
## 554  4.546009     0
## 555  3.955858     0
## 556  3.983870     0
## 557  3.375061     0
## 558  3.778807     0
## 559  2.969502     0
## 560  3.831368     0
## 561  4.418063     0
## 562  5.992268     0
## 563  6.792404     0
## 564  4.590817     0
## 565  9.056784     0
## 566  4.236531     0
## 567  5.667616     0
## 568  6.438350     0
## 569  3.637305     0
## 570  5.932968     0
## 571  2.991598     0
## 572  3.812865     0
## 573  7.871876     0
## 574  3.849024     0
## 575  4.837906     0
## 576  3.497418     0
## 577  4.333977     0
## 578  7.524825     0
## 579  3.475439     0
## 580  4.853153     0
## 581  3.424427     0
## 582  8.430263     0
## 583  3.853778     0
## 584  3.408512     0
## 585  8.379760     0
## 586  4.339116     0
## 587  3.483645     0
## 588  2.863382     0
## 589  2.617994     0
## 590  3.749782     0
## 591  3.396970     0
## 592  2.716243     0
## 593  3.337366     0
## 594  2.795472     0
## 595  3.882242     0
## 596  5.281380     0
## 597  3.195546     0
## 598  4.839594     0
## 599  2.532411     0
## 600  2.472096     0
## 601  3.252795     0
# dataX = dataX[complete.cases(dataX), ] No missing values are found

# define the task (positive = 1)
dataX_tsk = makeClassifTask(data = dataX, target = "y", positive = 1)

# split data into train/test
split_desc = makeResampleDesc(method = "Holdout", stratify = TRUE, split = 0.7)
set.seed(202)
split = makeResampleInstance(split_desc, task = dataX_tsk)
train = split$train.inds[[1]]; test = split$test.inds[[1]]
table(dataX[train,]$y)
## 
##   0   1 
## 235  79
### Support vector machine

# tune parameters for svm models
ps_extended = makeParamSet(makeDiscreteParam("kernel", values = c("vanilladot", "polydot", "rbfdot")), 
                           makeDiscreteParam("C", values = seq(1e-06, 5, length = 10)), 
                           makeDiscreteParam("sigma", values = c(0.001, 0.01, 0.1), 
                                             requires = quote(kernel == "rbfdot")),
                           makeIntegerParam("degree", lower = 1L, upper = 5L, 
                                            requires = quote(kernel == "polydot")))
set.seed(202)
cv5_stratified = makeResampleDesc("CV", iters = 5, stratify = TRUE)
ctrl = makeTuneControlIrace(maxExperiments = 200L)
svm.lrn.multi = makeLearner("classif.ksvm", predict.type = "prob")
tune_svm = tuneParams(svm.lrn.multi, subsetTask(dataX_tsk, train),
                      cv5_stratified, measures = list(auc), 
                      par.set = ps_extended, control = ctrl,
                      show.info = FALSE)

# find the optimal parameters 
svm.lrn_tuned = setHyperPars(makeLearner("classif.ksvm", predict.type = "prob"),
                             par.vals = tune_svm$x)
svm.lrn_tuned #rdfdot kernel (radial basis function) with C = 5 and sigma = 0.001 is chosen for the optimal parameters
## Learner classif.ksvm from package kernlab
## Type: classif
## Name: Support Vector Machines; Short name: ksvm
## Class: classif.ksvm
## Properties: twoclass,multiclass,numerics,factors,prob,class.weights
## Predict-Type: prob
## Hyperparameters: fit=FALSE,kernel=rbfdot,C=5,sigma=0.001
suppressMessages(library(glmnet))

### LASSO regression analysis

lasso.lnr =  makeLearner("classif.cvglmnet", fix.factors.prediction = TRUE, 
                         predict.type = "prob", alpha=1, type.measure='auc')

# choose the optimal lambda (which maximizes the cross-validated auc)
set.seed(202)
dataX.lasso = train(learner = lasso.lnr, task = dataX_tsk, subset=train)
ls.lambda.min = dataX.lasso$learner.model$lambda.min

### Ridge regression analysis
ridge.lnr = makeLearner("classif.cvglmnet", fix.factors.prediction = TRUE, 
                        predict.type = "prob", alpha = 0, type.measure = 'auc')

# Choose the optimal lambda (which maximizes the cross-validated auc)
set.seed(202)
dataX.ridge = train(learner = ridge.lnr, task = dataX_tsk, subset=train)
rg.lambda.min = dataX.ridge$learner.model$lambda.min
## compare CV-auc of the models
lrns = list(
  makeLearner("classif.ksvm", par.vals = list(kernel="rbfdot", C=5, sigma=0.001), predict.type = "prob", id = "svm"),
  makeLearner("classif.glmnet", lambda=ls.lambda.min, fix.factors.prediction = TRUE, 
                                   predict.type = "prob", alpha=1, id = "lasso"),
  makeLearner("classif.glmnet", lambda=rg.lambda.min, fix.factors.prediction = TRUE, 
                                   predict.type = "prob", alpha=0, id = "ridge")
)

tasks = list(dataX_tsk)
rdesc = makeResampleDesc("CV", iters = 5)
meas = list(auc, timetrain)
set.seed(202)
bmr = benchmark(lrns, tasks, rdesc, meas, show.info = FALSE)
perf = getBMRPerformances(bmr, as.df = TRUE)
aggregate(perf[, 4:5], list(perf$learner.id), mean)
##   Group.1       auc timetrain
## 1   lasso 0.9101598     0.032
## 2   ridge 0.6713624     0.016
## 3     svm 0.7350182     0.380
  • Given the size of data, cross-validation AUC is used for comparing the prediction performance of the models (Support vector machine, LASSO, and Ridge). LASSO is chosen as it shows the highest cross-validated AUC as well as low computational costs among these models. The LASSO model was trained with lambda min, which gives the highest cross-validated AUC. The results indicate that the true relationship between the features and reoccurence is likely to be linear and feature selection is preferable. The CV AUC (an averaged result from 5 folds of cross-validation) for the LASSO model is 0.9101.
  1. (13 points) Plot the test ROC curve and report the following: i) test AUC for the model selected, ii) the best sensitivity achievable if at least 50% specificity is required, iii) the best sensitivity achievable if at least 80% specificity is required, iv) the best specificity achievable if at least 90% sensitivity is required. Note: approximate values read off from the ROC curve are ok for i)-iv). Explain the risks associated with low sensitivity and with low specificity for predicting recurrence of cancer X.
# predict on the test set
set.seed(202)
lasso.lambda.min.lnr = makeLearner("classif.glmnet", lambda=ls.lambda.min, fix.factors.prediction = TRUE, 
                                   predict.type = "prob", alpha=1)
lasso_fit = train(learner = lasso.lambda.min.lnr, task = dataX_tsk) 
lasso_predict = predict(lasso_fit, newdata = dataX[test,], type = "prob")
calculateROCMeasures(lasso_predict)
##     predicted
## true 0         1                            
##    0 100       1         tpr: 0.6  fnr: 0.4 
##    1 14        21        fpr: 0.01 tnr: 0.99
##      ppv: 0.95 for: 0.12 lrp: 60.6 acc: 0.89
##      fdr: 0.05 npv: 0.88 lrm: 0.4  dor: 150 
## 
## 
## Abbreviations:
## tpr - True positive rate (Sensitivity, Recall)
## fpr - False positive rate (Fall-out)
## fnr - False negative rate (Miss rate)
## tnr - True negative rate (Specificity)
## ppv - Positive predictive value (Precision)
## for - False omission rate
## lrp - Positive likelihood ratio (LR+)
## fdr - False discovery rate
## npv - Negative predictive value
## acc - Accuracy
## lrm - Negative likelihood ratio (LR-)
## dor - Diagnostic odds ratio
lasso.auc = performance(lasso_predict, measures = auc)
print(paste("The test AUC of LASSO model is", round(lasso.auc, 4)))
## [1] "The test AUC of LASSO model is 0.9024"
df = generateThreshVsPerfData(list(lasso = lasso_predict), measures = list(fpr, tpr))
plotROCCurves(df)

data.frame(model = c("lasso"), test.auc = c(lasso.auc))
##     model  test.auc
## auc lasso 0.9024045
## subset expression features 
df.var.ls = data.frame(variable = c(names(dataX[, -1])), 
                       coefficient = ifelse(lasso_fit$learner.model$beta != 0, 
                                         as.vector(lasso_fit$learner.model$beta), 0))
df.imp.ls = df.var.ls[order(abs(df.var.ls$coefficient), decreasing = T),]
df.imp.ls = subset(df.imp.ls, coefficient !=0)
df.imp.ls
##     variable coefficient
## 301    x.301  1.94579869
## 30      x.30 -0.26127418
## 226    x.226  0.23704172
## 579    x.579  0.14898400
## 466    x.466 -0.03603050
## 130    x.130  0.02900232
## 295    x.295 -0.02639734
## 497    x.497  0.01775497
## 214    x.214 -0.01465742
  • The test AUC for LASSO is 0.9024. Given the ROC curve ploted above, the best sensitivity achievable if at least 50% specificity is required is around 0. 875. The best specificity achievable if at least 90% sensitivity is required is a little less than 0.875. The Risks associated with low sensitivity lead to a failure of designating patients with reoccurence as positive (i.e. reoccurence groups). The Risks associated with low specificity lead to a failure of designating patients without reoccurence as negative (i.e. non-reoccurence groups).
  1. (11 points) Predict the probability of cancer recurrence for each of the new patients in the file newdataX.csv and their predicted recurrence status using i) a probability cutoff of 0.5 and ii) a probability cutoff of 0.7. Which set of predictions will have higher sensitivity/specificity? Explain.
newdataX = read.csv("newdataX.csv")
dim(newdataX)
## [1]  10 600
# cutoff of 0.5
set.seed(202)
new_predict = predict(lasso_fit, newdata = newdataX)
pred_prob = getPredictionProbabilities(new_predict)
pred_prob
##  [1] 0.093279380 0.880726716 0.955102838 0.007639901 0.804855268 0.928241579
##  [7] 0.029532312 0.059713314 0.943627370 0.945299762
cutoff5 = ifelse(pred_prob >= 0.5, 1, 0)

# cutoff of 0.7
cutoff7 = ifelse(pred_prob >= 0.7, 1, 0)

data.frame(new.patient.id = c(1:10), cutoff5, cutoff7)
##    new.patient.id cutoff5 cutoff7
## 1               1       0       0
## 2               2       1       1
## 3               3       1       1
## 4               4       0       0
## 5               5       1       1
## 6               6       1       1
## 7               7       0       0
## 8               8       0       0
## 9               9       1       1
## 10             10       1       1
  • In this case, there was no change between setting the cutoff of 0.5 and 0.7. However, increasing the cutoff for classification generally leads to a higher sensitivity but lower specificity.


2. (15 points total. 5 points each).
Choose the best answer for each scenario below (there could be more than one best choice) and provide a short explanation for your choice/s. The explanation is the most important part of the answer and any partial credit will be based on it.
  1. Your regression model has a training \(R^2\) = 0.75 and a test \(R^2\) = 0.56. You conclude that your model: a) underfits, b) overfits, c) has high bias, d) has high variance, e) Can’t tell based on this information alone.
    1. test \(R^2\) or any other measures for test set is usually higher than training \(R^2\) or relevant performance measures for training set. Thus, we cannot conclude that the model unverfits or overfits solely depending on the given information.
  1. You are developing a prediction model for a quantitative outcome based on a dataset with n = 100,000 observations and 50 quantitative features. You compare standard linear regression with LASSO linear regression and find that the standard linear regression model yields the lowest cross-validation MSE. If you want to further improve prediction, which model/s among the following should you consider next? a) ridge regression, b) a linear regression with forward selection, c) Random Forest, d) a linear regression with polynomial terms in the features, or e) none of the above models is likely to yield further improvement.
    1. LASSO linear regression can perform poorly than the standard linear regression models when most of the features need to be used for prediction. Ridge regression analysis is likely to yield better performance as it does not drop features that are correlated with other features while controling for the model complexity.
  1. Your model overfits the data. Which course of action among the following would you recommend to improve performnace?: a) get more training data (assume that obtaining more data is feasible), b) use a more flexible model, c) use a less flexible model, d) perform forward feature selection, e) none of the above is likely to yield further improvement.
    1. Increasing the size of training data helps to reduce both the bias and variance. Likewise, using a more flexible or simpler model depends on the size of data. With less number of training data, simpler model is needed to prevent overfitting, while more complex model is prefered with enough number of data (lower bias).


3. (21 points total. 3 points each).
Decide whether each statement below is true or false and provide a short explanation. The explanation is the most important part of the answer and any partial credit will be based on it.
  1. Without pruning trees are likely to underfit.
  • False. Pruning is for preventing overfitting as classification/regression trees without pruning capture more noise rather than the trend from training data.
  1. Trees are pruned to reduce their bias.
  • True. Pruning also helps reducing the bias as more complex trees may overfit and increase the bias.
  1. Bagging improves upon single trees by reducing the variance.
  • True. Bagging reduces variance by averaging over several trees. With bagging, models are developed after averaging the results from bootstrapped data sets from the training set, thus improving the performance.
  1. Random Forest improves upon bagging by using less correlated tree ensembles.
  • True. Random Forest decorrelate trees by only using a few variables (m) rather than all possible variables (p) when splitting. While bagging considers all variables (m = p) for splitting on, Random Forest uses a random subset of all variables (m < p), thus reducing the use of highly correlated samples.
  1. The out-of-bag error for random forests is a computationally less expensive alternative to the cross-validation error.
  • True. Out-of-bag error is computed by averaging the prediction/majority vote over the samples that were trained but not used for that particular training. OOB error, unlike cross-validation error, needs no additional computation and faster.
  1. Principal component analysis guarantees a significant dimensionality reduction with a small loss of information.
  • False. Principal component analysis may reduce dimentionality by using only a few principle components that explain the variability of data well. Nonetheless, it does not guarantees a significant dimentionality reduction nor the prediction performance of these PCs. Information is lost when using PCs rather than the original variables.
  1. Unlike K-means, hierarchical clustering does not require the user to a-priori specify the number of desired clusters.
  • True. K-means methods require the user to choose the appropriate K (the number of clusters) before learning. However, hierarchical clustering does not specify K in advance the training and merges the most similar clusters from the bottom of the tree.