CBDA-SL and Knockoff Filter Results on the Null Dataset - n=300 p=1500, 9000 jobs x 8 experiments - MSE and Accuracy as ranking metrics for CBDA
Simeone Marino, Ivo Dinov, Jiachen Xu
Jun 25 2017
Some useful information
This is a summary of a set of 1 experiments using a LONI pipeline workflow file that performs 3000 independent jobs, each one with the CBDA-SL and the knockoff filter feature mining strategies. Each experiments has a total of 9000 jobs and is uniquely identified by 6 input arguments: # of jobs [M], % of missing values [misValperc], min [Kcol_min] and max [Kcol_max] % for FSR-Feature Sampling Range, min [Nrow_min] and max [Nrow_max] % for SSR-Subject Sampling Range.
This document has the final results, by experiment. See https://drive.google.com/file/d/0B5sz_T_1CNJQWmlsRTZEcjBEOEk/view?ths=true for some general documentation of the CBDA-SL project and github https://github.com/SOCR/CBDA for some of the code.
Features selected by both the knockoff filter and the CBDA-SL algorithms are shown as spikes in the histograms shown below. I list the top features selected, set to 15 here.
## [1] EXPERIMENT 1
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 1 5 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 1 9000 0 1 5 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "1"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 947 63 0.1294591 856 54 0.11355273 924 38 2.8400598
## 982 60 0.1232944 434 52 0.10934707 1174 28 2.0926756
## 492 57 0.1171297 1134 51 0.10724424 277 24 1.7937220
## 507 56 0.1150748 135 49 0.10303859 783 23 1.7189836
## 658 55 0.1130199 496 49 0.10303859 990 20 1.4947683
## 270 54 0.1109650 675 49 0.10303859 233 16 1.1958146
## 277 54 0.1109650 937 49 0.10303859 313 16 1.1958146
## 313 54 0.1109650 1082 49 0.10303859 1233 14 1.0463378
## 1039 54 0.1109650 1088 49 0.10303859 1248 14 1.0463378
## 182 53 0.1089101 1290 49 0.10303859 1486 14 1.0463378
## 38 52 0.1068552 454 48 0.10093576 207 13 0.9715994
## 615 51 0.1048003 1177 48 0.10093576 271 13 0.9715994
## 1396 51 0.1048003 1201 48 0.10093576 994 12 0.8968610
## 1044 50 0.1027454 1461 48 0.10093576 1142 12 0.8968610
## 1213 50 0.1027454 862 47 0.09883293 1351 12 0.8968610
##
##
##
##
##
##
## [1] EXPERIMENT 2
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 5 15 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 2 9000 0 5 15 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "2"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 277 154 0.10319917 335 140 0.08930736 924 21 3.047896
## 1277 140 0.09381743 1145 136 0.08675572 277 20 2.902758
## 510 136 0.09113693 1245 135 0.08611781 1174 19 2.757620
## 331 132 0.08845644 397 134 0.08547990 1142 13 1.886792
## 831 132 0.08845644 1176 134 0.08547990 990 12 1.741655
## 501 130 0.08711619 178 133 0.08484199 271 11 1.596517
## 405 129 0.08644606 284 132 0.08420408 994 10 1.451379
## 956 128 0.08577594 732 131 0.08356617 207 9 1.306241
## 1118 128 0.08577594 773 131 0.08356617 313 8 1.161103
## 1471 128 0.08577594 1328 131 0.08356617 584 8 1.161103
## 212 127 0.08510581 466 130 0.08292826 783 8 1.161103
## 328 127 0.08510581 659 130 0.08292826 1233 8 1.161103
## 827 127 0.08510581 718 129 0.08229035 1239 8 1.161103
## 1132 127 0.08510581 1430 129 0.08229035 38 7 1.015965
## 1372 127 0.08510581 302 128 0.08165244 233 7 1.015965
##
##
##
##
##
##
## [1] EXPERIMENT 5
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 5 15 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 5 9000 0 5 15 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "5"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 38 148 0.10016107 224 139 0.09152625 924 92 6.301370
## 1471 142 0.09610049 735 135 0.08889240 1174 77 5.273973
## 277 141 0.09542372 1115 135 0.08889240 783 60 4.109589
## 1326 137 0.09271667 490 132 0.08691701 277 56 3.835616
## 658 133 0.09000961 227 131 0.08625855 990 46 3.150685
## 1127 132 0.08933285 1158 131 0.08625855 1233 44 3.013699
## 1248 132 0.08933285 1185 130 0.08560009 510 38 2.602740
## 990 131 0.08865608 5 129 0.08494163 207 32 2.191781
## 81 129 0.08730255 16 129 0.08494163 584 31 2.123288
## 233 129 0.08730255 44 129 0.08494163 271 30 2.054795
## 1314 129 0.08730255 260 129 0.08494163 71 29 1.986301
## 1459 129 0.08730255 660 129 0.08494163 868 27 1.849315
## 572 128 0.08662579 447 128 0.08428317 1486 27 1.849315
## 688 128 0.08662579 477 128 0.08428317 1351 24 1.643836
## 1485 128 0.08662579 209 127 0.08362470 1326 23 1.575342
##
##
##
##
##
##
## [1] EXPERIMENT 7
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 1 5 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 7 9000 20 1 5 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "7"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 38 68 0.1430675 6 54 0.11472032 924 40 3.1620553
## 1039 60 0.1262361 676 52 0.11047142 277 24 1.8972332
## 277 57 0.1199243 1134 52 0.11047142 990 23 1.8181818
## 1280 54 0.1136125 284 51 0.10834697 207 18 1.4229249
## 313 52 0.1094046 396 51 0.10834697 1233 18 1.4229249
## 1248 52 0.1094046 503 50 0.10622251 1174 16 1.2648221
## 250 51 0.1073007 1075 50 0.10622251 783 15 1.1857708
## 261 51 0.1073007 327 48 0.10197361 868 14 1.1067194
## 983 51 0.1073007 789 48 0.10197361 38 13 1.0276680
## 990 51 0.1073007 1007 48 0.10197361 694 12 0.9486166
## 1277 51 0.1073007 546 47 0.09984916 1326 12 0.9486166
## 342 50 0.1051967 725 47 0.09984916 1351 12 0.9486166
## 483 50 0.1051967 29 46 0.09772471 1453 12 0.9486166
## 730 50 0.1051967 83 46 0.09772471 584 11 0.8695652
## 815 50 0.1051967 380 46 0.09772471 792 11 0.8695652
##
##
##
##
##
##
## [1] EXPERIMENT 11
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 5 15 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 11 9000 20 5 15 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "11"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 277 146 0.10028437 665 130 0.08805918 924 94 6.184211
## 1248 145 0.09959749 228 129 0.08738180 1174 77 5.065789
## 1295 142 0.09753685 1290 129 0.08738180 783 54 3.552632
## 119 140 0.09616309 1227 127 0.08602704 277 52 3.421053
## 38 137 0.09410245 214 126 0.08534966 990 49 3.223684
## 218 135 0.09272870 687 126 0.08534966 1142 43 2.828947
## 1255 133 0.09135494 1490 126 0.08534966 207 39 2.565789
## 1214 132 0.09066806 37 125 0.08467228 584 37 2.434211
## 313 130 0.08929430 118 125 0.08467228 510 31 2.039474
## 510 130 0.08929430 161 125 0.08467228 868 31 2.039474
## 578 129 0.08860742 514 125 0.08467228 1233 31 2.039474
## 1474 129 0.08860742 1107 125 0.08467228 1248 28 1.842105
## 407 128 0.08792054 1190 125 0.08467228 313 27 1.776316
## 526 128 0.08792054 85 124 0.08399491 38 26 1.710526
## 658 128 0.08792054 210 124 0.08399491 878 26 1.710526
