CBDA-SL and Knockoff Filter Results on the Null Dataset - n=300 p=100, 9000 jobs x 10 experiments - MSE and Accuracy as ranking metrics for CBDA
Simeone Marino, Ivo Dinov, Jiachen Xu
Jun 25 2017
Some useful information
This is a summary of a set of 1 experiments using a LONI pipeline workflow file that performs 3000 independent jobs, each one with the CBDA-SL and the knockoff filter feature mining strategies. Each experiments has a total of 9000 jobs and is uniquely identified by 6 input arguments: # of jobs [M], % of missing values [misValperc], min [Kcol_min] and max [Kcol_max] % for FSR-Feature Sampling Range, min [Nrow_min] and max [Nrow_max] % for SSR-Subject Sampling Range.
This document has the final results, by experiment. See https://drive.google.com/file/d/0B5sz_T_1CNJQWmlsRTZEcjBEOEk/view?ths=true for some general documentation of the CBDA-SL project and github https://github.com/SOCR/CBDA for some of the code.
Features selected by both the knockoff filter and the CBDA-SL algorithms are shown as spikes in the histograms shown below. I list the top features selected, set to 15 here.
## [1] EXPERIMENT 2
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 5 15 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 2 9000 0 5 15 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "2"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 185 1.817467 29 135 1.252319 48 178 8.392268
## 20 148 1.453974 83 134 1.243043 77 141 6.647808
## 11 141 1.385205 95 134 1.243043 56 108 5.091938
## 77 137 1.345908 86 130 1.205937 42 75 3.536068
## 92 136 1.336084 96 130 1.205937 43 54 2.545969
## 87 135 1.326260 32 129 1.196660 74 54 2.545969
## 56 133 1.306612 47 128 1.187384 22 52 2.451674
## 84 131 1.286963 15 126 1.168831 16 50 2.357379
## 100 128 1.257491 34 126 1.168831 29 48 2.263083
## 34 127 1.247667 40 126 1.168831 44 48 2.263083
## 49 126 1.237843 30 125 1.159555 71 48 2.263083
## 38 125 1.228018 52 125 1.159555 93 36 1.697313
## 61 125 1.228018 60 125 1.159555 76 33 1.555870
## 54 119 1.169074 69 125 1.159555 8 32 1.508722
## 71 119 1.169074 46 124 1.150278 4 27 1.272984
##
##
##
##
##
##
## [1] EXPERIMENT 3
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 15 30 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 3 9000 0 15 30 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "3"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 353 1.587516 10 278 1.184239 48 156 13.390558
## 92 291 1.308689 21 272 1.158679 77 139 11.931330
## 71 283 1.272711 90 272 1.158679 56 112 9.613734
## 77 279 1.254722 47 270 1.150160 42 71 6.094421
## 87 277 1.245728 95 269 1.145900 16 41 3.519313
## 20 268 1.205253 40 267 1.137380 74 36 3.090129
## 12 267 1.200756 88 266 1.133120 8 34 2.918455
## 11 266 1.196258 50 264 1.124601 76 33 2.832618
## 56 265 1.191761 62 264 1.124601 93 29 2.489270
## 76 262 1.178269 67 262 1.116081 22 28 2.403433
## 18 261 1.173772 78 262 1.116081 43 27 2.317597
## 93 260 1.169275 34 261 1.111821 71 24 2.060086
## 8 254 1.142292 30 260 1.107561 44 20 1.716738
## 31 251 1.128800 96 260 1.107561 89 17 1.459227
## 38 249 1.119806 41 258 1.099042 81 14 1.201717
##
##
##
##
##
##
## [1] EXPERIMENT 4
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 1 5 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 4 9000 0 1 5 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "4"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 20 105 2.711077 42 109 3.023578 48 212 4.336265
## 53 91 2.349600 48 102 2.829404 56 179 3.661280
## 84 83 2.143042 19 62 1.719834 77 178 3.640826
## 100 80 2.065582 43 59 1.636616 42 132 2.699939
## 87 77 1.988123 29 54 1.497920 16 106 2.168133
## 7 61 1.575006 31 54 1.497920 74 102 2.086316
## 11 60 1.549187 81 54 1.497920 43 93 1.902229
## 18 60 1.549187 85 54 1.497920 8 92 1.881775
## 36 59 1.523367 8 52 1.442441 22 89 1.820413
## 28 58 1.497547 41 48 1.331484 71 86 1.759051
## 92 58 1.497547 5 47 1.303745 44 84 1.718143
## 54 57 1.471727 68 47 1.303745 29 76 1.554510
## 56 57 1.471727 79 47 1.303745 81 76 1.554510
## 62 57 1.471727 13 46 1.276006 76 72 1.472694
## 12 53 1.368448 14 46 1.276006 2 69 1.411332
##
##
##
##
##
##
## [1] EXPERIMENT 5
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 5 15 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 5 9000 0 5 15 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "5"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 227 2.207527 13 149 1.345008 48 257 13.118938
## 84 183 1.779636 67 146 1.317927 77 239 12.200102
## 87 175 1.701838 90 141 1.272793 56 227 11.587545
## 11 157 1.526792 95 141 1.272793 42 125 6.380807
## 20 156 1.517067 58 140 1.263766 44 61 3.113834
## 100 153 1.487893 50 139 1.254739 74 58 2.960694
## 92 146 1.419819 45 135 1.218632 16 54 2.756508
## 56 133 1.293397 88 134 1.209605 8 49 2.501276
## 47 132 1.283672 6 132 1.191551 22 45 2.297090
## 49 132 1.283672 15 132 1.191551 71 45 2.297090
## 91 131 1.273947 94 132 1.191551 76 42 2.143951
## 77 127 1.235048 21 131 1.182524 93 41 2.092905
## 37 126 1.225323 32 131 1.182524 29 35 1.786626
## 48 125 1.215599 34 131 1.182524 43 28 1.429301
## 36 124 1.205874 64 131 1.182524 27 24 1.225115
##
##
##
##
##
##
## [1] EXPERIMENT 6
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 0 15 30 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 6 9000 0 15 30 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "6"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 409 1.833505 58 308 1.277213 48 259 25.9519038
## 11 346 1.551083 6 299 1.239892 77 201 20.1402806
## 87 312 1.398664 14 288 1.194277 56 184 18.4368737
## 48 311 1.394181 52 288 1.194277 42 98 9.8196393
## 77 306 1.371767 34 285 1.181837 44 27 2.7054108
## 56 302 1.353835 32 284 1.177690 16 19 1.9038076
## 20 281 1.259694 80 284 1.177690 74 19 1.9038076
## 93 269 1.205899 78 282 1.169397 8 18 1.8036072
## 92 267 1.196934 1 280 1.161103 93 18 1.8036072
## 37 264 1.183485 46 279 1.156956 43 15 1.5030060
## 100 263 1.179002 21 276 1.144516 71 14 1.4028056
## 36 261 1.170036 30 274 1.136222 22 12 1.2024048
## 12 255 1.143139 33 272 1.127929 29 11 1.1022044
## 16 250 1.120724 18 271 1.123782 76 9 0.9018036
## 27 249 1.116242 45 269 1.115488 2 6 0.6012024
##
##
##
##
##
##
## [1] EXPERIMENT 7
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 1 5 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 7 9000 20 1 5 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "7"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 92 2.492549 48 102 2.929351 48 136 2.901643
## 92 76 2.059063 42 86 2.469845 56 124 2.645616
## 20 73 1.977784 8 50 1.435956 77 112 2.389588
## 100 72 1.950691 75 50 1.435956 42 102 2.176232
## 77 67 1.815226 93 50 1.435956 43 82 1.749520
## 12 63 1.706855 34 48 1.378518 74 77 1.642842
## 74 60 1.625576 99 48 1.378518 16 76 1.621506
## 11 59 1.598483 52 45 1.292361 44 76 1.621506
## 34 59 1.598483 19 44 1.263642 8 70 1.493493
## 1 58 1.571390 66 44 1.263642 76 66 1.408150
## 49 56 1.517204 29 43 1.234922 29 65 1.386815
## 54 56 1.517204 9 42 1.206203 71 63 1.344143
## 87 55 1.490111 13 42 1.206203 81 63 1.344143
## 71 54 1.463018 14 42 1.206203 11 62 1.322808
## 84 54 1.463018 26 42 1.206203 22 62 1.322808
##
##
##
##
##
##
## [1] EXPERIMENT 8
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 5 15 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 8 9000 20 5 15 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "8"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 181 1.777821 78 150 1.410437 48 171 8.135109
## 87 159 1.561733 4 139 1.307005 77 159 7.564225
## 20 154 1.512622 64 134 1.259991 56 123 5.851570
## 100 151 1.483155 75 130 1.222379 42 82 3.901047
## 11 148 1.453688 14 129 1.212976 74 61 2.901998
## 92 144 1.414399 69 128 1.203573 16 53 2.521408
## 12 142 1.394755 38 126 1.184767 22 49 2.331113
## 74 138 1.355466 83 126 1.184767 44 49 2.331113
## 49 132 1.296533 30 124 1.165961 8 44 2.093245
## 84 131 1.286711 32 123 1.156559 93 44 2.093245
## 77 124 1.217955 10 121 1.137753 76 39 1.855376
## 34 123 1.208133 15 121 1.137753 43 38 1.807802
## 71 123 1.208133 67 121 1.137753 71 33 1.569933
## 7 120 1.178666 21 119 1.118947 89 32 1.522360
## 24 120 1.178666 73 119 1.118947 29 27 1.284491
##
##
##
##
##
##
## [1] EXPERIMENT 9
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 15 30 30 60
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9 9000 20 15 30 30 60
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "9"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 316 1.411092 80 286 1.207159 48 155 12.788779
## 77 307 1.370903 95 282 1.190275 77 150 12.376238
## 56 295 1.317317 9 278 1.173392 56 130 10.726073
## 11 283 1.263731 39 273 1.152288 42 80 6.600660
## 71 282 1.259266 36 269 1.135404 16 46 3.795380
## 74 272 1.214611 55 269 1.135404 74 35 2.887789
## 20 270 1.205680 83 268 1.131184 44 33 2.722772
## 100 265 1.183353 50 267 1.126963 93 33 2.722772
## 12 255 1.138698 15 265 1.118521 43 32 2.640264
## 40 255 1.138698 6 264 1.114300 22 27 2.227723
## 49 254 1.134232 63 264 1.114300 71 26 2.145215
## 61 253 1.129767 94 262 1.105859 8 25 2.062706
## 16 250 1.116370 14 260 1.097417 76 20 1.650165
## 87 250 1.116370 32 260 1.097417 29 19 1.567657
## 22 247 1.102974 67 260 1.097417 2 17 1.402640
##
##
##
##
##
##
## [1] EXPERIMENT 11
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 5 15 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 11 9000 20 5 15 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "11"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 219 2.115738 4 146 1.313658 48 262 13.774974
## 20 194 1.874215 9 144 1.295663 77 224 11.777077
## 87 172 1.661675 5 141 1.268670 56 213 11.198738
## 100 169 1.632692 78 141 1.268670 42 115 6.046267
## 11 166 1.603710 21 140 1.259672 16 75 3.943218
## 92 162 1.565066 15 139 1.250675 74 62 3.259727
## 84 160 1.545744 58 138 1.241677 22 59 3.101998
## 36 147 1.420153 62 138 1.241677 44 59 3.101998
## 16 138 1.333205 50 137 1.232680 93 54 2.839117
## 12 135 1.304222 80 137 1.232680 43 42 2.208202
## 37 133 1.284900 6 136 1.223682 76 37 1.945321
## 49 133 1.284900 85 136 1.223682 71 32 1.682440
## 7 132 1.275239 90 136 1.223682 8 30 1.577287
## 54 131 1.265578 41 134 1.205687 29 29 1.524711
## 91 131 1.265578 10 133 1.196689 81 25 1.314406
##
##
##
##
##
##
## [1] EXPERIMENT 12
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 9000 20 15 30 60 80
## M misValperc Kcol_min Kcol_max Nrow_min Nrow_max
## 12 9000 20 15 30 60 80
## [1] "TABLE with CBDA-SL & KNOCKOFF FILTER RESULTS"
## [1] "EXPERIMENT" "12"
## Accuracy Count Density MSE Count Density Knockoff Count Density
## 53 396 1.771654 67 314 1.305993 48 277 25.8154706
## 20 318 1.422691 6 294 1.222809 77 214 19.9440820
## 56 313 1.400322 80 289 1.202013 56 176 16.4026095
## 48 306 1.369005 9 286 1.189535 42 109 10.1584343
## 77 305 1.364531 18 283 1.177058 44 35 3.2618826
## 87 304 1.360057 10 281 1.168739 16 33 3.0754893
## 11 298 1.333214 50 278 1.156262 74 27 2.5163094
## 92 292 1.306371 58 277 1.152102 93 23 2.1435228
## 100 282 1.261632 61 272 1.131306 22 16 1.4911463
## 84 280 1.252684 86 271 1.127147 43 16 1.4911463
## 16 279 1.248210 95 270 1.122988 8 14 1.3047530
## 93 279 1.248210 60 268 1.114670 71 13 1.2115564
## 91 272 1.216893 99 267 1.110510 76 13 1.2115564
## 47 264 1.181102 17 266 1.106351 81 11 1.0251631
## 8 261 1.167681 28 266 1.106351 29 9 0.8387698
