FMG Study 2 Measurement invariance testing; 2000-2001 +lavaan+all
Design
A total of 189 conditions, with 2000 replications in each.
- n=400, 800, 2000 (group sample sizes)
- ngroups=2,4,8
- One-factor model with dim=5,10, 20
- distributions: normal, vm1, pl1, ig1, vm2, pl2, ig2
- method 2000, 2001
- 2000 replications
2001 vs 2000 Overall
2001 method has lower rejection rates for all tests
Non-normal distributions
VM1
rr plots for some tests by dimension, df diff at n=800
Ranking all tests aggregated over sample sizes, dims, and dfs:
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
peba6_ug | 27 | 0.297 | 0.239 | 100.000 |
peba4_ug | 27 | 0.309 | 0.257 | 100.000 |
pols2_ug | 27 | 0.311 | 0.261 | 100.000 |
pall | 27 | 0.345 | 0.293 | 100.000 |
peba6 | 27 | 0.453 | 0.324 | 100.000 |
pall_ug | 27 | 0.469 | 0.370 | 100.000 |
peba2_ug | 27 | 0.505 | 0.374 | 100.000 |
peba4 | 27 | 0.543 | 0.389 | 100.000 |
pols2 | 27 | 0.550 | 0.391 | 100.000 |
peba2 | 27 | 0.805 | 0.554 | 100.000 |
pall_ug_rls | 27 | 0.913 | 0.552 | 96.296 |
sb_ug | 27 | 0.921 | 0.667 | 96.296 |
pall_rls | 27 | 1.124 | 0.667 | 92.593 |
ss_rls | 27 | 1.219 | 1.083 | 100.000 |
sb | 27 | 1.243 | 0.933 | 92.593 |
peba6_ug_rls | 27 | 1.320 | 0.759 | 92.593 |
ss_ug_rls | 27 | 1.403 | 1.246 | 96.296 |
peba4_ug_rls | 27 | 1.418 | 0.824 | 92.593 |
pols2_ug_rls | 27 | 1.423 | 0.820 | 92.593 |
all_rls | 27 | 1.485 | 1.317 | 92.593 |
peba6_rls | 27 | 1.591 | 0.920 | 85.185 |
all_ug_rls | 27 | 1.637 | 1.454 | 88.889 |
ss | 27 | 1.705 | 1.441 | 81.481 |
peba2_ug_rls | 27 | 1.739 | 1.031 | 85.185 |
peba4_rls | 27 | 1.745 | 0.998 | 85.185 |
pols2_rls | 27 | 1.755 | 1.007 | 85.185 |
ss_ug | 27 | 1.873 | 1.587 | 81.481 |
all | 27 | 1.932 | 1.628 | 81.481 |
all_ug | 27 | 2.110 | 1.796 | 74.074 |
peba2_rls | 27 | 2.171 | 1.272 | 85.185 |
sb_ug_rls | 27 | 2.296 | 1.385 | 85.185 |
sb_rls | 27 | 2.697 | 1.659 | 77.778 |
std_rls | 27 | 81.158 | 78.676 | 0.000 |
std_ml | 27 | 81.624 | 79.206 | 0.000 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 27 | 2.492 | 2.017 | 62.963 |
sb_rls | 27 | 2.948 | 2.493 | 51.852 |
sb_ug_rls | 27 | 3.025 | 2.591 | 48.148 |
peba2_rls | 27 | 3.047 | 2.617 | 51.852 |
sb | 27 | 3.052 | 2.585 | 48.148 |
sb_ug | 27 | 3.109 | 2.667 | 48.148 |
pols2_rls | 27 | 3.111 | 2.696 | 44.444 |
peba2_ug_rls | 27 | 3.113 | 2.702 | 44.444 |
peba4_rls | 27 | 3.113 | 2.700 | 44.444 |
peba6_rls | 27 | 3.138 | 2.730 | 44.444 |
peba2 | 27 | 3.143 | 2.717 | 40.741 |
pall_rls | 27 | 3.190 | 2.783 | 44.444 |
peba4_ug_rls | 27 | 3.190 | 2.793 | 40.741 |
pols2_ug_rls | 27 | 3.192 | 2.793 | 44.444 |
peba4 | 27 | 3.207 | 2.798 | 40.741 |
pols2 | 27 | 3.211 | 2.804 | 37.037 |
peba2_ug | 27 | 3.212 | 2.806 | 40.741 |
peba6_ug_rls | 27 | 3.213 | 2.819 | 44.444 |
peba6 | 27 | 3.221 | 2.817 | 37.037 |
peba4_ug | 27 | 3.250 | 2.854 | 37.037 |
pols2_ug | 27 | 3.250 | 2.854 | 37.037 |
pall_ug_rls | 27 | 3.252 | 2.861 | 37.037 |
pall | 27 | 3.265 | 2.863 | 37.037 |
peba6_ug | 27 | 3.271 | 2.878 | 37.037 |
pall_ug | 27 | 3.310 | 2.919 | 33.333 |
ss_rls | 27 | 3.577 | 3.261 | 25.926 |
ss_ug_rls | 27 | 3.616 | 3.306 | 25.926 |
ss | 27 | 3.634 | 3.317 | 25.926 |
all_rls | 27 | 3.637 | 3.326 | 25.926 |
ss_ug | 27 | 3.662 | 3.352 | 25.926 |
all_ug_rls | 27 | 3.681 | 3.380 | 25.926 |
all | 27 | 3.685 | 3.370 | 25.926 |
all_ug | 27 | 3.709 | 3.402 | 25.926 |
std_rls | 27 | 81.158 | 78.676 | 0.000 |
std_ml | 27 | 81.624 | 79.206 | 0.000 |
VM2
rr plots for some tests by dimension, df diff at n=800
Ranking all tests aggregated over sample sizes, dims, and dfs:
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
pall_ug | 27 | 0.985 | 0.728 | 96.296 |
pall | 27 | 1.207 | 0.828 | 96.296 |
peba6_ug | 27 | 1.881 | 1.067 | 88.889 |
peba4_ug | 27 | 2.140 | 1.178 | 85.185 |
peba6 | 27 | 2.180 | 1.224 | 85.185 |
pols2_ug | 27 | 2.229 | 1.222 | 85.185 |
ss_ug_rls | 27 | 2.231 | 1.830 | 88.889 |
all_rls | 27 | 2.246 | 1.956 | 81.481 |
all_ug_rls | 27 | 2.279 | 2.061 | 77.778 |
ss_rls | 27 | 2.355 | 1.787 | 96.296 |
ss | 27 | 2.501 | 2.296 | 44.444 |
peba4 | 27 | 2.547 | 1.394 | 85.185 |
pols2 | 27 | 2.652 | 1.450 | 81.481 |
ss_ug | 27 | 2.665 | 2.461 | 40.741 |
all | 27 | 2.821 | 2.596 | 37.037 |
peba2_ug | 27 | 2.904 | 1.661 | 77.778 |
all_ug | 27 | 2.941 | 2.726 | 29.630 |
peba2 | 27 | 3.438 | 1.970 | 77.778 |
sb_ug | 27 | 3.895 | 2.374 | 77.778 |
sb | 27 | 4.426 | 2.717 | 70.370 |
pall_ug_rls | 27 | 5.419 | 2.663 | 77.778 |
pall_rls | 27 | 5.873 | 2.896 | 74.074 |
peba6_ug_rls | 27 | 7.264 | 3.711 | 66.667 |
peba4_ug_rls | 27 | 7.719 | 3.991 | 62.963 |
peba6_rls | 27 | 7.807 | 4.019 | 66.667 |
pols2_ug_rls | 27 | 7.846 | 4.046 | 62.963 |
peba4_rls | 27 | 8.274 | 4.296 | 59.259 |
pols2_rls | 27 | 8.466 | 4.383 | 59.259 |
peba2_ug_rls | 27 | 8.730 | 4.644 | 55.556 |
peba2_rls | 27 | 9.260 | 4.965 | 55.556 |
sb_ug_rls | 27 | 9.769 | 5.474 | 55.556 |
sb_rls | 27 | 10.410 | 5.896 | 51.852 |
std_rls | 27 | 91.212 | 90.781 | 0.000 |
std_ml | 27 | 91.381 | 90.970 | 0.000 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 27 | 3.628 | 3.339 | 33.333 |
sb_rls | 27 | 3.790 | 3.528 | 25.926 |
sb_ug_rls | 27 | 3.835 | 3.581 | 25.926 |
peba2_rls | 27 | 3.898 | 3.665 | 25.926 |
sb | 27 | 3.918 | 3.657 | 25.926 |
peba2_ug_rls | 27 | 3.932 | 3.707 | 25.926 |
sb_ug | 27 | 3.947 | 3.696 | 22.222 |
pols2_rls | 27 | 3.961 | 3.744 | 25.926 |
peba4_rls | 27 | 3.961 | 3.743 | 25.926 |
peba6_rls | 27 | 3.991 | 3.778 | 22.222 |
peba2 | 27 | 4.005 | 3.765 | 22.222 |
pols2_ug_rls | 27 | 4.006 | 3.800 | 22.222 |
peba4_ug_rls | 27 | 4.009 | 3.806 | 18.519 |
peba6_ug_rls | 27 | 4.033 | 3.835 | 18.519 |
peba2_ug | 27 | 4.037 | 3.809 | 22.222 |
pall_rls | 27 | 4.044 | 3.837 | 18.519 |
pols2 | 27 | 4.053 | 3.828 | 18.519 |
peba4 | 27 | 4.055 | 3.830 | 18.519 |
peba6 | 27 | 4.067 | 3.844 | 18.519 |
peba4_ug | 27 | 4.080 | 3.863 | 18.519 |
pols2_ug | 27 | 4.080 | 3.863 | 18.519 |
pall_ug_rls | 27 | 4.086 | 3.893 | 14.815 |
peba6_ug | 27 | 4.093 | 3.880 | 18.519 |
pall | 27 | 4.095 | 3.876 | 18.519 |
pall_ug | 27 | 4.132 | 3.922 | 14.815 |
ss_rls | 27 | 4.344 | 4.215 | 11.111 |
ss | 27 | 4.366 | 4.243 | 11.111 |
ss_ug_rls | 27 | 4.366 | 4.243 | 11.111 |
ss_ug | 27 | 4.382 | 4.265 | 11.111 |
all_rls | 27 | 4.385 | 4.261 | 11.111 |
all | 27 | 4.397 | 4.278 | 11.111 |
all_ug_rls | 27 | 4.409 | 4.291 | 11.111 |
all_ug | 27 | 4.421 | 4.309 | 7.407 |
std_rls | 27 | 91.212 | 90.781 | 0.000 |
std_ml | 27 | 91.381 | 90.970 | 0.000 |
PL1
rr plots for some tests by dimension, df diff at n=800
Ranking all tests aggregated over sample sizes, dims, and dfs:
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
peba4_ug_rls | 27 | 0.535 | 0.402 | 100.000 |
peba2_ug_rls | 27 | 0.545 | 0.409 | 100.000 |
pall_ug | 27 | 0.546 | 0.393 | 100.000 |
pols2_ug_rls | 27 | 0.546 | 0.413 | 100.000 |
peba6_ug_rls | 27 | 0.553 | 0.428 | 100.000 |
pall_ug_rls | 27 | 0.573 | 0.441 | 100.000 |
peba6_ug | 27 | 0.577 | 0.420 | 100.000 |
sb_ug_rls | 27 | 0.586 | 0.446 | 100.000 |
peba4_ug | 27 | 0.596 | 0.435 | 100.000 |
pols2_ug | 27 | 0.597 | 0.437 | 100.000 |
pall_rls | 27 | 0.599 | 0.437 | 100.000 |
peba4_rls | 27 | 0.603 | 0.450 | 100.000 |
peba6_rls | 27 | 0.604 | 0.450 | 100.000 |
pols2_rls | 27 | 0.606 | 0.456 | 100.000 |
peba2_rls | 27 | 0.649 | 0.504 | 100.000 |
peba2_ug | 27 | 0.683 | 0.502 | 100.000 |
ss | 27 | 0.739 | 0.559 | 100.000 |
pall | 27 | 0.761 | 0.580 | 100.000 |
peba6 | 27 | 0.790 | 0.598 | 100.000 |
sb_rls | 27 | 0.826 | 0.617 | 100.000 |
pols2 | 27 | 0.827 | 0.620 | 100.000 |
peba4 | 27 | 0.828 | 0.622 | 100.000 |
all | 27 | 0.865 | 0.633 | 100.000 |
ss_ug | 27 | 0.883 | 0.657 | 100.000 |
sb_ug | 27 | 0.905 | 0.663 | 100.000 |
peba2 | 27 | 0.958 | 0.715 | 100.000 |
ss_rls | 27 | 0.996 | 0.754 | 100.000 |
all_ug | 27 | 1.025 | 0.791 | 100.000 |
all_rls | 27 | 1.116 | 0.846 | 92.593 |
ss_ug_rls | 27 | 1.141 | 0.870 | 92.593 |
all_ug_rls | 27 | 1.261 | 0.989 | 88.889 |
sb | 27 | 1.262 | 0.917 | 92.593 |
std_rls | 27 | 84.609 | 82.941 | 0.000 |
std_ml | 27 | 85.046 | 83.424 | 0.000 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 27 | 1.232 | 0.830 | 88.889 |
sb | 27 | 2.636 | 2.094 | 62.963 |
sb_rls | 27 | 2.684 | 2.204 | 62.963 |
peba2 | 27 | 2.705 | 2.185 | 62.963 |
sb_ug | 27 | 2.708 | 2.202 | 62.963 |
peba4 | 27 | 2.738 | 2.222 | 62.963 |
sb_ug_rls | 27 | 2.738 | 2.280 | 59.259 |
peba2_rls | 27 | 2.739 | 2.274 | 59.259 |
pols2 | 27 | 2.739 | 2.226 | 62.963 |
peba6 | 27 | 2.742 | 2.228 | 62.963 |
pall | 27 | 2.747 | 2.235 | 62.963 |
peba4_rls | 27 | 2.759 | 2.305 | 59.259 |
pols2_rls | 27 | 2.765 | 2.313 | 59.259 |
peba6_rls | 27 | 2.768 | 2.317 | 59.259 |
pall_rls | 27 | 2.775 | 2.326 | 59.259 |
peba2_ug | 27 | 2.784 | 2.300 | 62.963 |
peba2_ug_rls | 27 | 2.797 | 2.372 | 59.259 |
peba4_ug | 27 | 2.805 | 2.328 | 62.963 |
pols2_ug | 27 | 2.811 | 2.337 | 62.963 |
peba6_ug | 27 | 2.812 | 2.337 | 62.963 |
peba4_ug_rls | 27 | 2.817 | 2.398 | 59.259 |
pols2_ug_rls | 27 | 2.823 | 2.404 | 59.259 |
peba6_ug_rls | 27 | 2.824 | 2.405 | 59.259 |
pall_ug | 27 | 2.826 | 2.355 | 62.963 |
pall_ug_rls | 27 | 2.833 | 2.415 | 59.259 |
ss | 27 | 2.997 | 2.561 | 51.852 |
ss_rls | 27 | 3.025 | 2.622 | 55.556 |
all | 27 | 3.030 | 2.605 | 51.852 |
ss_ug | 27 | 3.053 | 2.641 | 51.852 |
all_rls | 27 | 3.075 | 2.685 | 48.148 |
ss_ug_rls | 27 | 3.097 | 2.718 | 48.148 |
all_ug | 27 | 3.106 | 2.696 | 51.852 |
all_ug_rls | 27 | 3.129 | 2.754 | 48.148 |
std_rls | 27 | 84.609 | 82.941 | 0.000 |
std_ml | 27 | 85.046 | 83.424 | 0.000 |
PL2
rr plots for some tests by dimension, df diff at n=800
Ranking all tests aggregated over sample sizes, dims, and dfs:
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
ss_rls | 27 | 1.072 | 0.856 | 96.296 |
ss_ug_rls | 27 | 1.205 | 1.013 | 96.296 |
pall_ug | 27 | 1.220 | 0.819 | 92.593 |
ss | 27 | 1.269 | 1.024 | 96.296 |
all_rls | 27 | 1.308 | 1.094 | 92.593 |
pall_ug_rls | 27 | 1.374 | 0.817 | 92.593 |
ss_ug | 27 | 1.397 | 1.133 | 88.889 |
all_ug_rls | 27 | 1.448 | 1.250 | 92.593 |
all | 27 | 1.495 | 1.206 | 88.889 |
peba6_ug | 27 | 1.552 | 0.978 | 88.889 |
peba6_ug_rls | 27 | 1.620 | 0.954 | 92.593 |
all_ug | 27 | 1.654 | 1.356 | 85.185 |
pall | 27 | 1.667 | 1.078 | 88.889 |
pall_rls | 27 | 1.729 | 1.056 | 88.889 |
peba4_ug | 27 | 1.736 | 1.074 | 88.889 |
pols2_ug | 27 | 1.790 | 1.096 | 88.889 |
peba4_ug_rls | 27 | 1.845 | 1.065 | 88.889 |
pols2_ug_rls | 27 | 1.890 | 1.081 | 88.889 |
peba6 | 27 | 2.008 | 1.252 | 88.889 |
peba6_rls | 27 | 2.074 | 1.228 | 88.889 |
peba4 | 27 | 2.201 | 1.356 | 85.185 |
peba4_rls | 27 | 2.254 | 1.344 | 85.185 |
peba2_ug | 27 | 2.257 | 1.374 | 88.889 |
pols2 | 27 | 2.276 | 1.376 | 85.185 |
peba2_ug_rls | 27 | 2.308 | 1.350 | 81.481 |
pols2_rls | 27 | 2.321 | 1.367 | 81.481 |
peba2 | 27 | 2.704 | 1.687 | 77.778 |
peba2_rls | 27 | 2.896 | 1.706 | 77.778 |
sb_ug | 27 | 3.055 | 1.993 | 74.074 |
sb_ug_rls | 27 | 3.241 | 1.987 | 77.778 |
sb_rls | 27 | 3.614 | 2.267 | 77.778 |
sb | 27 | 3.684 | 2.389 | 70.370 |
std_rls | 27 | 92.692 | 92.500 | 0.000 |
std_ml | 27 | 92.803 | 92.620 | 0.000 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 27 | 2.767 | 2.272 | 55.556 |
sb | 27 | 3.589 | 3.150 | 37.037 |
sb_rls | 27 | 3.608 | 3.219 | 37.037 |
sb_ug | 27 | 3.630 | 3.213 | 37.037 |
sb_ug_rls | 27 | 3.636 | 3.269 | 37.037 |
peba2_rls | 27 | 3.673 | 3.320 | 37.037 |
peba2 | 27 | 3.673 | 3.267 | 37.037 |
peba2_ug | 27 | 3.704 | 3.317 | 33.333 |
peba4_rls | 27 | 3.706 | 3.365 | 33.333 |
pols2_rls | 27 | 3.708 | 3.367 | 33.333 |
pols2 | 27 | 3.709 | 3.320 | 33.333 |
peba4 | 27 | 3.711 | 3.322 | 33.333 |
peba2_ug_rls | 27 | 3.712 | 3.372 | 33.333 |
peba6 | 27 | 3.714 | 3.326 | 33.333 |
peba6_rls | 27 | 3.723 | 3.380 | 33.333 |
pall | 27 | 3.733 | 3.350 | 29.630 |
pall_rls | 27 | 3.739 | 3.396 | 29.630 |
peba4_ug | 27 | 3.756 | 3.383 | 29.630 |
pols2_ug | 27 | 3.757 | 3.383 | 29.630 |
peba6_ug | 27 | 3.758 | 3.385 | 29.630 |
peba4_ug_rls | 27 | 3.759 | 3.431 | 29.630 |
pols2_ug_rls | 27 | 3.762 | 3.437 | 29.630 |
pall_ug | 27 | 3.770 | 3.402 | 29.630 |
peba6_ug_rls | 27 | 3.770 | 3.444 | 29.630 |
pall_ug_rls | 27 | 3.779 | 3.456 | 29.630 |
ss | 27 | 3.981 | 3.694 | 25.926 |
ss_rls | 27 | 3.983 | 3.722 | 18.519 |
ss_ug | 27 | 4.004 | 3.728 | 18.519 |
ss_ug_rls | 27 | 4.008 | 3.754 | 18.519 |
all_rls | 27 | 4.017 | 3.763 | 18.519 |
all | 27 | 4.018 | 3.743 | 22.222 |
all_ug | 27 | 4.041 | 3.774 | 18.519 |
all_ug_rls | 27 | 4.044 | 3.800 | 18.519 |
std_rls | 27 | 92.692 | 92.500 | 0.000 |
std_ml | 27 | 92.803 | 92.620 | 0.000 |
IG1
rr plots for some tests by dimension, df diff at n=800
Ranking all tests aggregated over sample sizes, dims, and dfs:
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
pall_ug_rls | 27 | 1.396 | 1.250 | 100.000 |
pall_ug | 27 | 1.460 | 1.282 | 100.000 |
pall_rls | 27 | 1.615 | 1.478 | 96.296 |
pall | 27 | 1.705 | 1.524 | 88.889 |
ss | 27 | 1.725 | 1.548 | 92.593 |
ss_rls | 27 | 1.787 | 1.646 | 88.889 |
ss_ug | 27 | 1.861 | 1.678 | 85.185 |
ss_ug_rls | 27 | 1.891 | 1.735 | 81.481 |
all | 27 | 2.137 | 1.950 | 66.667 |
all_rls | 27 | 2.204 | 2.033 | 66.667 |
all_ug | 27 | 2.238 | 2.042 | 62.963 |
all_ug_rls | 27 | 2.313 | 2.122 | 55.556 |
peba6_ug | 27 | 2.587 | 2.269 | 59.259 |
peba6_ug_rls | 27 | 2.609 | 2.313 | 59.259 |
peba6 | 27 | 2.895 | 2.526 | 51.852 |
peba6_rls | 27 | 2.900 | 2.559 | 51.852 |
peba4_ug_rls | 27 | 2.971 | 2.630 | 51.852 |
peba4_ug | 27 | 2.999 | 2.613 | 51.852 |
pols2_ug_rls | 27 | 3.072 | 2.730 | 51.852 |
pols2_ug | 27 | 3.135 | 2.739 | 51.852 |
peba4_rls | 27 | 3.256 | 2.867 | 48.148 |
peba4 | 27 | 3.334 | 2.891 | 48.148 |
pols2_rls | 27 | 3.415 | 3.008 | 48.148 |
pols2 | 27 | 3.462 | 3.006 | 48.148 |
peba2_ug | 27 | 3.838 | 3.382 | 40.741 |
peba2_ug_rls | 27 | 3.851 | 3.434 | 44.444 |
peba2_rls | 27 | 4.158 | 3.704 | 33.333 |
peba2 | 27 | 4.264 | 3.706 | 33.333 |
sb_ug_rls | 27 | 4.749 | 4.347 | 11.111 |
sb_ug | 27 | 4.971 | 4.498 | 11.111 |
sb_rls | 27 | 5.075 | 4.624 | 11.111 |
sb | 27 | 5.359 | 4.839 | 7.407 |
std_rls | 27 | 77.802 | 75.490 | 0.000 |
std_ml | 27 | 78.487 | 76.298 | 0.000 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 27 | 1.326 | 1.069 | 96.296 |
sb | 27 | 1.522 | 1.215 | 85.185 |
sb_ug | 27 | 1.642 | 1.287 | 81.481 |
sb_rls | 27 | 1.690 | 1.335 | 81.481 |
peba2 | 27 | 1.711 | 1.265 | 81.481 |
sb_ug_rls | 27 | 1.768 | 1.382 | 81.481 |
peba2_ug | 27 | 1.796 | 1.352 | 81.481 |
peba2_rls | 27 | 1.823 | 1.369 | 81.481 |
pols2 | 27 | 1.904 | 1.444 | 77.778 |
peba2_ug_rls | 27 | 1.941 | 1.474 | 74.074 |
peba4 | 27 | 1.954 | 1.485 | 77.778 |
pols2_ug | 27 | 2.005 | 1.531 | 74.074 |
peba4_ug | 27 | 2.046 | 1.583 | 74.074 |
pols2_rls | 27 | 2.056 | 1.565 | 74.074 |
peba6 | 27 | 2.059 | 1.605 | 74.074 |
peba4_rls | 27 | 2.095 | 1.604 | 70.370 |
pols2_ug_rls | 27 | 2.140 | 1.631 | 70.370 |
peba6_ug | 27 | 2.141 | 1.692 | 70.370 |
peba4_ug_rls | 27 | 2.180 | 1.687 | 66.667 |
peba6_rls | 27 | 2.204 | 1.742 | 66.667 |
peba6_ug_rls | 27 | 2.299 | 1.835 | 62.963 |
pall | 27 | 2.378 | 1.937 | 66.667 |
pall_ug | 27 | 2.466 | 2.030 | 59.259 |
pall_rls | 27 | 2.533 | 2.089 | 59.259 |
pall_ug_rls | 27 | 2.583 | 2.141 | 55.556 |
ss | 27 | 3.594 | 3.415 | 25.926 |
ss_ug | 27 | 3.648 | 3.476 | 22.222 |
ss_rls | 27 | 3.670 | 3.489 | 25.926 |
ss_ug_rls | 27 | 3.716 | 3.535 | 25.926 |
all | 27 | 3.775 | 3.615 | 14.815 |
all_ug | 27 | 3.822 | 3.668 | 14.815 |
all_rls | 27 | 3.838 | 3.670 | 14.815 |
all_ug_rls | 27 | 3.873 | 3.711 | 14.815 |
std_rls | 27 | 77.802 | 75.490 | 0.000 |
std_ml | 27 | 78.487 | 76.298 | 0.000 |
IG2
rr plots for some tests by dimension, df diff at n=800
Ranking all tests aggregated over sample sizes, dims, and dfs:
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
ss_rls | 27 | 1.072 | 0.856 | 96.296 |
ss_ug_rls | 27 | 1.205 | 1.013 | 96.296 |
pall_ug | 27 | 1.220 | 0.819 | 92.593 |
ss | 27 | 1.269 | 1.024 | 96.296 |
all_rls | 27 | 1.308 | 1.094 | 92.593 |
pall_ug_rls | 27 | 1.374 | 0.817 | 92.593 |
ss_ug | 27 | 1.397 | 1.133 | 88.889 |
all_ug_rls | 27 | 1.448 | 1.250 | 92.593 |
all | 27 | 1.495 | 1.206 | 88.889 |
peba6_ug | 27 | 1.552 | 0.978 | 88.889 |
peba6_ug_rls | 27 | 1.620 | 0.954 | 92.593 |
all_ug | 27 | 1.654 | 1.356 | 85.185 |
pall | 27 | 1.667 | 1.078 | 88.889 |
pall_rls | 27 | 1.729 | 1.056 | 88.889 |
peba4_ug | 27 | 1.736 | 1.074 | 88.889 |
pols2_ug | 27 | 1.790 | 1.096 | 88.889 |
peba4_ug_rls | 27 | 1.845 | 1.065 | 88.889 |
pols2_ug_rls | 27 | 1.890 | 1.081 | 88.889 |
peba6 | 27 | 2.008 | 1.252 | 88.889 |
peba6_rls | 27 | 2.074 | 1.228 | 88.889 |
peba4 | 27 | 2.201 | 1.356 | 85.185 |
peba4_rls | 27 | 2.254 | 1.344 | 85.185 |
peba2_ug | 27 | 2.257 | 1.374 | 88.889 |
pols2 | 27 | 2.276 | 1.376 | 85.185 |
peba2_ug_rls | 27 | 2.308 | 1.350 | 81.481 |
pols2_rls | 27 | 2.321 | 1.367 | 81.481 |
peba2 | 27 | 2.704 | 1.687 | 77.778 |
peba2_rls | 27 | 2.896 | 1.706 | 77.778 |
sb_ug | 27 | 3.055 | 1.993 | 74.074 |
sb_ug_rls | 27 | 3.241 | 1.987 | 77.778 |
sb_rls | 27 | 3.614 | 2.267 | 77.778 |
sb | 27 | 3.684 | 2.389 | 70.370 |
std_rls | 27 | 92.692 | 92.500 | 0.000 |
std_ml | 27 | 92.803 | 92.620 | 0.000 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 27 | 2.767 | 2.272 | 55.556 |
sb | 27 | 3.589 | 3.150 | 37.037 |
sb_rls | 27 | 3.608 | 3.219 | 37.037 |
sb_ug | 27 | 3.630 | 3.213 | 37.037 |
sb_ug_rls | 27 | 3.636 | 3.269 | 37.037 |
peba2_rls | 27 | 3.673 | 3.320 | 37.037 |
peba2 | 27 | 3.673 | 3.267 | 37.037 |
peba2_ug | 27 | 3.704 | 3.317 | 33.333 |
peba4_rls | 27 | 3.706 | 3.365 | 33.333 |
pols2_rls | 27 | 3.708 | 3.367 | 33.333 |
pols2 | 27 | 3.709 | 3.320 | 33.333 |
peba4 | 27 | 3.711 | 3.322 | 33.333 |
peba2_ug_rls | 27 | 3.712 | 3.372 | 33.333 |
peba6 | 27 | 3.714 | 3.326 | 33.333 |
peba6_rls | 27 | 3.723 | 3.380 | 33.333 |
pall | 27 | 3.733 | 3.350 | 29.630 |
pall_rls | 27 | 3.739 | 3.396 | 29.630 |
peba4_ug | 27 | 3.756 | 3.383 | 29.630 |
pols2_ug | 27 | 3.757 | 3.383 | 29.630 |
peba6_ug | 27 | 3.758 | 3.385 | 29.630 |
peba4_ug_rls | 27 | 3.759 | 3.431 | 29.630 |
pols2_ug_rls | 27 | 3.762 | 3.437 | 29.630 |
pall_ug | 27 | 3.770 | 3.402 | 29.630 |
peba6_ug_rls | 27 | 3.770 | 3.444 | 29.630 |
pall_ug_rls | 27 | 3.779 | 3.456 | 29.630 |
ss | 27 | 3.981 | 3.694 | 25.926 |
ss_rls | 27 | 3.983 | 3.722 | 18.519 |
ss_ug | 27 | 4.004 | 3.728 | 18.519 |
ss_ug_rls | 27 | 4.008 | 3.754 | 18.519 |
all_rls | 27 | 4.017 | 3.763 | 18.519 |
all | 27 | 4.018 | 3.743 | 22.222 |
all_ug | 27 | 4.041 | 3.774 | 18.519 |
all_ug_rls | 27 | 4.044 | 3.800 | 18.519 |
std_rls | 27 | 92.692 | 92.500 | 0.000 |
std_ml | 27 | 92.803 | 92.620 | 0.000 |
Aggregated over all non-normal distributions, dims, sample sizes, groups
Method 2000:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
pall_ug | 189 | 0.946 | 0.692 | 98.413 |
pall | 189 | 1.172 | 0.843 | 96.296 |
ss_rls | 189 | 1.418 | 1.071 | 96.825 |
peba6_ug | 189 | 1.443 | 0.912 | 91.005 |
ss | 189 | 1.482 | 1.147 | 87.831 |
ss_ug_rls | 189 | 1.489 | 1.179 | 92.593 |
all_rls | 189 | 1.580 | 1.270 | 88.360 |
ss_ug | 189 | 1.612 | 1.250 | 85.185 |
peba4_ug | 189 | 1.635 | 1.005 | 89.418 |
all_ug_rls | 189 | 1.685 | 1.387 | 85.185 |
pols2_ug | 189 | 1.697 | 1.034 | 89.418 |
peba6 | 189 | 1.704 | 1.100 | 88.889 |
all | 189 | 1.727 | 1.344 | 82.011 |
all_ug | 189 | 1.848 | 1.458 | 78.307 |
peba4 | 189 | 1.931 | 1.218 | 87.302 |
pols2 | 189 | 1.996 | 1.247 | 86.772 |
peba2_ug | 189 | 2.121 | 1.293 | 85.714 |
pall_ug_rls | 189 | 2.241 | 0.958 | 95.238 |
pall_rls | 189 | 2.469 | 1.091 | 93.122 |
peba2 | 189 | 2.470 | 1.549 | 82.011 |
sb_ug | 189 | 2.811 | 1.784 | 77.249 |
peba6_ug_rls | 189 | 3.051 | 1.320 | 87.302 |
sb | 189 | 3.188 | 2.077 | 70.899 |
peba4_ug_rls | 189 | 3.276 | 1.437 | 85.185 |
peba6_rls | 189 | 3.333 | 1.492 | 84.656 |
pols2_ug_rls | 189 | 3.337 | 1.466 | 85.185 |
peba4_rls | 189 | 3.564 | 1.614 | 82.540 |
pols2_rls | 189 | 3.656 | 1.654 | 82.011 |
peba2_ug_rls | 189 | 3.802 | 1.752 | 80.952 |
peba2_rls | 189 | 4.114 | 1.963 | 78.836 |
sb_ug_rls | 189 | 4.418 | 2.203 | 75.661 |
sb_rls | 189 | 4.757 | 2.439 | 73.545 |
std_rls | 189 | 75.201 | 67.147 | 14.286 |
std_ml | 189 | 75.658 | 67.692 | 14.286 |
Method 2001:
test | numcond | RMSE | MAD | ARR |
---|---|---|---|---|
anova | 189 | 2.131 | 1.566 | 76.720 |
sb | 189 | 2.709 | 2.122 | 61.905 |
sb_rls | 189 | 2.725 | 2.175 | 61.376 |
sb_ug | 189 | 2.768 | 2.195 | 60.317 |
sb_ug_rls | 189 | 2.782 | 2.251 | 59.788 |
peba2 | 189 | 2.796 | 2.208 | 58.730 |
peba2_rls | 189 | 2.806 | 2.258 | 60.317 |
peba2_ug | 189 | 2.854 | 2.287 | 58.201 |
pols2 | 189 | 2.856 | 2.281 | 56.614 |
peba4 | 189 | 2.862 | 2.289 | 57.143 |
peba2_ug_rls | 189 | 2.869 | 2.345 | 57.143 |
pols2_rls | 189 | 2.874 | 2.338 | 56.614 |
peba4_rls | 189 | 2.878 | 2.344 | 56.085 |
peba6 | 189 | 2.882 | 2.316 | 56.085 |
peba6_rls | 189 | 2.907 | 2.382 | 55.026 |
pols2_ug | 189 | 2.911 | 2.356 | 55.026 |
peba4_ug | 189 | 2.915 | 2.362 | 55.026 |
peba6_ug | 189 | 2.937 | 2.394 | 54.497 |
pols2_ug_rls | 189 | 2.940 | 2.425 | 55.026 |
pall | 189 | 2.941 | 2.391 | 53.439 |
peba4_ug_rls | 189 | 2.945 | 2.433 | 53.439 |
peba6_ug_rls | 189 | 2.974 | 2.472 | 53.439 |
pall_rls | 189 | 2.976 | 2.462 | 52.910 |
pall_ug | 189 | 3.002 | 2.475 | 51.323 |
pall_ug_rls | 189 | 3.037 | 2.547 | 50.794 |
ss | 189 | 3.380 | 2.966 | 40.212 |
ss_rls | 189 | 3.413 | 3.019 | 39.683 |
ss_ug | 189 | 3.420 | 3.021 | 38.095 |
all | 189 | 3.447 | 3.046 | 37.566 |
ss_ug_rls | 189 | 3.458 | 3.079 | 37.037 |
all_rls | 189 | 3.481 | 3.098 | 36.508 |
all_ug | 189 | 3.491 | 3.102 | 35.979 |
all_ug_rls | 189 | 3.524 | 3.156 | 34.921 |
std_rls | 189 | 75.201 | 67.147 | 14.286 |
std_ml | 189 | 75.658 | 67.692 | 14.286 |