FMG Study 2 Measurement invariance testing; 2000-2001 +lavaan+all+sf

Design

A total of 189 conditions, with 2000 replications in each.

  • n=400, 800, 2000 (group sample sizes)
  • ngroups=2,4,8
  • One-factor model with dim=5,10, 20
  • distributions: normal, vm1, pl1, ig1, vm2, pl2, ig2
  • method 2000, 2001
  • 2000 replications

2001 vs 2000 Overall

2001 method has lower rejection rates for all tests

Consider only 2000 methods+ sb_2001 in the remaining.

#Normal distribution

rejection plots ML and RLS

Non-normal distributions

VM1

Ranking all tests aggregated over sample sizes, dims, dfs:

test numcond RMSE MAD BEL2.5 ARR ABO7.5
peba6_ug 27 0.297 0.239 0.0 100.0 0.0
peba4_ug 27 0.309 0.257 0.0 100.0 0.0
pols2_ug 27 0.311 0.261 0.0 100.0 0.0
pall 27 0.345 0.293 0.0 100.0 0.0
peba6 27 0.453 0.324 0.0 100.0 0.0
pall_ug 27 0.469 0.370 0.0 100.0 0.0
peba2_ug 27 0.505 0.374 0.0 100.0 0.0
peba4 27 0.543 0.389 0.0 100.0 0.0
pols2 27 0.550 0.391 0.0 100.0 0.0
peba2 27 0.805 0.554 0.0 100.0 0.0
pall_ug_rls 27 0.913 0.552 0.0 96.3 3.7
sb_ug 27 0.921 0.667 0.0 96.3 3.7
pall_rls 27 1.124 0.667 0.0 92.6 7.4
ss_rls 27 1.219 1.083 0.0 100.0 0.0
sb 27 1.243 0.933 0.0 92.6 7.4
peba6_ug_rls 27 1.320 0.759 0.0 92.6 7.4
ss_ug_rls 27 1.403 1.246 0.0 96.3 0.0
peba4_ug_rls 27 1.418 0.824 0.0 92.6 7.4
pols2_ug_rls 27 1.423 0.820 0.0 92.6 7.4
all_rls 27 1.485 1.317 3.7 92.6 0.0
sf_rls 27 1.487 1.313 7.4 92.6 0.0
peba6_rls 27 1.591 0.920 0.0 85.2 14.8
sf_ug_rls 27 1.634 1.446 11.1 88.9 0.0
all_ug_rls 27 1.637 1.454 11.1 88.9 0.0
ss 27 1.705 1.441 14.8 81.5 0.0
peba2_ug_rls 27 1.739 1.031 0.0 85.2 14.8
peba4_rls 27 1.745 0.998 0.0 85.2 14.8
pols2_rls 27 1.755 1.007 0.0 85.2 14.8
ss_ug 27 1.873 1.587 18.5 81.5 0.0
all 27 1.932 1.628 18.5 81.5 0.0
sf 27 1.940 1.631 18.5 81.5 0.0
all_ug 27 2.110 1.796 25.9 74.1 0.0
sf_ug 27 2.115 1.802 25.9 74.1 0.0
peba2_rls 27 2.171 1.272 0.0 85.2 14.8
sb_ug_rls 27 2.296 1.385 0.0 85.2 14.8
sb_2001 27 2.492 2.017 37.0 63.0 0.0
sb_rls 27 2.697 1.659 0.0 77.8 18.5
std_rls 27 81.158 78.676 0.0 0.0 100.0
std_ml 27 81.624 79.206 0.0 0.0 100.0

VM2

Ranking all tests aggregated over sample sizes, dims, dfs:

test numcond RMSE MAD BEL2.5 ARR ABO7.5
pall_ug 27 0.985 0.728 0.0 96.3 3.7
pall 27 1.207 0.828 0.0 96.3 3.7
peba6_ug 27 1.881 1.067 0.0 88.9 11.1
peba4_ug 27 2.140 1.178 0.0 85.2 14.8
peba6 27 2.180 1.224 0.0 85.2 14.8
pols2_ug 27 2.229 1.222 0.0 85.2 14.8
ss_ug_rls 27 2.231 1.830 3.7 88.9 3.7
sf_ug_rls 27 2.236 2.000 14.8 77.8 3.7
sf_rls 27 2.238 1.911 14.8 81.5 3.7
all_rls 27 2.246 1.956 14.8 81.5 3.7
all_ug_rls 27 2.279 2.061 14.8 77.8 3.7
ss_rls 27 2.355 1.787 0.0 96.3 3.7
ss 27 2.501 2.296 44.4 44.4 0.0
peba4 27 2.547 1.394 0.0 85.2 14.8
pols2 27 2.652 1.450 0.0 81.5 14.8
ss_ug 27 2.665 2.461 55.6 40.7 0.0
sf 27 2.800 2.565 59.3 37.0 0.0
all 27 2.821 2.596 63.0 37.0 0.0
peba2_ug 27 2.904 1.661 0.0 77.8 22.2
sf_ug 27 2.920 2.694 66.7 29.6 0.0
all_ug 27 2.941 2.726 66.7 29.6 0.0
peba2 27 3.438 1.970 0.0 77.8 22.2
sb_2001 27 3.628 3.339 66.7 33.3 0.0
sb_ug 27 3.895 2.374 0.0 77.8 22.2
sb 27 4.426 2.717 0.0 70.4 29.6
pall_ug_rls 27 5.419 2.663 0.0 77.8 22.2
pall_rls 27 5.873 2.896 0.0 74.1 25.9
peba6_ug_rls 27 7.264 3.711 0.0 66.7 33.3
peba4_ug_rls 27 7.719 3.991 0.0 63.0 33.3
peba6_rls 27 7.807 4.019 0.0 66.7 33.3
pols2_ug_rls 27 7.846 4.046 0.0 63.0 37.0
peba4_rls 27 8.274 4.296 0.0 59.3 37.0
pols2_rls 27 8.466 4.383 0.0 59.3 40.7
peba2_ug_rls 27 8.730 4.644 0.0 55.6 44.4
peba2_rls 27 9.260 4.965 0.0 55.6 44.4
sb_ug_rls 27 9.769 5.474 0.0 55.6 44.4
sb_rls 27 10.410 5.896 0.0 51.9 44.4
std_rls 27 91.212 90.781 0.0 0.0 100.0
std_ml 27 91.381 90.970 0.0 0.0 100.0

IG1

Ranking all tests aggregated over sample sizes, dims, dfs:

test numcond RMSE MAD BEL2.5 ARR ABO7.5
pall_ug_rls 27 0.715 0.619 0.0 100.0 0.0
peba6_ug_rls 27 0.805 0.709 0.0 100.0 0.0
pall_rls 27 0.854 0.752 0.0 100.0 0.0
peba4_ug_rls 27 0.861 0.782 0.0 100.0 0.0
pols2_ug_rls 27 0.880 0.804 0.0 100.0 0.0
ss 27 0.899 0.752 0.0 100.0 0.0
pall_ug 27 0.969 0.883 0.0 100.0 0.0
peba6_rls 27 0.982 0.919 0.0 100.0 0.0
sb_2001 27 0.995 0.891 0.0 100.0 0.0
peba4_rls 27 1.058 0.991 0.0 100.0 0.0
pols2_rls 27 1.075 1.009 0.0 100.0 0.0
ss_ug 27 1.091 0.904 0.0 100.0 0.0
peba2_ug_rls 27 1.100 1.030 0.0 100.0 0.0
peba6_ug 27 1.137 1.046 0.0 100.0 0.0
pall 27 1.156 1.076 0.0 100.0 0.0
sf 27 1.167 0.998 0.0 100.0 0.0
all 27 1.170 1.009 0.0 100.0 0.0
ss_rls 27 1.183 0.972 3.7 96.3 0.0
peba4_ug 27 1.205 1.109 0.0 100.0 0.0
pols2_ug 27 1.216 1.119 0.0 100.0 0.0
peba2_rls 27 1.311 1.232 0.0 100.0 0.0
sf_ug 27 1.337 1.146 3.7 96.3 0.0
ss_ug_rls 27 1.344 1.107 7.4 92.6 0.0
all_ug 27 1.348 1.170 3.7 96.3 0.0
peba6 27 1.370 1.248 0.0 96.3 3.7
sf_rls 27 1.429 1.196 7.4 92.6 0.0
all_rls 27 1.439 1.233 7.4 92.6 0.0
peba4 27 1.487 1.343 0.0 92.6 7.4
pols2 27 1.506 1.358 0.0 92.6 7.4
peba2_ug 27 1.507 1.387 0.0 92.6 7.4
sb_ug_rls 27 1.516 1.437 0.0 100.0 0.0
sf_ug_rls 27 1.574 1.348 7.4 92.6 0.0
all_ug_rls 27 1.587 1.368 7.4 92.6 0.0
sb_rls 27 1.748 1.652 0.0 96.3 3.7
peba2 27 1.837 1.672 0.0 85.2 14.8
sb_ug 27 2.042 1.889 0.0 81.5 18.5
sb 27 2.356 2.150 0.0 63.0 29.6
std_rls 27 53.631 49.279 0.0 0.0 100.0
std_ml 27 55.225 50.837 0.0 0.0 100.0

IG2

Ranking all tests aggregated over sample sizes, dims, dfs:

test numcond RMSE MAD BEL2.5 ARR ABO7.5
sb_2001 27 1.326 1.069 0.0 96.3 3.7
pall_ug_rls 27 1.396 1.250 0.0 100.0 0.0
pall_ug 27 1.460 1.282 0.0 100.0 0.0
pall_rls 27 1.615 1.478 0.0 96.3 3.7
pall 27 1.705 1.524 0.0 88.9 11.1
ss 27 1.725 1.548 7.4 92.6 0.0
ss_rls 27 1.787 1.646 7.4 88.9 0.0
ss_ug 27 1.861 1.678 11.1 85.2 0.0
ss_ug_rls 27 1.891 1.735 11.1 81.5 0.0
sf 27 2.102 1.905 33.3 66.7 0.0
all 27 2.137 1.950 33.3 66.7 0.0
sf_rls 27 2.141 1.968 33.3 66.7 0.0
sf_ug 27 2.200 1.992 37.0 63.0 0.0
all_rls 27 2.204 2.033 33.3 66.7 0.0
all_ug 27 2.238 2.042 37.0 63.0 0.0
sf_ug_rls 27 2.244 2.061 37.0 59.3 0.0
all_ug_rls 27 2.313 2.122 44.4 55.6 0.0
peba6_ug 27 2.587 2.269 0.0 59.3 40.7
peba6_ug_rls 27 2.609 2.313 0.0 59.3 40.7
peba6 27 2.895 2.526 0.0 51.9 48.1
peba6_rls 27 2.900 2.559 0.0 51.9 48.1
peba4_ug_rls 27 2.971 2.630 0.0 51.9 48.1
peba4_ug 27 2.999 2.613 0.0 51.9 48.1
pols2_ug_rls 27 3.072 2.730 0.0 51.9 48.1
pols2_ug 27 3.135 2.739 0.0 51.9 48.1
peba4_rls 27 3.256 2.867 0.0 48.1 51.9
peba4 27 3.334 2.891 0.0 48.1 51.9
pols2_rls 27 3.415 3.008 0.0 48.1 51.9
pols2 27 3.462 3.006 0.0 48.1 51.9
peba2_ug 27 3.838 3.382 0.0 40.7 59.3
peba2_ug_rls 27 3.851 3.434 0.0 44.4 55.6
peba2_rls 27 4.158 3.704 0.0 33.3 63.0
peba2 27 4.264 3.706 0.0 33.3 66.7
sb_ug_rls 27 4.749 4.347 0.0 11.1 88.9
sb_ug 27 4.971 4.498 0.0 11.1 88.9
sb_rls 27 5.075 4.624 0.0 11.1 88.9
sb 27 5.359 4.839 0.0 7.4 92.6
std_rls 27 77.802 75.490 0.0 0.0 100.0
std_ml 27 78.487 76.298 0.0 0.0 100.0

PL1

Ranking all tests aggregated over sample sizes, dims, dfs:

test numcond RMSE MAD BEL2.5 ARR ABO7.5
peba4_ug_rls 27 0.535 0.402 0.0 100.0 0.0
peba2_ug_rls 27 0.545 0.409 0.0 100.0 0.0
pall_ug 27 0.546 0.393 0.0 100.0 0.0
pols2_ug_rls 27 0.546 0.413 0.0 100.0 0.0
peba6_ug_rls 27 0.553 0.428 0.0 100.0 0.0
pall_ug_rls 27 0.573 0.441 0.0 100.0 0.0
peba6_ug 27 0.577 0.420 0.0 100.0 0.0
sb_ug_rls 27 0.586 0.446 0.0 100.0 0.0
peba4_ug 27 0.596 0.435 0.0 100.0 0.0
pols2_ug 27 0.597 0.437 0.0 100.0 0.0
pall_rls 27 0.599 0.437 0.0 100.0 0.0
peba4_rls 27 0.603 0.450 0.0 100.0 0.0
peba6_rls 27 0.604 0.450 0.0 100.0 0.0
pols2_rls 27 0.606 0.456 0.0 100.0 0.0
peba2_rls 27 0.649 0.504 0.0 100.0 0.0
peba2_ug 27 0.683 0.502 0.0 100.0 0.0
ss 27 0.739 0.559 0.0 100.0 0.0
pall 27 0.761 0.580 0.0 100.0 0.0
peba6 27 0.790 0.598 0.0 100.0 0.0
sb_rls 27 0.826 0.617 0.0 100.0 0.0
pols2 27 0.827 0.620 0.0 100.0 0.0
peba4 27 0.828 0.622 0.0 100.0 0.0
sf 27 0.862 0.632 0.0 100.0 0.0
all 27 0.865 0.633 0.0 100.0 0.0
ss_ug 27 0.883 0.657 0.0 100.0 0.0
sb_ug 27 0.905 0.663 0.0 100.0 0.0
peba2 27 0.958 0.715 0.0 100.0 0.0
ss_rls 27 0.996 0.754 0.0 100.0 0.0
sf_ug 27 1.022 0.791 0.0 100.0 0.0
all_ug 27 1.025 0.791 0.0 100.0 0.0
all_rls 27 1.116 0.846 7.4 92.6 0.0
sf_rls 27 1.116 0.848 7.4 92.6 0.0
ss_ug_rls 27 1.141 0.870 7.4 92.6 0.0
sb_2001 27 1.232 0.830 11.1 88.9 0.0
sf_ug_rls 27 1.261 0.993 7.4 88.9 0.0
all_ug_rls 27 1.261 0.989 7.4 88.9 0.0
sb 27 1.262 0.917 0.0 92.6 7.4
std_rls 27 84.609 82.941 0.0 0.0 100.0
std_ml 27 85.046 83.424 0.0 0.0 100.0

PL2

Ranking all tests aggregated over sample sizes, dims, dfs:

test numcond RMSE MAD BEL2.5 ARR ABO7.5
ss_rls 27 1.072 0.856 3.7 96.3 0.0
ss_ug_rls 27 1.205 1.013 3.7 96.3 0.0
pall_ug 27 1.220 0.819 0.0 92.6 7.4
ss 27 1.269 1.024 3.7 96.3 0.0
sf_rls 27 1.278 1.069 3.7 96.3 0.0
all_rls 27 1.308 1.094 7.4 92.6 0.0
pall_ug_rls 27 1.374 0.817 0.0 92.6 7.4
ss_ug 27 1.397 1.133 11.1 88.9 0.0
sf_ug_rls 27 1.441 1.241 7.4 92.6 0.0
all_ug_rls 27 1.448 1.250 7.4 92.6 0.0
sf 27 1.485 1.194 11.1 88.9 0.0
all 27 1.495 1.206 11.1 88.9 0.0
peba6_ug 27 1.552 0.978 0.0 88.9 11.1
peba6_ug_rls 27 1.620 0.954 0.0 92.6 7.4
sf_ug 27 1.638 1.333 14.8 85.2 0.0
all_ug 27 1.654 1.356 14.8 85.2 0.0
pall 27 1.667 1.078 0.0 88.9 11.1
pall_rls 27 1.729 1.056 0.0 88.9 11.1
peba4_ug 27 1.736 1.074 0.0 88.9 11.1
pols2_ug 27 1.790 1.096 0.0 88.9 11.1
peba4_ug_rls 27 1.845 1.065 0.0 88.9 11.1
pols2_ug_rls 27 1.890 1.081 0.0 88.9 11.1
peba6 27 2.008 1.252 0.0 88.9 11.1
peba6_rls 27 2.074 1.228 0.0 88.9 11.1
peba4 27 2.201 1.356 0.0 85.2 14.8
peba4_rls 27 2.254 1.344 0.0 85.2 14.8
peba2_ug 27 2.257 1.374 0.0 88.9 11.1
pols2 27 2.276 1.376 0.0 85.2 14.8
peba2_ug_rls 27 2.308 1.350 0.0 81.5 18.5
pols2_rls 27 2.321 1.367 0.0 81.5 18.5
peba2 27 2.704 1.687 0.0 77.8 22.2
sb_2001 27 2.767 2.272 44.4 55.6 0.0
peba2_rls 27 2.896 1.706 0.0 77.8 22.2
sb_ug 27 3.055 1.993 0.0 74.1 25.9
sb_ug_rls 27 3.241 1.987 0.0 77.8 22.2
sb_rls 27 3.614 2.267 0.0 77.8 22.2
sb 27 3.684 2.389 0.0 70.4 29.6
std_rls 27 92.692 92.500 0.0 0.0 100.0
std_ml 27 92.803 92.620 0.0 0.0 100.0

Overall Non-normal winners

test numcond RMSE MAD BEL2.5 ARR ABO7.5
pall_ug 162 1.004 0.746 0.0 98.1 1.9
pall 162 1.237 0.896 0.0 95.7 4.3
ss_rls 162 1.515 1.183 2.5 96.3 0.6
peba6_ug 162 1.547 1.003 0.0 89.5 10.5
ss_ug_rls 162 1.585 1.300 5.6 91.4 0.6
ss 162 1.586 1.270 11.7 85.8 0.0
sf_rls 162 1.669 1.384 12.3 87.0 0.6
all_rls 162 1.690 1.413 12.3 86.4 0.6
ss_ug 162 1.732 1.403 16.0 82.7 0.0
peba4_ug 162 1.756 1.111 0.0 87.7 12.3
sf_ug_rls 162 1.773 1.515 14.2 83.3 0.6
all_ug_rls 162 1.800 1.541 15.4 82.7 0.6
peba6 162 1.820 1.195 0.0 87.0 13.0
pols2_ug 162 1.822 1.146 0.0 87.7 12.3
sf 162 1.841 1.488 20.4 79.0 0.0
all 162 1.853 1.504 21.0 79.0 0.0
sf_ug 162 1.973 1.627 24.7 74.7 0.0
all_ug 162 1.988 1.647 24.7 74.7 0.0
peba4 162 2.067 1.332 0.0 85.2 14.8
pols2 162 2.139 1.367 0.0 84.6 14.8
peba2_ug 162 2.282 1.447 0.0 83.3 16.7
sb_2001 162 2.284 1.736 26.5 72.8 0.6
pall_ug_rls 162 2.411 1.057 0.0 94.4 5.6
peba2 162 2.654 1.717 0.0 79.0 21.0
pall_rls 162 2.660 1.214 0.0 92.0 8.0
sb_ug 162 3.029 2.014 0.0 73.5 26.5
peba6_ug_rls 162 3.289 1.479 0.0 85.2 14.8
sb 162 3.430 2.324 0.0 66.0 32.7
peba4_ug_rls 162 3.532 1.616 0.0 82.7 16.7
peba6_rls 162 3.595 1.682 0.0 82.1 17.9
pols2_ug_rls 162 3.598 1.649 0.0 82.7 17.3
peba4_rls 162 3.844 1.824 0.0 79.6 19.8
pols2_rls 162 3.944 1.872 0.0 79.0 21.0
peba2_ug_rls 162 4.101 1.983 0.0 77.8 22.2
peba2_rls 162 4.440 2.230 0.0 75.3 24.1
sb_ug_rls 162 4.768 2.513 0.0 71.6 28.4
sb_rls 162 5.135 2.786 0.0 69.1 29.6
std_rls 162 81.226 78.278 0.0 0.0 100.0
std_ml 162 81.719 78.892 0.0 0.0 100.0

Overall winners

test numcond RMSE MAD BEL2.5 ARR ABO7.5
pall_ug 189 0.946 0.692 0.0 98.4 1.6
pall 189 1.172 0.843 0.0 96.3 3.7
ss_rls 189 1.418 1.071 2.1 96.8 0.5
peba6_ug 189 1.443 0.912 0.0 91.0 9.0
ss 189 1.482 1.147 10.1 87.8 0.0
ss_ug_rls 189 1.489 1.179 4.8 92.6 0.5
sf_rls 189 1.561 1.245 10.6 88.9 0.5
all_rls 189 1.580 1.270 10.6 88.4 0.5
ss_ug 189 1.612 1.250 13.8 85.2 0.0
peba4_ug 189 1.635 1.005 0.0 89.4 10.6
sf_ug_rls 189 1.661 1.365 12.2 85.7 0.5
all_ug_rls 189 1.685 1.387 13.2 85.2 0.5
pols2_ug 189 1.697 1.034 0.0 89.4 10.6
peba6 189 1.704 1.100 0.0 88.9 11.1
sf 189 1.716 1.330 17.5 82.0 0.0
all 189 1.727 1.344 18.0 82.0 0.0
sf_ug 189 1.834 1.441 21.2 78.3 0.0
all_ug 189 1.848 1.458 21.2 78.3 0.0
peba4 189 1.931 1.218 0.0 87.3 12.7
pols2 189 1.996 1.247 0.0 86.8 12.7
peba2_ug 189 2.121 1.293 0.0 85.7 14.3
sb_2001 189 2.131 1.566 22.8 76.7 0.5
pall_ug_rls 189 2.241 0.958 0.0 95.2 4.8
pall_rls 189 2.469 1.091 0.0 93.1 6.9
peba2 189 2.470 1.549 0.0 82.0 18.0
sb_ug 189 2.811 1.784 0.0 77.2 22.8
peba6_ug_rls 189 3.051 1.320 0.0 87.3 12.7
sb 189 3.188 2.077 0.0 70.9 28.0
peba4_ug_rls 189 3.276 1.437 0.0 85.2 14.3
peba6_rls 189 3.333 1.492 0.0 84.7 15.3
pols2_ug_rls 189 3.337 1.466 0.0 85.2 14.8
peba4_rls 189 3.564 1.614 0.0 82.5 16.9
pols2_rls 189 3.656 1.654 0.0 82.0 18.0
peba2_ug_rls 189 3.802 1.752 0.0 81.0 19.0
peba2_rls 189 4.114 1.963 0.0 78.8 20.6
sb_ug_rls 189 4.418 2.203 0.0 75.7 24.3
sb_rls 189 4.757 2.439 0.0 73.5 25.4
std_rls 189 75.201 67.147 0.0 14.3 85.7
std_ml 189 75.658 67.692 0.0 14.3 85.7