FMG Study 2 Measurement invariance testing

Design

A total of 189 conditions, with 2000 replications in each.

  • n=400, 800, 2000 (group sample sizes)
  • ngroups=2,4,8
  • One-factor model with dim=5,10, 20
  • distributions: normal, vm1, pl1, ig1, vm2, pl2, ig2

Overall rejection rates

method rr
ss_ug 0.0389127
ss_ug_rls 0.0402090
ss 0.0407011
ss_rls 0.0418730
peba6_ug 0.0578810
peba4_ug 0.0591508
pols2_ug 0.0594974
peba6 0.0605079
peba6_ug_rls 0.0612381
peba4 0.0619233
pols2 0.0622275
peba4_ug_rls 0.0625873
peba2_ug 0.0626667
pols2_ug_rls 0.0628836
peba6_rls 0.0637884
peba4_rls 0.0651667
pols2_rls 0.0655476
peba2 0.0656270
peba2_ug_rls 0.0663889
sb_ug 0.0680661
peba2_rls 0.0690794
sb 0.0711984
sb_ug_rls 0.0714894
sb_rls 0.0744656
std_rls 0.7212963
std_ml 0.7270635

Normal data

dim=5

method numcond RMSE MAD ARR
ss_ug_rls 9 0.5552777 0.4444444 100
peba4_ug_rls 9 0.5734884 0.4444444 100
peba6_ug_rls 9 0.5734884 0.4444444 100
pols2_ug_rls 9 0.5734884 0.4444444 100
peba2_ug_rls 9 0.5785518 0.4500000 100
sb_ug_rls 9 0.5790318 0.4611111 100
ss_rls 9 0.6242774 0.5277778 100
ss_ug 9 0.6437736 0.5444444 100
peba2_ug 9 0.6708204 0.5777778 100
peba4_ug 9 0.6708204 0.5777778 100

dim=10

method numcond RMSE MAD ARR
peba4_ug_rls 9 0.2229848 0.1833333 100
peba6_ug_rls 9 0.2236068 0.1888889 100
pols2_ug_rls 9 0.2236068 0.1888889 100
peba2_ug_rls 9 0.2254625 0.1944444 100
sb_ug_rls 9 0.2315407 0.2055556 100
ss_rls 9 0.2333333 0.2111111 100
ss_ug_rls 9 0.2697736 0.2555556 100
ss_ug 9 0.2773886 0.2055556 100
peba4_rls 9 0.2891559 0.2500000 100
peba6_rls 9 0.2891559 0.2500000 100

dim=20

method numcond RMSE MAD ARR
peba2_rls 9 0.3415650 0.2777778 100
peba4_rls 9 0.3431877 0.2777778 100
peba6_rls 9 0.3431877 0.2777778 100
pols2_rls 9 0.3431877 0.2777778 100
sb_rls 9 0.3574602 0.3000000 100
ss_ug 9 0.4112988 0.2944444 100
std_rls 9 0.4346135 0.3777778 100
sb_ug_rls 9 0.4377975 0.3555556 100
peba2_ug_rls 9 0.4881940 0.4111111 100
peba4_ug_rls 9 0.4980518 0.4166667 100

Overall normal data

method numcond RMSE MAD ARR
sb_ug_rls 27 0.4399074 0.3407407 100
peba2_ug_rls 27 0.4560296 0.3518519 100
peba4_ug_rls 27 0.4570436 0.3481481 100
peba6_ug_rls 27 0.4571449 0.3500000 100
pols2_ug_rls 27 0.4571449 0.3500000 100
ss_ug 27 0.4692389 0.3481481 100
peba4_rls 27 0.4726795 0.3722222 100
peba6_rls 27 0.4726795 0.3722222 100
pols2_rls 27 0.4726795 0.3722222 100
peba2_rls 27 0.4756088 0.3759259 100

Non-normal data

dim=5

method numcond RMSE MAD ARR
peba6_ug_rls 54 0.7874890 0.6342593 100.00000
peba4_ug_rls 54 0.8318876 0.6555556 100.00000
pols2_ug_rls 54 0.8534245 0.6703704 100.00000
peba6_ug 54 0.8544546 0.7203704 100.00000
peba4_ug 54 0.8888455 0.7398148 100.00000
peba6_rls 54 0.9020039 0.7092593 100.00000
pols2_ug 54 0.9190625 0.7620370 100.00000
peba4_rls 54 0.9528952 0.7379630 98.14815
pols2_rls 54 0.9905339 0.7675926 98.14815
peba6 54 1.0089415 0.8222222 100.00000

dim=10

method numcond RMSE MAD ARR
ss_rls 54 1.260254 1.0518519 98.14815
peba6_ug 54 1.289128 0.9444444 92.59259
ss_ug_rls 54 1.388928 1.1990741 96.29630
peba4_ug 54 1.454877 1.0611111 90.74074
ss 54 1.462431 1.1685185 88.88889
pols2_ug 54 1.497498 1.0814815 90.74074
peba6 54 1.541584 1.1629630 90.74074
ss_ug 54 1.617497 1.3203704 87.03704
peba4 54 1.716545 1.2972222 90.74074
pols2 54 1.762193 1.3231481 88.88889

dim=20

method numcond RMSE MAD ARR
ss_rls 54 2.005236 1.580556 90.74074
ss 54 2.031055 1.742593 74.07407
ss_ug_rls 54 2.047028 1.697222 87.03704
ss_ug 54 2.210999 1.944444 68.51852
peba6_ug 54 2.302595 1.574074 77.77778
peba4_ug 54 2.628926 1.797222 75.92593
peba6 54 2.707278 1.888889 75.92593
pols2_ug 54 2.728146 1.874074 74.07407
peba4 54 3.067346 2.151852 68.51852
pols2 54 3.187061 2.225000 66.66667

Overall non-normal data

method numcond RMSE MAD ARR
ss_rls 162 1.489236 1.143519 95.67901
ss_ug_rls 162 1.562919 1.254321 93.20988
ss 162 1.568198 1.225617 86.41975
peba6_ug 162 1.601446 1.079630 90.12346
ss_ug 162 1.712306 1.365741 83.33333
peba4_ug 162 1.809048 1.199383 88.88889
pols2_ug 162 1.873495 1.239197 88.27160
peba6 162 1.890661 1.291358 88.88889
peba4 162 2.120753 1.435185 86.41975
pols2 162 2.195791 1.475617 84.56790

By group sample size

n=400

method numcond RMSE MAD ARR
ss 63 1.528369 1.229365 88.88889
ss_rls 63 1.712153 1.219841 95.23810
ss_ug 63 1.743890 1.405556 82.53968
ss_ug_rls 63 1.790384 1.400000 90.47619
peba6_ug 63 2.090654 1.403968 84.12698
peba4_ug 63 2.384973 1.588889 82.53968
pols2_ug 63 2.466385 1.646825 80.95238
peba6 63 2.573414 1.852381 80.95238
peba4 63 2.893472 2.078571 76.19048
pols2 63 2.995095 2.130952 74.60317

n=800

method numcond RMSE MAD ARR
peba6_ug 63 1.048960 0.7873016 95.23810
peba4_ug 63 1.200083 0.8785714 95.23810
peba6 63 1.207598 0.9293651 95.23810
pols2_ug 63 1.240024 0.9007937 95.23810
ss_rls 63 1.369437 1.1246032 95.23810
peba4 63 1.383262 1.0444444 95.23810
pols2 63 1.430493 1.0722222 92.06349
ss_ug_rls 63 1.467235 1.2174603 93.65079
peba2_ug 63 1.606386 1.1619048 90.47619
ss 63 1.639904 1.2619048 82.53968

n=2000

method numcond RMSE MAD ARR
ss_rls 63 1.000873 0.7682540 98.41270
ss_ug_rls 63 1.034274 0.8055556 98.41270
peba6_ug 63 1.118140 0.7761905 95.23810
peba6 63 1.165186 0.8055556 95.23810
peba4_ug 63 1.189004 0.8079365 93.65079
ss 63 1.202214 0.8682540 93.65079
peba4 63 1.236547 0.8349206 93.65079
pols2_ug 63 1.237573 0.8301587 93.65079
ss_ug 63 1.240632 0.9071429 93.65079
pols2 63 1.274786 0.8571429 93.65079

By distribution

normal

method numcond RMSE MAD ARR
sb_ug_rls 27 0.4399074 0.3407407 100
peba2_ug_rls 27 0.4560296 0.3518519 100
peba4_ug_rls 27 0.4570436 0.3481481 100
peba6_ug_rls 27 0.4571449 0.3500000 100
pols2_ug_rls 27 0.4571449 0.3500000 100
ss_ug 27 0.4692389 0.3481481 100
peba4_rls 27 0.4726795 0.3722222 100
peba6_rls 27 0.4726795 0.3722222 100
pols2_rls 27 0.4726795 0.3722222 100
peba2_rls 27 0.4756088 0.3759259 100

vm1

method numcond RMSE MAD ARR
peba6_ug 27 0.4182193 0.3240741 100.0000
peba4_ug 27 0.4493823 0.3537037 100.0000
pols2_ug 27 0.4694362 0.3703704 100.0000
peba6 27 0.5900408 0.4370370 100.0000
peba2_ug 27 0.6684005 0.5055556 100.0000
peba4 27 0.6890009 0.5092593 100.0000
pols2 27 0.6962864 0.5074074 100.0000
peba2 27 0.9456528 0.6814815 96.2963
sb_ug 27 1.0723805 0.7703704 96.2963
ss_rls 27 1.1543396 1.0611111 100.0000

vm2

method numcond RMSE MAD ARR
peba6_ug 27 1.944055 1.090741 88.88889
ss_ug_rls 27 2.161211 1.720370 92.59259
peba4_ug 27 2.232172 1.240741 88.88889
ss_rls 27 2.254974 1.672222 96.29630
peba6 27 2.287749 1.272222 88.88889
pols2_ug 27 2.312927 1.285185 85.18519
ss 27 2.482793 2.277778 55.55556
peba4 27 2.619196 1.448148 85.18519
ss_ug 27 2.621704 2.425926 48.14815
pols2 27 2.727467 1.518519 81.48148

pl1

method numcond RMSE MAD ARR
peba4_ug_rls 27 0.5833333 0.4537037 100
peba6_ug_rls 27 0.5925463 0.4629630 100
pols2_ug_rls 27 0.5935611 0.4648148 100
peba2_ug_rls 27 0.6020028 0.4851852 100
sb_ug_rls 27 0.6426940 0.5166667 100
peba6_rls 27 0.6569430 0.5277778 100
peba4_rls 27 0.6602469 0.5333333 100
pols2_rls 27 0.6669444 0.5370370 100
peba6_ug 27 0.6790516 0.5444444 100
peba4_ug 27 0.6936217 0.5518519 100

pl2

method numcond RMSE MAD ARR
ss_rls 27 0.9684849 0.7407407 100.00000
ss_ug_rls 27 1.0984838 0.8777778 100.00000
ss 27 1.1294050 0.8222222 96.29630
ss_ug 27 1.2962196 0.9777778 88.88889
peba6_ug 27 1.6711883 1.1462963 88.88889
peba6_ug_rls 27 1.6884137 1.0851852 88.88889
peba4_ug 27 1.8691204 1.2685185 88.88889
peba4_ug_rls 27 1.9038946 1.2037037 88.88889
pols2_ug 27 1.9309852 1.2962963 88.88889
pols2_ug_rls 27 1.9715805 1.2314815 88.88889

ig1

method numcond RMSE MAD ARR
ss 27 0.8655441 0.6500000 100.0000
peba6_ug_rls 27 1.0016191 0.8722222 100.0000
ss_ug 27 1.0426107 0.7888889 100.0000
peba4_ug_rls 27 1.0502204 0.9259259 100.0000
pols2_ug_rls 27 1.0634065 0.9351852 100.0000
peba6_rls 27 1.1380213 1.0351852 100.0000
ss_rls 27 1.1399724 0.8796296 96.2963
peba4_rls 27 1.2024666 1.1000000 100.0000
pols2_rls 27 1.2137713 1.1166667 100.0000
peba2_ug_rls 27 1.2579010 1.1537037 100.0000

ig2

method numcond RMSE MAD ARR
ss 27 1.812304 1.566667 81.48148
ss_rls 27 1.881390 1.644444 81.48148
ss_ug 27 1.953914 1.703704 81.48148
ss_ug_rls 27 1.987391 1.742593 77.77778
peba6_ug_rls 27 2.552214 2.227778 66.66667
peba6_ug 27 2.562171 2.201852 62.96296
peba6_rls 27 2.853117 2.468519 59.25926
peba6 27 2.857868 2.448148 59.25926
peba4_ug_rls 27 2.923183 2.544444 59.25926
peba4_ug 27 2.947189 2.540741 55.55556

By number of groups

2 groups

method numcond RMSE MAD ARR
ss_rls 63 0.8510031 0.7047619 100.00000
ss 63 0.8532617 0.6801587 100.00000
ss_ug_rls 63 0.9306296 0.7690476 100.00000
ss_ug 63 0.9484532 0.7357143 98.41270
peba6_ug_rls 63 1.0324497 0.7301587 95.23810
peba6_ug 63 1.0407377 0.7960317 95.23810
peba4_ug_rls 63 1.0918348 0.7722222 93.65079
peba6_rls 63 1.1025763 0.7896825 93.65079
peba4_ug 63 1.1270132 0.8507937 95.23810
pols2_ug_rls 63 1.1373005 0.7896825 93.65079

4 groups

method numcond RMSE MAD ARR
peba6_ug 63 1.306167 0.9373016 93.65079
ss_rls 63 1.386643 1.1587302 96.82540
ss_ug_rls 63 1.491431 1.2730159 95.23810
peba4_ug 63 1.510046 1.0380952 90.47619
peba6 63 1.530173 1.0873016 92.06349
pols2_ug 63 1.576262 1.0761905 90.47619
ss 63 1.632896 1.2666667 84.12698
ss_ug 63 1.752425 1.3849206 82.53968
peba4 63 1.766038 1.2190476 88.88889
pols2 63 1.841131 1.2603175 87.30159

8 groups

method numcond RMSE MAD ARR
ss 63 1.753885 1.412698 80.95238
ss_rls 63 1.778085 1.249206 92.06349
ss_ug_rls 63 1.826893 1.380952 87.30159
ss_ug 63 1.913952 1.540476 76.19048
peba6_ug 63 1.982973 1.234127 85.71429
peba4_ug 63 2.234195 1.386508 85.71429
pols2_ug 63 2.308817 1.433333 85.71429
peba6 63 2.394289 1.573016 84.12698
peba4 63 2.660715 1.750000 84.12698
pols2 63 2.751738 1.800000 80.95238

OVERALL

By RMSE

method numcond RMSE MAD ARR
ss_rls 189 1.391471 1.0375661 96.29630
ss_ug_rls 189 1.463782 1.1410053 94.17989
ss 189 1.468618 1.1198413 88.35979
peba6_ug 189 1.496818 0.9891534 91.53439
ss_ug 189 1.595177 1.2203704 85.71429
peba4_ug 189 1.687407 1.0917989 90.47619
pols2_ug 189 1.746645 1.1259259 89.94709
peba6 189 1.773735 1.1957672 90.47619
peba4 189 1.984493 1.3193122 88.35979
pols2 189 2.052802 1.3534392 86.77249
peba2_ug 189 2.188667 1.3973545 85.71429
peba2 189 2.528172 1.6500000 80.42328
sb_ug 189 2.875437 1.8780423 75.66138
peba6_ug_rls 189 3.074576 1.3470899 88.35979
sb 189 3.245446 2.1664021 70.37037
peba4_ug_rls 189 3.290760 1.4619048 86.24339
peba6_rls 189 3.361819 1.5328042 85.18519
pols2_ug_rls 189 3.364678 1.4957672 86.24339
peba4_rls 189 3.601181 1.6563492 83.59788
pols2_rls 189 3.684784 1.6986772 83.59788
peba2_ug_rls 189 3.845245 1.8034392 81.48148
peba2_rls 189 4.174659 2.0206349 79.36508
sb_ug_rls 189 4.475756 2.2584656 76.19048
sb_rls 189 4.827799 2.5153439 73.01587
std_rls 189 75.230164 67.1793651 14.28571
std_ml 189 75.685896 67.7264550 14.28571

By MAD

method numcond RMSE MAD ARR
peba6_ug 189 1.496818 0.9891534 91.53439
ss_rls 189 1.391471 1.0375661 96.29630
peba4_ug 189 1.687407 1.0917989 90.47619
ss 189 1.468618 1.1198413 88.35979
pols2_ug 189 1.746645 1.1259259 89.94709
ss_ug_rls 189 1.463782 1.1410053 94.17989
peba6 189 1.773735 1.1957672 90.47619
ss_ug 189 1.595177 1.2203704 85.71429
peba4 189 1.984493 1.3193122 88.35979
peba6_ug_rls 189 3.074576 1.3470899 88.35979
pols2 189 2.052802 1.3534392 86.77249
peba2_ug 189 2.188667 1.3973545 85.71429
peba4_ug_rls 189 3.290760 1.4619048 86.24339
pols2_ug_rls 189 3.364678 1.4957672 86.24339
peba6_rls 189 3.361819 1.5328042 85.18519
peba2 189 2.528172 1.6500000 80.42328
peba4_rls 189 3.601181 1.6563492 83.59788
pols2_rls 189 3.684784 1.6986772 83.59788
peba2_ug_rls 189 3.845245 1.8034392 81.48148
sb_ug 189 2.875437 1.8780423 75.66138
peba2_rls 189 4.174659 2.0206349 79.36508
sb 189 3.245446 2.1664021 70.37037
sb_ug_rls 189 4.475756 2.2584656 76.19048
sb_rls 189 4.827799 2.5153439 73.01587
std_rls 189 75.230164 67.1793651 14.28571
std_ml 189 75.685896 67.7264550 14.28571

By ARR

method numcond RMSE MAD ARR
ss_rls 189 1.391471 1.0375661 96.29630
ss_ug_rls 189 1.463782 1.1410053 94.17989
peba6_ug 189 1.496818 0.9891534 91.53439
peba4_ug 189 1.687407 1.0917989 90.47619
peba6 189 1.773735 1.1957672 90.47619
pols2_ug 189 1.746645 1.1259259 89.94709
peba4 189 1.984493 1.3193122 88.35979
peba6_ug_rls 189 3.074576 1.3470899 88.35979
ss 189 1.468618 1.1198413 88.35979
pols2 189 2.052802 1.3534392 86.77249
peba4_ug_rls 189 3.290760 1.4619048 86.24339
pols2_ug_rls 189 3.364678 1.4957672 86.24339
peba2_ug 189 2.188667 1.3973545 85.71429
ss_ug 189 1.595177 1.2203704 85.71429
peba6_rls 189 3.361819 1.5328042 85.18519
peba4_rls 189 3.601181 1.6563492 83.59788
pols2_rls 189 3.684784 1.6986772 83.59788
peba2_ug_rls 189 3.845245 1.8034392 81.48148
peba2 189 2.528172 1.6500000 80.42328
peba2_rls 189 4.174659 2.0206349 79.36508
sb_ug_rls 189 4.475756 2.2584656 76.19048
sb_ug 189 2.875437 1.8780423 75.66138
sb_rls 189 4.827799 2.5153439 73.01587
sb 189 3.245446 2.1664021 70.37037
std_ml 189 75.685896 67.7264550 14.28571
std_rls 189 75.230164 67.1793651 14.28571

ARR2 to penalize underrejection

Instead of \(2.5 - 7.5\) let us use \(3-8\) to penalize underrejection.

method numcond RMSE MAD ARR ARR2
peba6_ug 189 1.496818 0.9891534 91.53439 93.65079
peba4_ug 189 1.687407 1.0917989 90.47619 93.12169
peba6 189 1.773735 1.1957672 90.47619 93.12169
pols2_ug 189 1.746645 1.1259259 89.94709 91.53439
peba6_ug_rls 189 3.074576 1.3470899 88.35979 91.00529
peba4 189 1.984493 1.3193122 88.35979 90.47619
peba4_ug_rls 189 3.290760 1.4619048 86.24339 89.94709
peba6_rls 189 3.361819 1.5328042 85.18519 89.41799
pols2 189 2.052802 1.3534392 86.77249 89.41799
peba2_ug 189 2.188667 1.3973545 85.71429 88.88889
pols2_ug_rls 189 3.364678 1.4957672 86.24339 88.88889
peba4_rls 189 3.601181 1.6563492 83.59788 87.30159
ss_rls 189 1.391471 1.0375661 96.29630 87.30159
peba2 189 2.528172 1.6500000 80.42328 86.77249
pols2_rls 189 3.684784 1.6986772 83.59788 86.24339
peba2_ug_rls 189 3.845245 1.8034392 81.48148 85.18519
ss_ug_rls 189 1.463782 1.1410053 94.17989 82.53968
peba2_rls 189 4.174659 2.0206349 79.36508 82.01058
sb_ug 189 2.875437 1.8780423 75.66138 80.95238
ss 189 1.468618 1.1198413 88.35979 78.83598
sb_ug_rls 189 4.475756 2.2584656 76.19048 77.77778
sb_rls 189 4.827799 2.5153439 73.01587 76.71958
sb 189 3.245446 2.1664021 70.37037 76.19048
ss_ug 189 1.595177 1.2203704 85.71429 76.19048
std_ml 189 75.685896 67.7264550 14.28571 14.28571
std_rls 189 75.230164 67.1793651 14.28571 14.28571

postscript on Unbiased Gamma

overall

ub numcond RMSE MAD ARR
FALSE 2268 3.040002 1.605511 83.81834
TRUE 2268 2.773471 1.434193 85.97884

By sample size

n ub numcond RMSE MAD ARR
400 FALSE 756 4.451075 2.4063492 73.28042
400 TRUE 756 4.004687 2.0449074 77.51323
800 FALSE 756 2.343440 1.4253968 86.50794
800 TRUE 756 2.184184 1.2965608 88.09524
2000 FALSE 756 1.555973 0.9847884 91.66667
2000 TRUE 756 1.506070 0.9611111 92.32804

By test

All tests, except ss, seem to benefit from ug?

method numcond RMSE MAD ARR
peba2 189 2.528172 1.6500000 80.42328
peba2_ug 189 2.188667 1.3973545 85.71429
peba2_rls 189 4.174659 2.0206349 79.36508
peba2_ug_rls 189 3.845245 1.8034392 81.48148
peba4 189 1.984493 1.3193122 88.35979
peba4_ug 189 1.687407 1.0917989 90.47619
peba4_rls 189 3.601181 1.6563492 83.59788
peba4_ug_rls 189 3.290760 1.4619048 86.24339
peba6 189 1.773735 1.1957672 90.47619
peba6_ug 189 1.496818 0.9891534 91.53439
peba6_rls 189 3.361819 1.5328042 85.18519
peba6_ug_rls 189 3.074576 1.3470899 88.35979
pols2 189 2.052802 1.3534392 86.77249
pols2_ug 189 1.746645 1.1259259 89.94709
pols2_rls 189 3.684784 1.6986772 83.59788
pols2_ug_rls 189 3.364678 1.4957672 86.24339
sb 189 3.245446 2.1664021 70.37037
sb_ug 189 2.875437 1.8780423 75.66138
sb_rls 189 4.827799 2.5153439 73.01587
sb_ug_rls 189 4.475756 2.2584656 76.19048
ss 189 1.468618 1.1198413 88.35979
ss_ug 189 1.595177 1.2203704 85.71429
ss_rls 189 1.391471 1.0375661 96.29630
ss_ug_rls 189 1.463782 1.1410053 94.17989

CHANGING the Bradley criterion in ARR