FMG Study 1 CFA model testing

Design

A total of 84 conditions, with 2000 replications in each.

  • n=400, 800, 2000
  • Five-factor model with dim=15,30, 60, 100
  • distributions: normal, vm1, pl1, ig1, vm2, pl2, ig2

Normal data

dim=15

method numcond RMSE MAD ARR
std_rls 3 0.2738613 0.2666667 100
peba6_ug 3 0.3227486 0.3166667 100
peba4_ug 3 0.3391165 0.3333333 100
pols2_ug 3 0.3391165 0.3333333 100
sb_rls 3 0.3947573 0.3833333 100
peba2_ug 3 0.4406435 0.4166667 100
peba4 3 0.5212165 0.4666667 100
sb_ug 3 0.5212165 0.4666667 100
peba6 3 0.5267827 0.4833333 100
pols2 3 0.5267827 0.4833333 100

dim=30

method numcond RMSE MAD ARR
sb_ug_rls 3 0.4358899 0.3666667 100
peba2_rls 3 0.5809475 0.3833333 100
std_rls 3 0.7193747 0.6166667 100
sb_rls 3 0.7794229 0.6500000 100
pols2_rls 3 0.9721111 0.7666667 100
peba4_rls 3 0.9836158 0.7833333 100
peba6_rls 3 1.0218447 0.8166667 100
peba2_ug_rls 3 1.1528949 0.9500000 100
peba4_ug_rls 3 1.3677841 1.2166667 100
pols2_ug_rls 3 1.3723459 1.2333333 100

dim=60

method numcond RMSE MAD ARR
std_rls 3 0.6770032 0.5666667 100.00000
sb_ug_rls 3 0.8981462 0.8666667 100.00000
sb_rls 3 1.1184066 0.9166667 100.00000
peba2_rls 3 1.4913640 1.3833333 100.00000
peba2_ug_rls 3 2.2386752 2.2333333 100.00000
peba4_rls 3 2.6043233 2.5833333 33.33333
peba6_rls 3 2.9281678 2.8833333 33.33333
pols2_rls 3 2.9487285 2.8666667 33.33333
ss 3 3.0357591 3.0166667 0.00000
peba4_ug_rls 3 3.1158198 3.0666667 33.33333

dim=100

method numcond RMSE MAD ARR
std_rls 3 0.5787918 0.5333333 100.00000
peba2_rls 3 0.6976150 0.6000000 100.00000
sb_ug_rls 3 1.1273124 0.8833333 100.00000
sb_rls 3 1.7019597 1.5666667 66.66667
peba2_ug_rls 3 2.2907422 2.2500000 66.66667
peba4_rls 3 2.8109903 2.8000000 0.00000
peba6_rls 3 3.5677958 3.5500000 0.00000
peba4_ug_rls 3 3.6378794 3.6166667 0.00000
ss_ug 3 4.0851357 4.0666667 0.00000
peba6_ug_rls 3 4.1269440 4.1000000 0.00000

Overall normal data

method numcond RMSE MAD ARR
std_rls 12 0.5886071 0.4958333 100.00000
sb_ug_rls 12 0.8243684 0.6750000 100.00000
peba2_rls 12 0.9278110 0.7333333 100.00000
sb_rls 12 1.1080200 0.8791667 91.66667
peba2_ug_rls 12 1.7544586 1.5208333 91.66667
peba4_rls 12 2.0063960 1.6875000 58.33333
peba6_rls 12 2.3887758 1.9583333 58.33333
peba4_ug_rls 12 2.5290809 2.1416667 58.33333
pols2_rls 12 2.7224376 2.1416667 58.33333
peba6_ug_rls 12 2.7780764 2.3291667 58.33333

Non-normal data

dim=15

method numcond RMSE MAD ARR
pols2 18 0.3921097 0.3361111 100
peba2_ug 18 0.3980857 0.3250000 100
peba4 18 0.4387482 0.3611111 100
peba6 18 0.5358016 0.3916667 100
pols2_ug 18 0.5433947 0.4166667 100
sb_ug_rls 18 0.5571505 0.4861111 100
peba4_ug 18 0.5899623 0.4388889 100
sb_rls 18 0.6106690 0.4750000 100
peba2_rls 18 0.6145233 0.4861111 100
peba2 18 0.6482883 0.4888889 100

dim=30

method numcond RMSE MAD ARR
peba6_ug 18 1.007472 0.8111111 94.44444
pols2_rls 18 1.121445 0.9027778 100.00000
peba4_rls 18 1.129098 0.9194444 94.44444
peba6_rls 18 1.256262 0.9472222 94.44444
peba4_ug_rls 18 1.370422 1.0666667 88.88889
pols2_ug_rls 18 1.394334 1.1388889 94.44444
peba2_ug_rls 18 1.469363 1.2750000 100.00000
peba6_ug_rls 18 1.633928 1.3111111 83.33333
pols2_ug 18 1.721514 1.3944444 88.88889
peba6 18 1.831287 1.4666667 83.33333

dim=60

method numcond RMSE MAD ARR
peba4_ug_rls 18 2.003747 1.627778 72.22222
peba6_rls 18 2.049593 1.744444 66.66667
peba4_rls 18 2.210612 1.769444 83.33333
peba6_ug_rls 18 2.548338 2.280556 61.11111
peba2_ug_rls 18 2.691963 2.100000 77.77778
pols2_rls 18 2.751792 2.475000 50.00000
pols2_ug_rls 18 3.247542 2.997222 27.77778
peba2_rls 18 4.559940 3.022222 66.66667
sb_ug_rls 18 4.585060 3.288889 50.00000
ss 18 4.832112 4.825000 0.00000

dim=100

method numcond RMSE MAD ARR
peba6_ug_rls 18 3.334208 3.105556 33.33333
peba4_ug_rls 18 3.964565 3.150000 38.88889
pols2_rls 18 4.494472 4.466667 0.00000
peba6_rls 18 4.566439 3.469444 44.44444
pols2_ug_rls 18 4.652553 4.641667 0.00000
ss 18 4.948218 4.947222 0.00000
ss_ug 18 4.988960 4.988889 0.00000
ss_rls 18 5.000000 5.000000 0.00000
ss_ug_rls 18 5.000000 5.000000 0.00000
peba2_ug_rls 18 6.345712 3.938889 61.11111

Overall non-normal data

method numcond RMSE MAD ARR
peba6_ug_rls 72 2.361717 1.977778 66.666667
peba4_ug_rls 72 2.412986 1.729167 73.611111
peba6_rls 72 2.643631 1.774306 75.000000
pols2_rls 72 2.734902 2.150694 62.500000
pols2_ug_rls 72 2.989147 2.455556 54.166667
peba2_ug_rls 72 3.549227 1.998611 84.722222
peba4_rls 72 3.592223 1.856250 86.111111
ss 72 4.325257 4.181944 11.111111
ss_ug 72 4.377543 4.242361 9.722222
ss_rls 72 4.468311 4.350694 8.333333

By sample size

n=400

method numcond RMSE MAD ARR
peba6_ug_rls 28 3.008084 2.714286 50.000000
pols2_rls 28 3.054110 2.580357 53.571429
peba4_ug_rls 28 3.320459 2.594643 57.142857
pols2_ug_rls 28 3.454345 3.110714 42.857143
peba6_rls 28 3.692923 2.553571 64.285714
ss_ug 28 4.527702 4.444643 3.571429
ss_rls 28 4.699611 4.655357 3.571429
ss_ug_rls 28 4.744179 4.712500 0.000000
peba2_ug_rls 28 4.824297 2.812500 78.571429
ss 28 4.983777 4.603571 7.142857

n=800

method numcond RMSE MAD ARR
peba6_rls 28 2.088211 1.662500 67.85714
peba4_ug_rls 28 2.128883 1.685714 71.42857
peba6_ug_rls 28 2.350988 1.996429 64.28571
peba4_rls 28 2.372141 1.619643 85.71429
pols2_rls 28 2.812377 2.228571 57.14286
peba2_ug_rls 28 2.877871 1.828571 85.71429
pols2_ug_rls 28 3.020850 2.489286 50.00000
ss 28 4.177833 3.975000 17.85714
peba2_rls 28 4.210436 2.283929 71.42857
ss_ug 28 4.252510 4.073214 17.85714

n=2000

method numcond RMSE MAD ARR
peba4_rls 28 1.285461 0.9732143 92.85714
peba2_ug_rls 28 1.468174 1.1500000 92.85714
peba4_ug_rls 28 1.468266 1.0839286 85.71429
peba6_rls 28 1.555118 1.1857143 85.71429
peba6_ug_rls 28 1.753440 1.3732143 82.14286
peba2_rls 28 1.765467 1.2196429 92.85714
pols2_ug 28 1.906029 1.3017857 85.71429
peba6_ug 28 2.090113 1.2464286 85.71429
pols2 28 2.101105 1.3678571 85.71429
pols2_rls 28 2.274392 1.6392857 75.00000

By distribution

normal

method numcond RMSE MAD ARR
std_rls 12 0.5886071 0.4958333 100.00000
sb_ug_rls 12 0.8243684 0.6750000 100.00000
peba2_rls 12 0.9278110 0.7333333 100.00000
sb_rls 12 1.1080200 0.8791667 91.66667
peba2_ug_rls 12 1.7544586 1.5208333 91.66667
peba4_rls 12 2.0063960 1.6875000 58.33333
peba6_rls 12 2.3887758 1.9583333 58.33333
peba4_ug_rls 12 2.5290809 2.1416667 58.33333
pols2_rls 12 2.7224376 2.1416667 58.33333
peba6_ug_rls 12 2.7780764 2.3291667 58.33333

vm1

method numcond RMSE MAD ARR
peba2_ug_rls 12 0.8856495 0.7708333 100.00000
sb_ug_rls 12 1.1344970 0.9333333 100.00000
peba4_rls 12 1.3664888 1.2041667 100.00000
peba2_rls 12 1.8383643 1.1666667 91.66667
peba6_rls 12 2.0937904 1.8875000 66.66667
peba4_ug_rls 12 2.1092060 1.9416667 66.66667
peba6_ug_rls 12 2.7485602 2.4416667 50.00000
pols2_rls 12 2.8870400 2.3250000 58.33333
pols2_ug_rls 12 3.0771334 2.6083333 58.33333
sb_rls 12 3.6573271 2.4791667 66.66667

vm2

method numcond RMSE MAD ARR
peba6_ug_rls 12 2.046593 1.495833 83.333333
pols2_rls 12 2.213500 1.758333 75.000000
pols2_ug_rls 12 2.623849 2.166667 58.333333
peba4_ug_rls 12 3.701604 2.079167 83.333333
peba6_rls 12 4.196700 2.195833 83.333333
ss 12 4.578460 4.495833 8.333333
ss_ug 12 4.609998 4.533333 8.333333
ss_rls 12 4.630582 4.562500 8.333333
ss_ug_rls 12 4.642220 4.579167 8.333333
peba4_rls 12 6.836133 3.645833 75.000000

pl1

method numcond RMSE MAD ARR
sb_ug_rls 12 0.9484637 0.8166667 100.00000
peba2_ug_rls 12 1.1137399 0.8666667 100.00000
peba2_rls 12 1.5092769 0.8583333 91.66667
peba4_rls 12 1.6093477 1.3000000 91.66667
peba6_rls 12 2.3211438 1.9375000 58.33333
peba4_ug_rls 12 2.3529680 2.0208333 50.00000
peba6_ug_rls 12 2.7865750 2.3916667 50.00000
pols2_rls 12 2.9887288 2.3250000 58.33333
pols2_ug_rls 12 3.1798519 2.5958333 50.00000
sb_rls 12 3.1842058 2.1250000 66.66667

pl2

method numcond RMSE MAD ARR
peba6_ug_rls 12 1.614388 1.491667 91.66667
peba4_ug_rls 12 2.011788 1.312500 83.33333
pols2_rls 12 2.624246 2.108333 58.33333
peba6_rls 12 2.739944 1.687500 83.33333
pols2_ug_rls 12 2.993082 2.520833 41.66667
peba2_ug_rls 12 4.520440 3.104167 58.33333
ss 12 4.625496 4.570833 0.00000
peba4_rls 12 4.646101 2.458333 75.00000
ss_ug 12 4.647894 4.600000 0.00000
ss_rls 12 4.672682 4.629167 0.00000

ig1

method numcond RMSE MAD ARR
sb_ug_rls 12 0.5762667 0.5083333 100.00000
peba2_rls 12 0.7558108 0.6333333 100.00000
peba2_ug_rls 12 1.3496913 1.1333333 91.66667
peba4_rls 12 1.7949002 1.5250000 83.33333
sb_rls 12 2.0474069 1.5708333 75.00000
peba6_rls 12 2.4079296 2.0208333 58.33333
peba4_ug_rls 12 2.4878873 2.1416667 58.33333
peba6_ug_rls 12 2.8974486 2.5208333 41.66667
pols2_rls 12 2.9808765 2.3375000 58.33333
pols2_ug_rls 12 3.1795571 2.6333333 58.33333

ig2

method numcond RMSE MAD ARR
peba4_ug_rls 12 1.005506 0.8791667 100.00000
peba6_rls 12 1.115049 0.9166667 100.00000
peba4_rls 12 1.194344 1.0041667 91.66667
peba2_ug_rls 12 1.654728 1.5541667 100.00000
peba6_ug_rls 12 1.719496 1.5250000 83.33333
pols2_rls 12 2.634151 2.0500000 66.66667
pols2_ug_rls 12 2.842021 2.2083333 58.33333
sb_ug_rls 12 3.662735 3.2125000 25.00000
peba2_rls 12 3.773592 2.9583333 50.00000
ss 12 4.457952 4.4000000 0.00000

OVERALL

By RMSE

method numcond RMSE MAD ARR
peba6_ug_rls 84 2.425577 2.027976 65.476191
peba4_ug_rls 84 2.429910 1.788095 71.428571
peba6_rls 84 2.608748 1.800595 72.619048
pols2_rls 84 2.733125 2.149405 61.904762
pols2_ug_rls 84 2.986637 2.453571 54.761905
peba2_ug_rls 84 3.352189 1.930357 85.714286
peba4_rls 84 3.411116 1.832143 82.142857
ss_ug 84 4.205938 4.006548 14.285714
ss 84 4.332054 4.000595 16.666667
ss_rls 84 4.377860 4.213095 11.904762
ss_ug_rls 84 4.417383 4.269048 9.523810
sb_ug_rls 84 4.815396 2.614286 67.857143
peba2_rls 84 5.575571 2.660119 75.000000
sb_rls 84 7.485981 4.204167 53.571429
pols2_ug 84 19.418397 7.517262 65.476191
pols2 84 22.475817 9.345833 58.333333
peba6_ug 84 27.637414 12.186310 64.285714
peba6 84 29.495759 13.968452 54.761905
peba4_ug 84 29.735216 14.285714 53.571429
peba4 84 31.701241 16.191071 47.619048
peba2_ug 84 32.782694 17.713095 41.666667
sb_ug 84 35.059250 20.619643 32.142857
peba2 84 35.125002 19.936310 36.904762
sb 84 37.450730 22.932738 28.571429
std_rls 84 62.710965 49.160714 16.666667
std_ml 84 69.020651 57.887500 7.142857

By MAD

method numcond RMSE MAD ARR
peba4_ug_rls 84 2.429910 1.788095 71.428571
peba6_rls 84 2.608748 1.800595 72.619048
peba4_rls 84 3.411116 1.832143 82.142857
peba2_ug_rls 84 3.352189 1.930357 85.714286
peba6_ug_rls 84 2.425577 2.027976 65.476191
pols2_rls 84 2.733125 2.149405 61.904762
pols2_ug_rls 84 2.986637 2.453571 54.761905
sb_ug_rls 84 4.815396 2.614286 67.857143
peba2_rls 84 5.575571 2.660119 75.000000
ss 84 4.332054 4.000595 16.666667
ss_ug 84 4.205938 4.006548 14.285714
sb_rls 84 7.485981 4.204167 53.571429
ss_rls 84 4.377860 4.213095 11.904762
ss_ug_rls 84 4.417383 4.269048 9.523810
pols2_ug 84 19.418397 7.517262 65.476191
pols2 84 22.475817 9.345833 58.333333
peba6_ug 84 27.637414 12.186310 64.285714
peba6 84 29.495759 13.968452 54.761905
peba4_ug 84 29.735216 14.285714 53.571429
peba4 84 31.701241 16.191071 47.619048
peba2_ug 84 32.782694 17.713095 41.666667
peba2 84 35.125002 19.936310 36.904762
sb_ug 84 35.059250 20.619643 32.142857
sb 84 37.450730 22.932738 28.571429
std_rls 84 62.710965 49.160714 16.666667
std_ml 84 69.020651 57.887500 7.142857

By ARR

method numcond RMSE MAD ARR
peba2_ug_rls 84 3.352189 1.930357 85.714286
peba4_rls 84 3.411116 1.832143 82.142857
peba2_rls 84 5.575571 2.660119 75.000000
peba6_rls 84 2.608748 1.800595 72.619048
peba4_ug_rls 84 2.429910 1.788095 71.428571
sb_ug_rls 84 4.815396 2.614286 67.857143
peba6_ug_rls 84 2.425577 2.027976 65.476191
pols2_ug 84 19.418397 7.517262 65.476191
peba6_ug 84 27.637414 12.186310 64.285714
pols2_rls 84 2.733125 2.149405 61.904762
pols2 84 22.475817 9.345833 58.333333
peba6 84 29.495759 13.968452 54.761905
pols2_ug_rls 84 2.986637 2.453571 54.761905
peba4_ug 84 29.735216 14.285714 53.571429
sb_rls 84 7.485981 4.204167 53.571429
peba4 84 31.701241 16.191071 47.619048
peba2_ug 84 32.782694 17.713095 41.666667
peba2 84 35.125002 19.936310 36.904762
sb_ug 84 35.059250 20.619643 32.142857
sb 84 37.450730 22.932738 28.571429
ss 84 4.332054 4.000595 16.666667
std_rls 84 62.710965 49.160714 16.666667
ss_ug 84 4.205938 4.006548 14.285714
ss_rls 84 4.377860 4.213095 11.904762
ss_ug_rls 84 4.417383 4.269048 9.523810
std_ml 84 69.020651 57.887500 7.142857

postscript on Unbiased Gamma

overall

ub numcond RMSE MAD ARR
FALSE 1008 20.75135 8.602877 50.00000
TRUE 1008 19.19478 7.617659 52.18254

By sample size

n ub numcond RMSE MAD ARR
400 FALSE 336 31.848980 15.871429 36.60714
400 TRUE 336 29.909665 14.129613 39.58333
800 FALSE 336 16.126847 7.392411 48.21429
800 TRUE 336 14.028223 6.383631 51.19048
2000 FALSE 336 4.174122 2.544792 65.17857
2000 TRUE 336 3.733624 2.339732 65.77381

By test

All tests, except ss, seem to benefit from ug?

method numcond RMSE MAD ARR
peba2 84 35.125002 19.936310 36.90476
peba2_ug 84 32.782694 17.713095 41.66667
peba2_rls 84 5.575571 2.660119 75.00000
peba2_ug_rls 84 3.352189 1.930357 85.71429
peba4 84 31.701241 16.191071 47.61905
peba4_ug 84 29.735216 14.285714 53.57143
peba4_rls 84 3.411116 1.832143 82.14286
peba4_ug_rls 84 2.429910 1.788095 71.42857
peba6 84 29.495759 13.968452 54.76190
peba6_ug 84 27.637414 12.186310 64.28571
peba6_rls 84 2.608748 1.800595 72.61905
peba6_ug_rls 84 2.425577 2.027976 65.47619
pols2 84 22.475817 9.345833 58.33333
pols2_ug 84 19.418397 7.517262 65.47619
pols2_rls 84 2.733125 2.149405 61.90476
pols2_ug_rls 84 2.986637 2.453571 54.76190
sb 84 37.450730 22.932738 28.57143
sb_ug 84 35.059250 20.619643 32.14286
sb_rls 84 7.485981 4.204167 53.57143
sb_ug_rls 84 4.815396 2.614286 67.85714
ss 84 4.332054 4.000595 16.66667
ss_ug 84 4.205938 4.006548 14.28571
ss_rls 84 4.377860 4.213095 11.90476
ss_ug_rls 84 4.417383 4.269048 9.52381