Examiner IRT abridged
Mark Sanderson-cimino
2023-04-20
Project summary
Create executive factor score using IRT (MIRT package) for combined TabCat and Examiner
Variables:
Tabcat: Running dots, dot coutning, flanker, set shifting, match UDS: Animal fluency, vegitable fluency, L words, F words
Issues:
We have more fluency data than Tabcat data (see desriptives). Should we restrit data in some way?
Data is pretty good after binning, but there is some skewness. Transforming reduces the skewness and also leads to more bins for several variables. Is it worth it to do this transformation?
Data: Below I provide some data looking at these issues. I ran most analsyes with: the full dataset, a reduced dataset (needs 1+ TabCat), and transformed data (also reduced to 1+ TabCat)
Outline
- Descriptives (raw data)
- EFA summary
- IRT models
Descriptives
Lookup table full data. Restricted data is similar.
## recode_score min_rundots max_rundots min_dotcount max_dotcount min_flanker
## 1 1 0 70 0 4 4000.0
## 2 2 95 140 5 6 4687.5
## 3 3 145 210 7 7 5116.0
## 4 4 215 285 8 9 5620.0
## 5 5 290 390 10 11 6135.0
## 6 6 NA NA 12 13 6680.0
## 7 7 NA NA 14 15 7230.0
## 8 8 NA NA 16 17 7737.7
## 9 9 NA NA 18 19 8290.0
## 10 10 NA NA 20 21 8810.0
## 11 11 NA NA 22 22 NA
## 12 12 NA NA 23 27 NA
## max_flanker min_setshift max_setshift min_match max_match min_animals
## 1 4380.0 0.0 2500.0 0 5 1
## 2 5049.8 2968.8 3780.0 6 10 4
## 3 5592.7 3860.0 4840.0 11 15 7
## 4 6090.0 4889.7 5950.0 16 20 10
## 5 6650.0 6015.2 7017.7 21 25 13
## 6 7180.0 7035.5 8090.0 26 31 16
## 7 7730.0 8113.1 10000.0 32 36 18
## 8 8260.0 NA NA 37 41 21
## 9 8790.0 NA NA 42 46 24
## 10 9820.0 NA NA 47 51 27
## 11 NA NA NA 52 57 30
## 12 NA NA NA 58 68 33
## max_animals min_veg max_veg min_fwords max_fwords min_lwords max_lwords
## 1 3 1 3 1 3 1 3
## 2 6 4 5 4 5 4 5
## 3 9 6 7 6 7 6 7
## 4 12 8 9 8 9 8 9
## 5 15 10 11 10 11 10 11
## 6 17 12 13 12 13 12 13
## 7 20 14 15 14 15 14 15
## 8 23 16 17 16 18 16 17
## 9 26 18 19 19 20 18 19
## 10 29 20 21 21 22 20 21
## 11 32 22 23 23 24 22 23
## 12 38 24 33 25 35 24 28Skewness: |minimim| shows transformation that brings skewness closest to zero
## raw square.root log+10 1/x+100 squared exp |minimum|
## rundots -0.94 -2.59 -3.47 3.42 0.42 NaN 0.42
## dotcount -0.23 -1.77 -1.11 0.52 0.71 12.03 0.23
## flanker -0.76 -0.90 -1.04 1.33 -0.33 NaN 0.33
## setshift -1.08 -2.53 -10.33 16.89 -0.28 NaN 0.28
## match -1.02 -1.84 -2.06 1.57 -0.07 28.90 0.07
## animals -0.07 -0.67 -0.76 0.38 0.92 23.28 0.07
## veg 0.09 -0.46 -0.46 0.13 1.24 28.25 0.09
## fwords 0.09 -0.53 -0.55 0.17 1.37 27.38 0.09
## lwords 0.09 -0.51 -0.50 0.15 1.16 13.70 0.09Lookup table skewed, restricted data.
## recode_score min_rundots max_rundots min_dotcount max_dotcount min_flanker
## 1 1 -152100 -143075 0 4 -80432400
## 2 2 -140000 -131075 5 6 -74459744
## 3 3 -129600 -121475 7 7 -69212181
## 4 4 -119700 -110075 8 9 -63369900
## 5 5 -108000 -101475 10 11 -57893136
## 6 6 -99200 -89600 12 13 -52342800
## 7 7 -87075 -79200 14 15 -46758096
## 8 8 -76475 -68000 16 17 -41227500
## 9 9 -65075 -59075 18 19 -35592400
## 10 10 -56000 -46475 20 21 -30140236
## 11 11 -43200 -36500 22 22 -24304748
## 12 12 -33075 0 23 27 -18816300
## max_flanker min_setshift max_setshift min_match max_match min_animals
## 1 -77248000 -100000000 -93750000 -4624 -4335 1
## 2 -70258944 -91186227 -85100400 -4300 -3999 4
## 3 -63714000 -84555100 -79468227 -3948 -3663 7
## 4 -58794175 -77070268 -69859900 -3600 -3399 10
## 5 -52787879 -69639900 -62681341 -3328 -3024 13
## 6 -46986189 -61952075 -54707100 -2943 -2775 16
## 7 -41642796 -53686531 -47036538 -2688 -2415 18
## 8 -36214800 -46941887 -39471600 -2320 -2124 21
## 9 -30172800 -39362631 -32090864 -2023 -1815 24
## 10 -25029900 -31275900 -24483900 -1708 -1488 27
## 11 -19168300 -24038317 0 -1375 -1143 30
## 12 0 NA NA -1024 0 33
## max_animals min_veg max_veg min_fwords max_fwords min_lwords max_lwords
## 1 3 1 3 1 3 1 3
## 2 6 4 5 4 5 4 5
## 3 9 6 7 6 7 6 7
## 4 12 8 9 8 9 8 9
## 5 15 10 11 10 11 10 11
## 6 17 12 13 12 13 12 13
## 7 20 14 15 14 15 14 15
## 8 23 16 17 16 18 16 17
## 9 26 18 19 19 20 18 19
## 10 29 20 21 21 22 20 21
## 11 32 22 23 23 24 22 23
## 12 38 24 33 25 35 24 28EFA summary
4 factor model was best.
Figure 1: EFA 4 factor model, unrestricted, nontransformed data.
## Figure 2: EFA 4 factor model, restricted, transformed data
IRT
A. IRT with full data
B. IRT with reduced data (all subs need 1+ Tabcat)
##
Iteration: 1, Log-Lik: -13678.309, Max-Change: 1.90663
Iteration: 2, Log-Lik: -12095.987, Max-Change: 2.53825
Iteration: 3, Log-Lik: -11431.930, Max-Change: 1.38813
Iteration: 4, Log-Lik: -11168.744, Max-Change: 0.72857
Iteration: 5, Log-Lik: -11069.665, Max-Change: 0.40417
Iteration: 6, Log-Lik: -11024.825, Max-Change: 0.23650
Iteration: 7, Log-Lik: -10996.867, Max-Change: 0.18836
Iteration: 8, Log-Lik: -10977.209, Max-Change: 0.13987
Iteration: 9, Log-Lik: -10962.616, Max-Change: 0.11158
Iteration: 10, Log-Lik: -10951.685, Max-Change: 0.09591
Iteration: 11, Log-Lik: -10943.294, Max-Change: 0.08940
Iteration: 12, Log-Lik: -10936.712, Max-Change: 0.07692
Iteration: 13, Log-Lik: -10931.493, Max-Change: 0.07247
Iteration: 14, Log-Lik: -10927.291, Max-Change: 0.06569
Iteration: 15, Log-Lik: -10923.867, Max-Change: 0.05893
Iteration: 16, Log-Lik: -10911.796, Max-Change: 0.03931
Iteration: 17, Log-Lik: -10910.880, Max-Change: 0.03390
Iteration: 18, Log-Lik: -10910.318, Max-Change: 0.02650
Iteration: 19, Log-Lik: -10909.292, Max-Change: 0.02910
Iteration: 20, Log-Lik: -10909.000, Max-Change: 0.01886
Iteration: 21, Log-Lik: -10908.785, Max-Change: 0.01627
Iteration: 22, Log-Lik: -10908.113, Max-Change: 0.01229
Iteration: 23, Log-Lik: -10908.041, Max-Change: 0.01021
Iteration: 24, Log-Lik: -10907.984, Max-Change: 0.00804
Iteration: 25, Log-Lik: -10907.788, Max-Change: 0.00710
Iteration: 26, Log-Lik: -10907.763, Max-Change: 0.00516
Iteration: 27, Log-Lik: -10907.752, Max-Change: 0.00403
Iteration: 28, Log-Lik: -10907.730, Max-Change: 0.00749
Iteration: 29, Log-Lik: -10907.721, Max-Change: 0.00128
Iteration: 30, Log-Lik: -10907.720, Max-Change: 0.00197
Iteration: 31, Log-Lik: -10907.717, Max-Change: 0.00126
Iteration: 32, Log-Lik: -10907.716, Max-Change: 0.00172
Iteration: 33, Log-Lik: -10907.714, Max-Change: 0.00107
Iteration: 34, Log-Lik: -10907.714, Max-Change: 0.00214
Iteration: 35, Log-Lik: -10907.711, Max-Change: 0.00164
Iteration: 36, Log-Lik: -10907.709, Max-Change: 0.00208
Iteration: 37, Log-Lik: -10907.704, Max-Change: 0.00253
Iteration: 38, Log-Lik: -10907.701, Max-Change: 0.00066
Iteration: 39, Log-Lik: -10907.701, Max-Change: 0.00021
Iteration: 40, Log-Lik: -10907.701, Max-Change: 0.00020
Iteration: 41, Log-Lik: -10907.701, Max-Change: 0.00014
Iteration: 42, Log-Lik: -10907.701, Max-Change: 0.00063
Iteration: 43, Log-Lik: -10907.701, Max-Change: 0.00039
Iteration: 44, Log-Lik: -10907.700, Max-Change: 0.00053
Iteration: 45, Log-Lik: -10907.700, Max-Change: 0.00055
Iteration: 46, Log-Lik: -10907.700, Max-Change: 0.00056
Iteration: 47, Log-Lik: -10907.700, Max-Change: 0.00016
Iteration: 48, Log-Lik: -10907.700, Max-Change: 0.00034
Iteration: 49, Log-Lik: -10907.700, Max-Change: 0.00017
Iteration: 50, Log-Lik: -10907.700, Max-Change: 0.00043
Iteration: 51, Log-Lik: -10907.700, Max-Change: 0.00014
Iteration: 52, Log-Lik: -10907.700, Max-Change: 0.00012
Iteration: 53, Log-Lik: -10907.700, Max-Change: 0.00028
Iteration: 54, Log-Lik: -10907.700, Max-Change: 0.00045
Iteration: 55, Log-Lik: -10907.700, Max-Change: 0.00017
Iteration: 56, Log-Lik: -10907.700, Max-Change: 0.00036
Iteration: 57, Log-Lik: -10907.700, Max-Change: 0.00014
Iteration: 58, Log-Lik: -10907.700, Max-Change: 0.00012
Iteration: 59, Log-Lik: -10907.700, Max-Change: 0.00025
Iteration: 60, Log-Lik: -10907.700, Max-Change: 0.00046
Iteration: 61, Log-Lik: -10907.700, Max-Change: 0.00020
Iteration: 62, Log-Lik: -10907.700, Max-Change: 0.00038
Iteration: 63, Log-Lik: -10907.700, Max-Change: 0.00015
Iteration: 64, Log-Lik: -10907.700, Max-Change: 0.00012
Iteration: 65, Log-Lik: -10907.700, Max-Change: 0.00027
Iteration: 66, Log-Lik: -10907.699, Max-Change: 0.00050
Iteration: 67, Log-Lik: -10907.699, Max-Change: 0.00019
Iteration: 68, Log-Lik: -10907.699, Max-Change: 0.00038
Iteration: 69, Log-Lik: -10907.699, Max-Change: 0.00016
Iteration: 70, Log-Lik: -10907.699, Max-Change: 0.00013
Iteration: 71, Log-Lik: -10907.699, Max-Change: 0.00026
Iteration: 72, Log-Lik: -10907.699, Max-Change: 0.00049
Iteration: 73, Log-Lik: -10907.699, Max-Change: 0.00020
Iteration: 74, Log-Lik: -10907.699, Max-Change: 0.00037
Iteration: 75, Log-Lik: -10907.699, Max-Change: 0.00014
Iteration: 76, Log-Lik: -10907.699, Max-Change: 0.00012
Iteration: 77, Log-Lik: -10907.699, Max-Change: 0.00026
Iteration: 78, Log-Lik: -10907.699, Max-Change: 0.00050
Iteration: 79, Log-Lik: -10907.699, Max-Change: 0.00017
Iteration: 80, Log-Lik: -10907.699, Max-Change: 0.00035
Iteration: 81, Log-Lik: -10907.699, Max-Change: 0.00015
Iteration: 82, Log-Lik: -10907.699, Max-Change: 0.00012
Iteration: 83, Log-Lik: -10907.699, Max-Change: 0.00025
Iteration: 84, Log-Lik: -10907.699, Max-Change: 0.00046
Iteration: 85, Log-Lik: -10907.699, Max-Change: 0.00018
Iteration: 86, Log-Lik: -10907.699, Max-Change: 0.00034
Iteration: 87, Log-Lik: -10907.699, Max-Change: 0.00013
Iteration: 88, Log-Lik: -10907.699, Max-Change: 0.00011
Iteration: 89, Log-Lik: -10907.699, Max-Change: 0.00024
Iteration: 90, Log-Lik: -10907.699, Max-Change: 0.00045
Iteration: 91, Log-Lik: -10907.699, Max-Change: 0.00016
Iteration: 92, Log-Lik: -10907.699, Max-Change: 0.00032
Iteration: 93, Log-Lik: -10907.699, Max-Change: 0.00013
Iteration: 94, Log-Lik: -10907.699, Max-Change: 0.00011
Iteration: 95, Log-Lik: -10907.699, Max-Change: 0.00023
Iteration: 96, Log-Lik: -10907.699, Max-Change: 0.00042
Iteration: 97, Log-Lik: -10907.699, Max-Change: 0.00016
Iteration: 98, Log-Lik: -10907.699, Max-Change: 0.00031
Iteration: 99, Log-Lik: -10907.699, Max-Change: 0.00012
Iteration: 100, Log-Lik: -10907.699, Max-Change: 0.00010## ExecutiveComposite_r F1 F2 h2
## rundots_r 0.586 0.000 0.000 0.343
## dotcount_r 0.548 0.000 0.000 0.301
## flanker_r 0.777 0.000 0.000 0.604
## setshift_r 0.762 0.000 0.000 0.580
## match_r 0.799 0.000 0.000 0.638
## animals_r 0.856 0.000 0.266 0.804
## veg_r 0.749 0.000 0.342 0.677
## fwords_r 0.666 0.617 0.000 0.825
## lwords_r 0.677 0.610 0.000 0.829
##
## SS loadings: 4.662 0.752 0.187
## Proportion Var: 0.518 0.084 0.021
##
## Factor correlations:
##
## ExecutiveComposite_r F1 F2
## ExecutiveComposite_r 1
## F1 0 1
## F2 0 0 1## Degrees of freedom (lower triangle) and p-values:
##
## rundots_r dotcount_r flanker_r setshift_r match_r animals_r veg_r
## rundots_r NA 0.116 0.265 0.002 0.021 0.004 0.001
## dotcount_r 44 NA 0.218 0.190 0.977 0.176 0.184
## flanker_r 36 99.000 NA 0.322 0.873 0.008 0.001
## setshift_r 24 66.000 54.000 NA 0.061 0.033 0.001
## match_r 44 121.000 99.000 66.000 NA 0.014 0.000
## animals_r 44 121.000 99.000 66.000 121.000 NA 0.848
## veg_r 44 121.000 99.000 66.000 121.000 121.000 NA
## fwords_r 44 121.000 99.000 66.000 121.000 121.000 121.000
## lwords_r 44 121.000 99.000 66.000 121.000 121.000 121.000
## fwords_r lwords_r
## rundots_r 0.039 0.134
## dotcount_r 0.034 0.010
## flanker_r 0.005 0.055
## setshift_r 0.008 0.024
## match_r 0.000 0.018
## animals_r 0.849 0.493
## veg_r 0.476 0.247
## fwords_r NA 0.000
## lwords_r 121.000 NA
##
## LD matrix (lower triangle) and standardized values.
##
## Upper triangle summary:
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## -0.424 -0.356 -0.236 -0.138 0.120 0.330
##
## rundots_r dotcount_r flanker_r setshift_r match_r animals_r veg_r
## rundots_r NA 0.274 0.275 -0.256 0.278 -0.378 -0.405
## dotcount_r 55.429 NA -0.278 -0.216 0.192 -0.315 -0.318
## flanker_r 40.884 109.659 NA 0.239 0.234 -0.415 -0.424
## setshift_r 48.585 75.860 58.234 NA 0.243 -0.355 -0.398
## match_r 65.019 91.957 83.183 84.632 NA -0.135 -0.147
## animals_r 72.410 135.378 136.156 88.661 157.456 NA -0.110
## veg_r 81.290 134.851 148.723 110.199 180.943 105.107 NA
## fwords_r 61.823 150.912 139.597 97.045 224.365 105.051 121.262
## lwords_r 54.478 160.180 122.418 90.509 155.954 120.596 131.236
## fwords_r lwords_r
## rundots_r -0.346 -0.327
## dotcount_r 0.330 -0.343
## flanker_r -0.413 -0.385
## setshift_r -0.369 -0.359
## match_r -0.165 -0.138
## animals_r 0.111 0.119
## veg_r -0.120 0.125
## fwords_r NA -0.281
## lwords_r 672.918 NAC. IRT with reduced data (all subs need 1+ Tabcat and skew correction
D. IRT model comparisons.
1. EF factor loadings by model
##
##
## | | Full | Restrict | Skew |
## |:----------:|:------:|:--------:|:------:|
## | **n** | 1324 | 1323 | 1323 |
## | **LogLik** | -10914 | -10908 | -11670 |
## | **AIC** | 22020 | 22007 | 23559 |
## | **BIC** | 22518 | 22505 | 24124 |
##
## Table: Model comparison##
##
## | | Full | Restrict | Skew |
## |:--------------:|:------:|:--------:|:------:|
## | **rundots_r** | 0.586 | 0.5858 | 0.5275 |
## | **dotcount_r** | 0.5486 | 0.5485 | 0.5306 |
## | **flanker_r** | 0.7773 | 0.7775 | 0.7613 |
## | **setshift_r** | 0.7618 | 0.7619 | 0.766 |
## | **match_r** | 0.7986 | 0.7985 | 0.8037 |
## | **animals_r** | 0.8567 | 0.8565 | 0.862 |
## | **veg_r** | 0.7489 | 0.7487 | 0.749 |
## | **fwords_r** | 0.667 | 0.6665 | 0.6662 |
## | **lwords_r** | 0.6769 | 0.6766 | 0.6751 |
##
## Table: EF factor loadings by model##
##
## | | Full | Restrict | Skew |
## |:--------------:|:-----:|:--------:|:-----:|
## | **rundots_r** | 0.343 | 0.343 | 0.278 |
## | **dotcount_r** | 0.301 | 0.301 | 0.282 |
## | **flanker_r** | 0.604 | 0.604 | 0.58 |
## | **setshift_r** | 0.58 | 0.58 | 0.587 |
## | **match_r** | 0.638 | 0.638 | 0.646 |
## | **animals_r** | 0.804 | 0.804 | 0.81 |
## | **veg_r** | 0.677 | 0.677 | 0.676 |
## | **fwords_r** | 0.825 | 0.825 | 0.826 |
## | **lwords_r** | 0.829 | 0.829 | 0.829 |
##
## Table: H2: variance accounted for by variable