From Philip Dale: “One highly attractive function of a CAT vocabulary measure would be for screening purposes. Evaluating its utility for that means looking at conventional measures of diagnostic validity such as sensitivity, specificity, PPV, and NPV, which are focused on the low end, rather than the broad range of scores. I’m wondering if that would be a lot of work to get those from your CAT simulations. You would have to compute the 10th percentile for each month of age on the full CDI and classify each child as delayed/not delayed. Then the same thing for the simulated CAT. That would give the 2x2 contingency table from which the diagnostic indices can be computed. I suppose we should do this separately for each month of age, as the validity might vary with development. For a start, we could just look at 18-30 months because the data are good, and also that’s the age range of most interest for screening.”
We’ve loaded the fits from the 2PL model, and will now calculate percentile scores per age (in months).
d_demo$ability = fscores_2pl$ability
d_demo$CATability = min25_max50_ML_age$parms$thetaCAT
d_demo <- d_demo %>% filter(age > 17, age < 31)
ten_pct <- d_demo %>% group_by(age) %>%
summarise(
ability = quantile(ability, c(0.10)),
CATability = quantile(CATability, c(0.10)),
production = quantile(production, c(0.10))
)## `summarise()` ungrouping output (override with `.groups` argument)# for each age, look up 10th pct score and classify children according to whether they are at/below 10th pct or not
d_demo$delayed_ability = NA
d_demo$delayed_CATability = NA
d_demo$delayed_production = NA
for(thisage in unique(ten_pct$age)) {
cur = subset(ten_pct, age==thisage)
idx = which(d_demo$age==thisage)
d_demo[idx,]$delayed_ability = ifelse(d_demo[idx,]$ability <= cur$ability, 1, 0)
d_demo[idx,]$delayed_CATability = ifelse(d_demo[idx,]$CATability <= cur$CATability, 1, 0)
d_demo[idx,]$delayed_production = ifelse(d_demo[idx,]$production <= cur$production, 1, 0)
}Compare delayed (at or below 10th percentile) production scores vs. CAT-estimated ability, by months of age.
print(with(d_demo, table(delayed_production, delayed_CATability, age)))## , , age = 18
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 466 4
## TRUE 7 49
##
## , , age = 19
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 291 4
## TRUE 4 29
##
## , , age = 20
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 240 4
## TRUE 6 24
##
## , , age = 21
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 181 1
## TRUE 1 20
##
## , , age = 22
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 177 2
## TRUE 3 18
##
## , , age = 23
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 233 2
## TRUE 2 25
##
## , , age = 24
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 517 7
## TRUE 7 52
##
## , , age = 25
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 293 4
## TRUE 4 30
##
## , , age = 26
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 159 3
## TRUE 3 16
##
## , , age = 27
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 171 4
## TRUE 4 16
##
## , , age = 28
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 702 13
## TRUE 15 67
##
## , , age = 29
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 174 3
## TRUE 4 17
##
## , , age = 30
##
## delayed_CATability
## delayed_production FALSE TRUE
## FALSE 262 3
## TRUE 3 27Compare delayed (at or below 10th percentile) ability vs. CAT-estimated ability, by months of age.
print(with(d_demo, table(delayed_ability, delayed_CATability, age)))## , , age = 18
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 468 5
## TRUE 5 48
##
## , , age = 19
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 292 3
## TRUE 3 30
##
## , , age = 20
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 242 4
## TRUE 4 24
##
## , , age = 21
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 180 2
## TRUE 2 19
##
## , , age = 22
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 178 2
## TRUE 2 18
##
## , , age = 23
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 233 2
## TRUE 2 25
##
## , , age = 24
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 517 7
## TRUE 7 52
##
## , , age = 25
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 293 4
## TRUE 4 30
##
## , , age = 26
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 159 3
## TRUE 3 16
##
## , , age = 27
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 172 3
## TRUE 3 17
##
## , , age = 28
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 704 13
## TRUE 13 67
##
## , , age = 29
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 175 3
## TRUE 3 17
##
## , , age = 30
##
## delayed_CATability
## delayed_ability FALSE TRUE
## FALSE 263 2
## TRUE 2 28