Simplified CDM and IRT:
Parameter Comparison
July 14, 2022
1. Simulation Study Design
The purpose of this simulation study is to explore the
relationships between parameters estimated from cognitive diagnostic
models (CDM) and ones from item response theory models (IRT). Among the
CDMs, this study focused on DINA and NIDA models with items that involve
one single skill per item (see the Q matrix below). We generated learner
response data using the following true values. The simulated data by
DINA and NIDA were analyzed by one of the IRT models, Rasch, to estimate
item difficulty and learner skill values.
| Learner | Skill1 | Skill2 | Skill3 | Skill4 | Skill5 | Total # of Skills |
|---|---|---|---|---|---|---|
| 1 | 1 | 0 | 1 | 0 | 0 | 2 |
| 2 | 0 | 0 | 1 | 1 | 0 | 2 |
| 3 | 1 | 0 | 1 | 1 | 1 | 4 |
| 4 | 1 | 1 | 0 | 1 | 0 | 3 |
| 5 | 1 | 0 | 0 | 0 | 0 | 1 |
| 6 | 1 | 0 | 1 | 1 | 1 | 4 |
| 7 | 1 | 1 | 1 | 0 | 0 | 3 |
| 8 | 1 | 1 | 1 | 0 | 0 | 3 |
| 9 | 1 | 1 | 1 | 1 | 0 | 4 |
| 10 | 1 | 1 | 1 | 1 | 1 | 5 |
| 11 | 1 | 1 | 1 | 0 | 0 | 3 |
| 12 | 0 | 0 | 1 | 1 | 0 | 2 |
| 13 | 1 | 0 | 1 | 0 | 1 | 3 |
| 14 | 1 | 0 | 1 | 1 | 0 | 3 |
| 15 | 0 | 0 | 1 | 1 | 0 | 2 |
| 16 | 1 | 0 | 0 | 1 | 1 | 3 |
| 17 | 1 | 1 | 1 | 1 | 0 | 4 |
| 18 | 1 | 0 | 1 | 1 | 0 | 3 |
| 19 | 1 | 0 | 1 | 1 | 1 | 4 |
| 20 | 0 | 0 | 1 | 0 | 0 | 1 |
| 21 | 1 | 0 | 1 | 0 | 0 | 2 |
| 22 | 1 | 0 | 1 | 1 | 1 | 4 |
| 23 | 1 | 0 | 1 | 1 | 1 | 4 |
| 24 | 1 | 1 | 1 | 1 | 0 | 4 |
| 25 | 0 | 1 | 1 | 0 | 0 | 2 |
| 26 | 1 | 0 | 1 | 1 | 0 | 3 |
| 27 | 1 | 0 | 1 | 1 | 0 | 3 |
| 28 | 1 | 0 | 1 | 1 | 1 | 4 |
| 29 | 1 | 0 | 1 | 1 | 0 | 3 |
| 30 | 0 | 0 | 1 | 1 | 1 | 3 |
| 31 | 0 | 0 | 1 | 0 | 0 | 1 |
| 32 | 1 | 1 | 1 | 1 | 0 | 4 |
| 33 | 0 | 0 | 1 | 0 | 1 | 2 |
| 34 | 1 | 0 | 1 | 1 | 0 | 3 |
| 35 | 1 | 1 | 1 | 1 | 0 | 4 |
| 36 | 0 | 1 | 1 | 0 | 1 | 3 |
| 37 | 1 | 1 | 1 | 0 | 0 | 3 |
| 38 | 0 | 0 | 1 | 1 | 1 | 3 |
| 39 | 1 | 1 | 1 | 0 | 0 | 3 |
| 40 | 1 | 1 | 1 | 1 | 0 | 4 |
| 41 | 1 | 0 | 1 | 1 | 0 | 3 |
| 42 | 0 | 1 | 0 | 1 | 1 | 3 |
| 43 | 1 | 0 | 1 | 1 | 0 | 3 |
| 44 | 1 | 0 | 0 | 1 | 0 | 2 |
| 45 | 1 | 1 | 1 | 1 | 0 | 4 |
| 46 | 1 | 1 | 1 | 1 | 0 | 4 |
| 47 | 1 | 1 | 1 | 1 | 0 | 4 |
| 48 | 1 | 1 | 1 | 1 | 0 | 4 |
| 49 | 1 | 0 | 0 | 0 | 0 | 1 |
| 50 | 1 | 1 | 1 | 1 | 1 | 5 |
| 51 | 1 | 1 | 1 | 0 | 0 | 3 |
| 52 | 1 | 1 | 1 | 0 | 0 | 3 |
| 53 | 1 | 0 | 0 | 1 | 0 | 2 |
| 54 | 0 | 0 | 1 | 1 | 0 | 2 |
| 55 | 1 | 0 | 1 | 1 | 0 | 3 |
| 56 | 0 | 0 | 0 | 0 | 1 | 1 |
| 57 | 0 | 1 | 1 | 1 | 1 | 4 |
| 58 | 1 | 1 | 1 | 1 | 1 | 5 |
| 59 | 0 | 0 | 1 | 0 | 1 | 2 |
| 60 | 0 | 0 | 1 | 1 | 0 | 2 |
| 61 | 1 | 0 | 1 | 1 | 0 | 3 |
| 62 | 1 | 1 | 1 | 1 | 0 | 4 |
| 63 | 1 | 0 | 1 | 1 | 0 | 3 |
| 64 | 0 | 1 | 1 | 0 | 0 | 2 |
| 65 | 0 | 0 | 1 | 0 | 1 | 2 |
| 66 | 1 | 0 | 1 | 0 | 0 | 2 |
| 67 | 1 | 0 | 1 | 0 | 0 | 2 |
| 68 | 0 | 1 | 1 | 1 | 0 | 3 |
| 69 | 1 | 0 | 1 | 1 | 0 | 3 |
| 70 | 0 | 1 | 1 | 1 | 1 | 4 |
| 71 | 0 | 0 | 1 | 0 | 0 | 1 |
| 72 | 1 | 1 | 1 | 1 | 0 | 4 |
| 73 | 1 | 1 | 1 | 1 | 0 | 4 |
| 74 | 0 | 1 | 1 | 1 | 0 | 3 |
| 75 | 0 | 1 | 1 | 0 | 1 | 3 |
| 76 | 1 | 0 | 1 | 1 | 1 | 4 |
| 77 | 0 | 0 | 1 | 0 | 1 | 2 |
| 78 | 0 | 1 | 1 | 1 | 1 | 4 |
| 79 | 1 | 0 | 1 | 1 | 1 | 4 |
| 80 | 0 | 0 | 1 | 0 | 0 | 1 |
| 81 | 1 | 1 | 1 | 1 | 1 | 5 |
| 82 | 1 | 1 | 1 | 0 | 1 | 4 |
| 83 | 1 | 0 | 1 | 0 | 1 | 3 |
| 84 | 0 | 1 | 1 | 1 | 0 | 3 |
| 85 | 0 | 1 | 0 | 1 | 1 | 3 |
| 86 | 1 | 0 | 1 | 1 | 0 | 3 |
| 87 | 0 | 0 | 1 | 0 | 0 | 1 |
| 88 | 1 | 0 | 1 | 1 | 1 | 4 |
| 89 | 1 | 0 | 1 | 1 | 0 | 3 |
| 90 | 1 | 1 | 1 | 1 | 0 | 4 |
| 91 | 1 | 1 | 1 | 1 | 0 | 4 |
| 92 | 1 | 1 | 1 | 1 | 0 | 4 |
| 93 | 1 | 0 | 1 | 1 | 0 | 3 |
| 94 | 1 | 0 | 1 | 1 | 1 | 4 |
| 95 | 1 | 0 | 1 | 0 | 0 | 2 |
| 96 | 1 | 0 | 1 | 1 | 0 | 3 |
| 97 | 0 | 1 | 1 | 0 | 0 | 2 |
| 98 | 1 | 0 | 1 | 0 | 1 | 3 |
| 99 | 1 | 1 | 1 | 0 | 1 | 4 |
| 100 | 1 | 1 | 1 | 0 | 1 | 4 |
| 101 | 1 | 1 | 0 | 1 | 0 | 3 |
| 102 | 1 | 0 | 1 | 1 | 0 | 3 |
| 103 | 1 | 0 | 1 | 1 | 1 | 4 |
| 104 | 1 | 0 | 1 | 1 | 1 | 4 |
| 105 | 1 | 1 | 1 | 1 | 1 | 5 |
| 106 | 1 | 0 | 1 | 0 | 1 | 3 |
| 107 | 1 | 1 | 1 | 0 | 0 | 3 |
| 108 | 1 | 0 | 1 | 0 | 0 | 2 |
| 109 | 1 | 1 | 1 | 0 | 1 | 4 |
| 110 | 1 | 0 | 1 | 0 | 0 | 2 |
| 111 | 1 | 0 | 1 | 1 | 0 | 3 |
| 112 | 1 | 1 | 0 | 1 | 0 | 3 |
| 113 | 0 | 1 | 1 | 0 | 1 | 3 |
| 114 | 1 | 0 | 1 | 1 | 0 | 3 |
| 115 | 1 | 0 | 1 | 1 | 0 | 3 |
| 116 | 1 | 0 | 1 | 1 | 1 | 4 |
| 117 | 1 | 1 | 0 | 1 | 0 | 3 |
| 118 | 0 | 0 | 1 | 1 | 1 | 3 |
| 119 | 1 | 0 | 1 | 1 | 0 | 3 |
| 120 | 1 | 0 | 1 | 1 | 0 | 3 |
| 121 | 1 | 0 | 1 | 1 | 1 | 4 |
| 122 | 0 | 0 | 1 | 1 | 0 | 2 |
| 123 | 0 | 1 | 1 | 0 | 0 | 2 |
| 124 | 1 | 1 | 1 | 0 | 1 | 4 |
| 125 | 1 | 0 | 0 | 1 | 0 | 2 |
| 126 | 1 | 0 | 1 | 1 | 0 | 3 |
| 127 | 1 | 1 | 1 | 1 | 1 | 5 |
| 128 | 1 | 1 | 0 | 1 | 1 | 4 |
| 129 | 1 | 0 | 1 | 0 | 0 | 2 |
| 130 | 1 | 0 | 1 | 1 | 0 | 3 |
| 131 | 1 | 0 | 1 | 0 | 0 | 2 |
| 132 | 1 | 1 | 1 | 1 | 0 | 4 |
| 133 | 0 | 1 | 1 | 0 | 1 | 3 |
| 134 | 1 | 1 | 1 | 1 | 0 | 4 |
| 135 | 1 | 1 | 0 | 1 | 0 | 3 |
| 136 | 1 | 0 | 1 | 0 | 0 | 2 |
| 137 | 1 | 0 | 1 | 1 | 1 | 4 |
| 138 | 1 | 1 | 1 | 1 | 1 | 5 |
| 139 | 1 | 0 | 1 | 1 | 0 | 3 |
| 140 | 1 | 0 | 1 | 1 | 1 | 4 |
| 141 | 1 | 1 | 1 | 1 | 1 | 5 |
| 142 | 1 | 1 | 1 | 0 | 1 | 4 |
| 143 | 1 | 0 | 1 | 1 | 1 | 4 |
| 144 | 0 | 0 | 1 | 0 | 0 | 1 |
| 145 | 1 | 1 | 0 | 0 | 0 | 2 |
| 146 | 1 | 0 | 1 | 1 | 0 | 3 |
| 147 | 1 | 0 | 1 | 1 | 1 | 4 |
| 148 | 1 | 0 | 1 | 0 | 0 | 2 |
| 149 | 1 | 0 | 1 | 1 | 0 | 3 |
| 150 | 0 | 1 | 1 | 1 | 0 | 3 |
| 151 | 1 | 0 | 1 | 1 | 0 | 3 |
| 152 | 1 | 1 | 1 | 0 | 1 | 4 |
| 153 | 0 | 0 | 1 | 0 | 1 | 2 |
| 154 | 0 | 1 | 0 | 1 | 0 | 2 |
| 155 | 1 | 0 | 1 | 0 | 0 | 2 |
| 156 | 1 | 1 | 1 | 0 | 0 | 3 |
| 157 | 1 | 0 | 1 | 1 | 0 | 3 |
| 158 | 0 | 0 | 1 | 0 | 1 | 2 |
| 159 | 1 | 1 | 1 | 0 | 0 | 3 |
| 160 | 1 | 0 | 1 | 1 | 1 | 4 |
| 161 | 1 | 0 | 1 | 0 | 0 | 2 |
| 162 | 0 | 1 | 1 | 0 | 0 | 2 |
| 163 | 1 | 0 | 1 | 1 | 0 | 3 |
| 164 | 1 | 1 | 1 | 0 | 0 | 3 |
| 165 | 1 | 0 | 1 | 0 | 1 | 3 |
| 166 | 1 | 1 | 1 | 0 | 0 | 3 |
| 167 | 0 | 1 | 1 | 1 | 1 | 4 |
| 168 | 1 | 1 | 1 | 1 | 1 | 5 |
| 169 | 1 | 0 | 0 | 0 | 0 | 1 |
| 170 | 1 | 0 | 1 | 1 | 0 | 3 |
| 171 | 1 | 1 | 1 | 1 | 1 | 5 |
| 172 | 1 | 1 | 0 | 0 | 0 | 2 |
| 173 | 1 | 1 | 1 | 0 | 1 | 4 |
| 174 | 1 | 1 | 0 | 1 | 0 | 3 |
| 175 | 0 | 1 | 1 | 1 | 0 | 3 |
| 176 | 1 | 1 | 1 | 1 | 0 | 4 |
| 177 | 1 | 1 | 1 | 1 | 0 | 4 |
| 178 | 1 | 0 | 1 | 1 | 0 | 3 |
| 179 | 0 | 0 | 1 | 0 | 0 | 1 |
| 180 | 1 | 1 | 1 | 0 | 0 | 3 |
| 181 | 1 | 0 | 1 | 1 | 0 | 3 |
| 182 | 1 | 1 | 1 | 1 | 0 | 4 |
| 183 | 1 | 0 | 1 | 1 | 0 | 3 |
| 184 | 1 | 0 | 1 | 0 | 0 | 2 |
| 185 | 1 | 0 | 1 | 1 | 0 | 3 |
| 186 | 1 | 1 | 1 | 1 | 0 | 4 |
| 187 | 1 | 1 | 0 | 0 | 0 | 2 |
| 188 | 0 | 1 | 1 | 0 | 0 | 2 |
| 189 | 1 | 0 | 0 | 1 | 0 | 2 |
| 190 | 1 | 0 | 1 | 1 | 1 | 4 |
| 191 | 1 | 0 | 1 | 0 | 0 | 2 |
| 192 | 1 | 0 | 1 | 0 | 0 | 2 |
| 193 | 0 | 0 | 1 | 1 | 1 | 3 |
| 194 | 0 | 1 | 1 | 0 | 0 | 2 |
| 195 | 1 | 0 | 0 | 0 | 0 | 1 |
| 196 | 1 | 0 | 1 | 0 | 0 | 2 |
| 197 | 0 | 1 | 1 | 1 | 0 | 3 |
| 198 | 0 | 0 | 1 | 1 | 0 | 2 |
| 199 | 1 | 1 | 1 | 1 | 0 | 4 |
| 200 | 0 | 1 | 1 | 0 | 0 | 2 |
| Item | Skill1 | Skill2 | Skill3 | Skill4 | Skill5 |
|---|---|---|---|---|---|
| 1 | 1 | 0 | 0 | 0 | 0 |
| 2 | 0 | 1 | 0 | 0 | 0 |
| 3 | 0 | 0 | 1 | 0 | 0 |
| 4 | 0 | 0 | 0 | 1 | 0 |
| 5 | 0 | 0 | 0 | 0 | 1 |
| 6 | 0 | 0 | 0 | 1 | 0 |
| 7 | 0 | 0 | 1 | 0 | 0 |
| 8 | 0 | 0 | 0 | 0 | 1 |
| 9 | 1 | 0 | 0 | 0 | 0 |
| 10 | 0 | 1 | 0 | 0 | 0 |
| 11 | 1 | 0 | 0 | 0 | 0 |
| 12 | 0 | 0 | 0 | 0 | 1 |
| 13 | 0 | 0 | 1 | 0 | 0 |
| 14 | 0 | 0 | 0 | 1 | 0 |
| 15 | 0 | 1 | 0 | 0 | 0 |
| 16 | 0 | 0 | 1 | 0 | 0 |
| 17 | 0 | 1 | 0 | 0 | 0 |
| 18 | 1 | 0 | 0 | 0 | 0 |
| 19 | 0 | 0 | 0 | 0 | 1 |
| 20 | 0 | 0 | 0 | 1 | 0 |
| 21 | 0 | 1 | 0 | 0 | 0 |
| 22 | 1 | 0 | 0 | 0 | 0 |
| 23 | 0 | 0 | 1 | 0 | 0 |
| 24 | 0 | 0 | 0 | 0 | 1 |
| 25 | 0 | 0 | 0 | 1 | 0 |
| 26 | 0 | 0 | 0 | 0 | 1 |
| 27 | 0 | 0 | 1 | 0 | 0 |
| 28 | 0 | 1 | 0 | 0 | 0 |
| 29 | 0 | 0 | 0 | 1 | 0 |
| 30 | 1 | 0 | 0 | 0 | 0 |
| Item | Hit (1 - Slip) | Guess |
|---|---|---|
| 1 | 0.6402064 | 0.0663373 |
| 2 | 0.7664291 | 0.1487147 |
| 3 | 0.8615828 | 0.0857883 |
| 4 | 0.6315809 | 0.2794831 |
| 5 | 0.6475758 | 0.2280644 |
| 6 | 0.7611137 | 0.2544553 |
| 7 | 0.7280119 | 0.1616461 |
| 8 | 0.6234309 | 0.0553341 |
| 9 | 0.8232062 | 0.1940295 |
| 10 | 0.6850177 | 0.2268624 |
| 11 | 0.9187429 | 0.1327583 |
| 12 | 0.8295276 | 0.2293021 |
| 13 | 0.8530992 | 0.0538474 |
| 14 | 0.6248707 | 0.0769292 |
| 15 | 0.8828779 | 0.0924592 |
| 16 | 0.6346501 | 0.2723798 |
| 17 | 0.8044818 | 0.1538909 |
| 18 | 0.8455635 | 0.2221435 |
| 19 | 0.8350879 | 0.2928427 |
| 20 | 0.6163942 | 0.1492838 |
| 21 | 0.8327854 | 0.0571125 |
| 22 | 0.7558969 | 0.2731436 |
| 23 | 0.6581821 | 0.1814247 |
| 24 | 0.7604780 | 0.1831064 |
| 25 | 0.7299039 | 0.1463419 |
| 26 | 0.9415226 | 0.2894229 |
| 27 | 0.6286622 | 0.1635177 |
| 28 | 0.6672417 | 0.0677156 |
| 29 | 0.6071370 | 0.2382317 |
| 30 | 0.7590172 | 0.1941605 |
| Skill | Hit (1 - Slip) | Guess |
|---|---|---|
| Skill 1 | 0.95 | 0.25 |
| Skill 2 | 0.70 | 0.10 |
| Skill 3 | 0.85 | 0.25 |
| Skill 4 | 0.80 | 0.10 |
| Skill 5 | 0.60 | 0.10 |
Based on the true values (learner skill mastery (\(\alpha\)), Q matrix, Hit (\(h\)), and Guess (\(g\)) parameters, 20 sets of response data
matrices were created using the DINA model described below. Each
response data matrix was for 200 learners and 30 items.
The function of a correct item response under DINA is described as
follows:
\[P_{ij} = h_{j}^{\eta_{ij}}g_{j}^{(1 -
\eta_{ij})}\] \[\eta_{ij} = \prod_{k =
1}^{K} \alpha_{ik}^{q_{jk}}\]
\(\alpha_{ik}\): learner skill
mastery per skill \(k\) by each learner
\(i\)
\(h_{j}\): 1 - slip per item \(j\)
\(g_{j}\): guess per item \(j\)
Another 20 sets of response data matrices were created using the NIDA
model for 200 learners and 30 items. The function of a correct item
response under NIDA is described as follows:
\[P_{ij} = \prod_{k = 1}^{K}
(h_{k}^{\alpha_{ik}}g_{k}^{(1 - \alpha_{ik})})^{q_{jk}}\]
\(\alpha_{ik}\): learner skill
mastery per skill \(k\) by each learner
\(i\)
\(h_{k}\): 1 - slip per skill \(k\)
\(g_{k}\): guess per skill \(k\)
2. Rasch Model Parameter Estimation
We applied the Rasch model to each response data matrix and
calculated item difficulty and skill estimates. The average item
difficulty and skill estimates from 20 data sets were calculated for the
DINA and NIDA models, respectively.
The function of a correct item response under Rasch is described as
follows:
\[P_{ij} = \frac{exp(\theta_{i} -
\beta_{j})}{1 + exp(\theta_{i} - \beta_{j})}\]
\(\theta_{i}\): learner skill \(i\)
\(\beta_{j}\): item difficulty \(j\)
3. Parameter Comparison
As we calculated skill estimates by fitting the Rasch model to
the DINA and NIDA simulated data, we compared these skill estimates with
the true values that were used for simulating the DINA and NIDA response
data. The comparison showed that the Rasch skill estimates from
both DINA and NIDA simulated data were perfectly correlated with the
true \(\alpha\) (skill mastery)
values.
3.1. DINA vs. Rasch
We compared the Rasch item difficulty estimates with the true
item hit \(h\) and guess \(g\) values that were used for simulating
the DINA response data. We found that the Rasch item difficulty
estimates from the DINA simulated data were negatively correlated with
the item hit \(h\) parameters (\(\rho\) = -0.41), but the relationship with
the item guess \(g\) parameters was
weak (\(\rho\) = -0.1).
3.2. NIDA vs. Rasch
We compared the Rasch skill estimates with the true skill \(\alpha\) values that were used for
simulating the NIDA response data. The NIDA skill mastery
patterns that included Skill 5 (with the lowest hit \(h\) (the highest slip) parameter) and Skill
2 (with the second lowest hit \(h\)
parameter) were related to lower Rasch skill estimates. The NIDA skill
mastery patterns that included Skill 1 (with the highest hit \(h\) parameter) were related to higher Rasch
skill estimates.
| NIDA Skill Mastery Pattern | IRT Skill Estimates |
|---|---|
| 0.0.0.0.1 | -1.2693237 |
| 0.0.1.0.0 | -1.1413860 |
| 1.0.0.0.0 | -1.1343732 |
| 0.0.1.0.1 | -0.7963927 |
| 0.1.0.1.0 | -0.6212416 |
| 0.1.1.0.0 | -0.6148298 |
| 0.0.1.1.0 | -0.5826221 |
| 1.1.0.0.0 | -0.5768498 |
| 1.0.1.0.0 | -0.5566502 |
| 1.0.0.1.0 | -0.4535163 |
| 0.1.1.0.1 | -0.2223022 |
| 0.1.0.1.1 | -0.1746597 |
| 1.0.1.0.1 | -0.1604756 |
| 1.1.1.0.0 | -0.0361790 |
| 0.0.1.1.1 | -0.0333555 |
| 0.1.1.1.0 | -0.0162578 |
| 1.1.0.1.0 | 0.0359713 |
| 1.0.0.1.1 | 0.0515063 |
| 1.0.1.1.0 | 0.0623549 |
| 1.1.1.0.1 | 0.3867322 |
| 0.1.1.1.1 | 0.4258707 |
| 1.0.1.1.1 | 0.5248952 |
| 1.1.1.1.0 | 0.6063866 |
| 1.1.0.1.1 | 0.6150383 |
| 1.1.1.1.1 | 1.0730383 |
4. Discussion
We could confirm that the IRT based skill estimates are aligned with the CDM based skill estimates when we analyze the same data using these two different models. We could also find relationships between the IRT parameters and the CDM hit \(h\) parameters (see Sections 3.1 and 3.2). However, it was not clear or easy to find any relationships between the Rasch parameters and the guess \(g\) parameters from both the DINA (item level) and NIDA (skill level) models. To explore further, we can fit either 2PL or 3PL IRT models to the simulated response data and compare the item discrimination parameters with the CDM guess parameters.