CoT Standard Setting
Session 4: revisiting the DCS Construct (using Spring 2022 only)
Design Computational Solutions:
Identify a problem and develop (creative) solutions by identifying input, output, and algorithms
DCS construct (public URL):
(please copy and paste these links)
https://cr4cr.bear-apps.com/constructs/public/5xXhDYnYmUve9c9DkpCgIMRsh6nCLI
Spring 2022 report (requires login):
https://cr4cr.bear-apps.com/scores_reports?scheduled_activities=267
BASS page:
1 Construct levels
DCS6: Strategic/Step Beyond:
DCS5: Integrated Relational - Complex:
DCS4: Integrated Relational - Simple:
DCS3: Multi-Step Solution:
DCS2: One-Step Solution:
DCS1: Attempting/Partial:
2 Analyses
Unidimensional Rasch calibration of the DCS items (7 items)
3 Datasets (raw files)
There are n=296 respondents at Pretest. Raw data file is available here:
4 Calibration
4.1 Model specification
## [1] "item+item*step"4.2 Reliability
## EAP/PV RELIABILITY: 0.7374.3 Variance
## errors
## [1,] 1.48 0.1224.4 Item parameters
The parameter difficulty parameters may be thought of as a kind of “average” item difficulty for a partial credit item. This may be useful, if one wishes to have one indicative difficulty parameter for a partial credit item as a whole. Otherwise, to describe the difficulty of a partial credit item, one needs to describe the difficulties of individual score categories within the item, such as using the Thurstonian thresholds (Wu et al., 2016).
## n_item item est error U.fit U.Low U.High U.T W.fit W.Low W.High
## 1 1 Deliv.00abc.. 0.859 0.158 0.97 0.70 1.30 -0.1 0.98 0.71 1.29
## 2 2 Elev.00abc_.. 0.516 0.152 1.18 0.67 1.33 1.1 1.16 0.67 1.33
## 3 3 Elev.02abc_.. 0.319 0.094 0.86 0.77 1.23 -1.2 0.86 0.78 1.22
## 4 4 Mark.00abc_.. 0.225 0.102 0.97 0.78 1.22 -0.2 0.98 0.78 1.22
## 5 5 Mark.01abc_.. 0.495 0.125 1.01 0.69 1.31 0.1 1.04 0.70 1.30
## 6 6 Mark.01abc_.. -0.497 0.161 1.01 0.67 1.33 0.1 1.07 0.65 1.35
## 7 7 Trav.00abc_.. 0.424 0.154 1.09 0.70 1.30 0.6 1.08 0.70 1.30
## W.T
## 1 -0.1
## 2 0.9
## 3 -1.3
## 4 -0.1
## 5 0.3
## 6 0.4
## 7 0.6## W.fit: Weighted fit (Infit)## [,1] [,2]
## [1,] "item1" "Deliv.00abc_OE"
## [2,] "item2" "Elev.00abc_OE"
## [3,] "item3" "Elev.02abc_OE"
## [4,] "item4" "Mark.00abc_OE"
## [5,] "item5" "Mark.01abc_OE"
## [6,] "item6" "Mark.01abc_MC"
## [7,] "item7" "Trav.00abc_OE"4.5 Step parameters
Delta is the point at which the probability of being in category k − 1 and category k is equals. This mathematical fact provides an interpretation for the delta (d) parameters (Wu et al., 2016)
## n_item item step est error U.fit U.Low U.High U.T W.fit W.Low
## 1 1 Deliv.00abc.. 0 NA NA 0.91 0.70 1.30 -0.6 0.87 0.50
## 2 1 Deliv.00abc.. 1 -2.952 0.376 0.98 0.70 1.30 -0.1 0.99 0.83
## 3 1 Deliv.00abc.. 2 -1.121 0.258 0.96 0.70 1.30 -0.2 0.99 0.88
## 4 1 Deliv.00abc.. 3 0.933 0.345 1.04 0.70 1.30 0.3 1.04 0.61
## 5 1 Deliv.00abc.. 4 3.140 NA 0.42 0.70 1.30 -4.8 1.13 0.00
## 6 2 Elev.00abc_.. 0 NA NA 6.33 0.67 1.33 15.4 1.04 0.57
## 7 2 Elev.00abc_.. 1 -1.359 0.336 1.21 0.67 1.33 1.2 1.00 0.59
## 8 2 Elev.00abc_.. 2 -1.683 0.291 1.02 0.67 1.33 0.2 1.07 0.86
## 9 2 Elev.00abc_.. 3 0.972 0.330 0.79 0.67 1.33 -1.3 1.05 0.66
## 10 2 Elev.00abc_.. 4 2.071 NA 0.86 0.67 1.33 -0.8 1.17 0.05
## 11 3 Elev.02abc_.. 0 NA NA 0.47 0.77 1.23 -5.7 0.75 0.72
## 12 3 Elev.02abc_.. 1 -1.889 0.197 0.87 0.77 1.23 -1.1 0.93 0.87
## 13 3 Elev.02abc_.. 2 0.080 0.187 1.04 0.77 1.23 0.4 1.01 0.82
## 14 3 Elev.02abc_.. 3 0.971 0.259 0.88 0.77 1.23 -1.0 1.02 0.69
## 15 3 Elev.02abc_.. 4 0.838 NA 0.51 0.77 1.23 -5.1 0.93 0.70
## 16 4 Mark.00abc_.. 0 NA NA 0.35 0.78 1.22 -7.8 0.81 0.66
## 17 4 Mark.00abc_.. 1 -2.210 0.221 1.03 0.78 1.22 0.3 0.96 0.85
## 18 4 Mark.00abc_.. 2 -0.695 0.180 0.93 0.78 1.22 -0.6 0.98 0.90
## 19 4 Mark.00abc_.. 3 1.519 0.252 1.33 0.78 1.22 2.6 1.03 0.70
## 20 4 Mark.00abc_.. 4 1.386 NA 1.78 0.78 1.22 5.6 1.11 0.61
## 21 5 Mark.01abc_.. 0 NA NA 0.67 0.69 1.31 -2.3 0.83 0.67
## 22 5 Mark.01abc_.. 1 -1.491 0.258 1.39 0.69 1.31 2.3 1.00 0.81
## 23 5 Mark.01abc_.. 2 0.047 0.259 0.87 0.69 1.31 -0.8 0.98 0.70
## 24 5 Mark.01abc_.. 3 0.536 0.334 0.86 0.69 1.31 -0.9 0.98 0.65
## 25 5 Mark.01abc_.. 4 0.908 NA 0.78 0.69 1.31 -1.5 1.17 0.59
## 26 6 Mark.01abc_.. 0 NA NA 0.17 0.67 1.33 -7.9 0.98 0.00
## 27 6 Mark.01abc_.. 1 -4.326 0.418 0.96 0.67 1.33 -0.2 0.90 0.75
## 28 6 Mark.01abc_.. 2 0.226 0.257 0.93 0.67 1.33 -0.3 0.97 0.86
## 29 6 Mark.01abc_.. 3 2.652 0.440 0.92 0.67 1.33 -0.4 1.04 0.36
## 30 6 Mark.01abc_.. 4 1.448 NA 0.78 0.67 1.33 -1.4 1.15 0.47
## 31 7 Trav.00abc_.. 0 NA NA 0.70 0.70 1.30 -2.1 0.95 0.45
## 32 7 Trav.00abc_.. 1 -2.283 0.378 1.19 0.70 1.30 1.2 1.05 0.69
## 33 7 Trav.00abc_.. 2 -1.686 0.273 1.02 0.70 1.30 0.2 1.03 0.89
## 34 7 Trav.00abc_.. 3 0.900 0.293 0.88 0.70 1.30 -0.7 0.99 0.71
## 35 7 Trav.00abc_.. 4 3.069 NA 2.07 0.70 1.30 5.4 1.27 0.00
## W.High W.T
## 1 1.50 -0.5
## 2 1.17 -0.1
## 3 1.12 -0.2
## 4 1.39 0.2
## 5 2.92 0.4
## 6 1.43 0.2
## 7 1.41 0.1
## 8 1.14 1.0
## 9 1.34 0.3
## 10 1.95 0.5
## 11 1.28 -1.9
## 12 1.13 -1.1
## 13 1.18 0.1
## 14 1.31 0.2
## 15 1.30 -0.4
## 16 1.34 -1.2
## 17 1.15 -0.5
## 18 1.10 -0.4
## 19 1.30 0.3
## 20 1.39 0.5
## 21 1.33 -1.0
## 22 1.19 0.0
## 23 1.30 -0.1
## 24 1.35 0.0
## 25 1.41 0.8
## 26 2.79 0.3
## 27 1.25 -0.8
## 28 1.14 -0.4
## 29 1.64 0.2
## 30 1.53 0.6
## 31 1.55 -0.2
## 32 1.31 0.4
## 33 1.11 0.5
## 34 1.29 0.0
## 35 2.31 0.64.6 Thurstonian thresholds
For partial credit items, to achieve a score of 2, students would generally need to accomplish more tasks than for achieving a score of 1. To reflect this “cumulative achievement”, the Thurstonian thresholds are sometimes used as indicators of “score difficulties”. The Thurstonian threshold for a score category is defined as the ability at which the probability of achieving that score or higher reaches 0.50.
## [,1] [,2] [,3] [,4]
## Deliv.00abc_OE -2.226 -0.241 1.808 4.096
## Elev.00abc_OE -1.435 -0.670 1.327 2.824
## Elev.02abc_OE -1.691 0.184 1.073 1.747
## Mark.00abc_OE -2.158 -0.407 1.335 2.150
## Mark.01abc_OE -1.179 0.293 1.032 1.851
## Mark.01abc_MC -4.833 -0.357 1.391 1.865
## Trav.00abc_OE -2.198 -1.002 1.296 3.592## DCS1 DCS2 DCS3 DCS4
## Deliv.00abc_OE -2.226 -0.241 1.808 4.096
## Elev.00abc_OE -1.435 -0.670 1.327 2.824
## Elev.02abc_OE -1.691 0.184 1.073 1.747
## Mark.00abc_OE -2.158 -0.407 1.335 2.150
## Mark.01abc_OE -1.179 0.293 1.032 1.851
## Mark.01abc_MC -4.833 -0.357 1.391 1.865
## Trav.00abc_OE -2.198 -1.002 1.296 3.592## DCS1 DCS2 DCS3 DCS4
## Min. :-4.833 Min. :-1.0020 Min. :1.032 Min. :1.747
## 1st Qu.:-2.212 1st Qu.:-0.5385 1st Qu.:1.185 1st Qu.:1.858
## Median :-2.158 Median :-0.3570 Median :1.327 Median :2.150
## Mean :-2.246 Mean :-0.3143 Mean :1.323 Mean :2.589
## 3rd Qu.:-1.563 3rd Qu.:-0.0285 3rd Qu.:1.363 3rd Qu.:3.208
## Max. :-1.179 Max. : 0.2930 Max. :1.808 Max. :4.0965 Links to CQ files (please copy/paste these links)
5.2 CTT Item Analysis (ITN) file
5.4 Command file
https://www.dropbox.com/s/6qfd1l6yqir3z7o/command_DCS.txt?dl=0
5.8 Parameter file
https://www.dropbox.com/s/q5z2axsz316k9eg/DCS_PARMS.txt?dl=0
6 Item fit graph
7 Wright Map
8 Category Characteristic Curves (CCC)
9 Calibration summary from TAM
## ------------------------------------------------------------
## TAM 4.1-4 (2022-08-28 16:03:54)
## R version 4.2.2 (2022-10-31) x86_64, darwin17.0 | nodename=Permans-MacBook-Pro-4.local | login=root
##
## Date of Analysis: 2023-02-16 22:13:24
## Time difference of 0.07152104 secs
## Computation time: 0.07152104
##
## Multidimensional Item Response Model in TAM
##
## IRT Model: PCM2
## Call:
## TAM::tam.mml(resp = data2[, c(2:8)], irtmodel = "PCM2", verbose = FALSE)
##
## ------------------------------------------------------------
## Number of iterations = 69
## Numeric integration with 21 integration points
##
## Deviance = 1835.57
## Log likelihood = -917.79
## Number of persons = 295
## Number of persons used = 248
## Number of items = 7
## Number of estimated parameters = 29
## Item threshold parameters = 28
## Item slope parameters = 0
## Regression parameters = 0
## Variance/covariance parameters = 1
##
## AIC = 1894 | penalty=58 | AIC=-2*LL + 2*p
## AIC3 = 1923 | penalty=87 | AIC3=-2*LL + 3*p
## BIC = 1995 | penalty=159.89 | BIC=-2*LL + log(n)*p
## aBIC = 1903 | penalty=67.49 | aBIC=-2*LL + log((n-2)/24)*p (adjusted BIC)
## CAIC = 2024 | penalty=188.89 | CAIC=-2*LL + [log(n)+1]*p (consistent AIC)
## AICc = 1902 | penalty=65.98 | AICc=-2*LL + 2*p + 2*p*(p+1)/(n-p-1) (bias corrected AIC)
## GHP = 1.36228 | GHP=( -LL + p ) / (#Persons * #Items) (Gilula-Haberman log penalty)
##
## ------------------------------------------------------------
## EAP Reliability
## [1] 0.578
## ------------------------------------------------------------
## Covariances and Variances
## [,1]
## [1,] 1.482
## ------------------------------------------------------------
## Correlations and Standard Deviations (in the diagonal)
## [,1]
## [1,] 1.217
## ------------------------------------------------------------
## Regression Coefficients
## [,1]
## [1,] 0
## ------------------------------------------------------------
## Item Parameters -A*Xsi
## item N M xsi.item AXsi_.Cat1 AXsi_.Cat2 AXsi_.Cat3 AXsi_.Cat4
## 1 Deliv.00abc_OE 84 1.619 0.860 -2.093 -2.354 -0.561 3.441
## 2 Elev.00abc_OE 72 1.750 0.516 -0.844 -2.011 -0.523 2.063
## 3 Elev.02abc_OE 148 1.709 0.319 -1.570 -1.171 0.119 1.276
## 4 Mark.00abc_OE 155 1.787 0.224 -1.986 -2.457 -0.714 0.896
## 5 Mark.01abc_OE 80 1.650 0.494 -0.997 -0.456 0.574 1.977
## 6 Mark.01abc_MC 71 1.859 -0.499 -4.827 -5.099 -2.944 -1.994
## 7 Trav.00abc_OE 85 1.882 0.424 -1.860 -3.121 -1.797 1.698
## B.Cat1.Dim1 B.Cat2.Dim1 B.Cat3.Dim1 B.Cat4.Dim1
## 1 1 2 3 4
## 2 1 2 3 4
## 3 1 2 3 4
## 4 1 2 3 4
## 5 1 2 3 4
## 6 1 2 3 4
## 7 1 2 3 4
##
## Item Parameters Xsi
## xsi se.xsi
## Deliv.00abc_OE 0.860 0.158
## Elev.00abc_OE 0.516 0.152
## Elev.02abc_OE 0.319 0.094
## Mark.00abc_OE 0.224 0.102
## Mark.01abc_OE 0.494 0.125
## Mark.01abc_MC -0.499 0.161
## Trav.00abc_OE 0.424 0.154
## Deliv.00abc_OE_step1 -2.953 0.376
## Deliv.00abc_OE_step2 -1.122 0.258
## Deliv.00abc_OE_step3 0.933 0.345
## Elev.00abc_OE_step1 -1.360 0.336
## Elev.00abc_OE_step2 -1.683 0.291
## Elev.00abc_OE_step3 0.972 0.330
## Elev.02abc_OE_step1 -1.889 0.197
## Elev.02abc_OE_step2 0.080 0.187
## Elev.02abc_OE_step3 0.971 0.259
## Mark.00abc_OE_step1 -2.210 0.221
## Mark.00abc_OE_step2 -0.695 0.180
## Mark.00abc_OE_step3 1.519 0.252
## Mark.01abc_OE_step1 -1.492 0.258
## Mark.01abc_OE_step2 0.047 0.259
## Mark.01abc_OE_step3 0.536 0.334
## Mark.01abc_MC_step1 -4.329 0.418
## Mark.01abc_MC_step2 0.227 0.257
## Mark.01abc_MC_step3 2.653 0.440
## Trav.00abc_OE_step1 -2.284 0.378
## Trav.00abc_OE_step2 -1.686 0.273
## Trav.00abc_OE_step3 0.900 0.293
##
## Item Parameters in IRT parameterization
## item alpha beta tau.Cat1 tau.Cat2 tau.Cat3 tau.Cat4
## 1 Deliv.00abc_OE 1 0.860 -2.953 -1.122 0.933 3.142
## 2 Elev.00abc_OE 1 0.516 -1.360 -1.683 0.972 2.071
## 3 Elev.02abc_OE 1 0.319 -1.889 0.080 0.971 0.838
## 4 Mark.00abc_OE 1 0.224 -2.210 -0.695 1.519 1.386
## 5 Mark.01abc_OE 1 0.494 -1.492 0.047 0.536 0.909
## 6 Mark.01abc_MC 1 -0.499 -4.329 0.227 2.653 1.449
## 7 Trav.00abc_OE 1 0.424 -2.284 -1.686 0.900 3.070