CoT Standard Setting
Session 3: DCS Construct
Design Computational Solutions:
Identify a problem and develop (creative) solutions by identifying input, output, and algorithms
DCS construct (public URL):
(please copy and paste these links)
https://cr4cr.bear-apps.com/constructs/public/5xXhDYnYmUve9c9DkpCgIMRsh6nCLI
Fall 2021 report (requires login):
https://cr4cr.bear-apps.com/scores_reports?scheduled_activities=141
Fall 2021 (AFE) report (requires login):
https://cr4cr.bear-apps.com/scores_reports?scheduled_activities=219
BASS page:
1 Construct levels
DCS6: Strategic/Step Beyond:
DCS5: Integrated Relational - Complex:
DCS4: Integrated Relational - Simple:
DCS3: Multi-Step Solution:
DCS2: One-Step Solution:
DCS1: Attempting/Partial:
2 Analyses
Unidimensional Rasch calibration with latent regression
3 Datasets (raw files)
There are n=876 respondents at Pretest. Raw data file is available here:
4 Calibration
4.1 Model specification
## [1] "item+item*step"4.2 Reliability
## EAP/PV RELIABILITY: 0.7074.3 Variance
## errors
## [1,] 1.032 0.054.4 Item parameters
The parameter difficulty parameters may be thought of as a kind of “average” item difficulty for a partial credit item. This may be useful, if one wishes to have one indicative difficulty parameter for a partial credit item as a whole. Otherwise, to describe the difficulty of a partial credit item, one needs to describe the difficulties of individual score categories within the item, such as using the Thurstonian thresholds (Wu et al., 2016).
## n_item item est error U.fit U.Low U.High U.T W.fit W.Low W.High
## 1 1 Deliv.00ab_.. 0.040 0.061 0.98 0.86 1.14 -0.2 0.98 0.87 1.13
## 2 2 Elev.00ab_O.. 0.511 0.070 0.89 0.86 1.14 -1.6 0.90 0.85 1.15
## 3 3 Elev.02abc_.. 0.525 0.061 0.87 0.86 1.14 -1.9 0.89 0.86 1.14
## 4 4 Mark.01abc_.. 0.289 0.041 1.07 0.90 1.10 1.4 1.05 0.90 1.10
## 5 5 Tick.02_OE 0.252 0.046 1.17 0.90 1.10 3.2 1.17 0.90 1.10
## 6 6 Trav.00ab_O.. -0.187 0.053 0.95 0.90 1.10 -0.9 0.95 0.91 1.09
## W.T
## 1 -0.3
## 2 -1.4
## 3 -1.6
## 4 1.0
## 5 3.3
## 6 -1.1## W.fit: Weighted fit (Infit)## [,1] [,2]
## [1,] "item1" "Del.00ab.OEOE"
## [2,] "item2" "Elev.00ab.OE2"
## [3,] "item3" "Elev.02abc.OE3"
## [4,] "item4" "Mark.01abc.OE"
## [5,] "item5" "Tick.02.OE"
## [6,] "item6" "Trav.00ab.OEOE"4.5 Step parameters
Delta is the point at which the probability of being in category k − 1 and category k is equals. This mathematical fact provides an interpretation for the delta (d) parameters (Wu et al., 2016)
## n_item item step est error U.fit U.Low U.High U.T W.fit W.Low
## 1 1 Deliv.00ab_.. 0 NA NA 0.66 0.86 1.14 -5.5 0.97 0.69
## 2 1 Deliv.00ab_.. 1 -2.538 0.153 0.93 0.86 1.14 -1.0 0.96 0.91
## 3 1 Deliv.00ab_.. 2 -0.139 0.109 0.95 0.86 1.14 -0.7 0.99 0.92
## 4 1 Deliv.00ab_.. 3 0.683 0.128 1.58 0.86 1.14 7.0 1.01 0.88
## 5 1 Deliv.00ab_.. 4 1.994 NA 1.03 0.86 1.14 0.4 1.04 0.72
## 6 2 Elev.00ab_O.. 0 NA NA 0.46 0.86 1.14 -9.5 0.78 0.79
## 7 2 Elev.00ab_O.. 1 -1.631 0.155 0.95 0.86 1.14 -0.7 0.99 0.83
## 8 2 Elev.00ab_O.. 2 -1.888 0.123 0.95 0.86 1.14 -0.7 0.97 0.92
## 9 2 Elev.00ab_O.. 3 1.754 0.178 0.84 0.86 1.14 -2.4 0.99 0.76
## 10 2 Elev.00ab_O.. 4 1.764 NA 1.23 0.86 1.14 3.0 1.13 0.49
## 11 3 Elev.02abc_.. 0 NA NA 0.67 0.86 1.14 -5.2 0.78 0.82
## 12 3 Elev.02abc_.. 1 -1.990 0.134 0.98 0.86 1.14 -0.3 0.99 0.93
## 13 3 Elev.02abc_.. 2 -0.023 0.117 0.88 0.86 1.14 -1.8 0.97 0.88
## 14 3 Elev.02abc_.. 3 0.414 0.151 0.85 0.86 1.14 -2.2 0.99 0.84
## 15 3 Elev.02abc_.. 4 1.598 NA 0.59 0.86 1.14 -6.7 0.94 0.65
## 16 4 Mark.01abc_.. 0 NA NA 1.71 0.90 1.10 11.6 0.90 0.86
## 17 4 Mark.01abc_.. 1 -2.137 0.092 1.04 0.90 1.10 0.8 1.00 0.96
## 18 4 Mark.01abc_.. 2 0.408 0.081 1.12 0.90 1.10 2.4 1.01 0.91
## 19 4 Mark.01abc_.. 3 0.670 0.109 1.28 0.90 1.10 5.0 1.01 0.87
## 20 4 Mark.01abc_.. 4 1.059 NA 1.49 0.90 1.10 8.4 1.05 0.83
## 21 5 Tick.02_OE 0 NA NA 1.38 0.90 1.10 6.7 0.95 0.82
## 22 5 Tick.02_OE 1 -2.011 0.119 0.99 0.90 1.10 -0.1 0.98 0.91
## 23 5 Tick.02_OE 2 -0.979 0.088 1.13 0.90 1.10 2.5 1.06 0.95
## 24 5 Tick.02_OE 3 0.125 0.085 1.29 0.90 1.10 5.2 1.14 0.94
## 25 5 Tick.02_OE 4 2.866 NA 4.09 0.90 1.10 35.2 1.10 0.67
## 26 6 Trav.00ab_O.. 0 NA NA 0.52 0.90 1.10 -11.6 0.78 0.81
## 27 6 Trav.00ab_O.. 1 -2.153 0.092 1.01 0.90 1.10 0.3 1.01 0.95
## 28 6 Trav.00ab_O.. 2 0.302 0.078 0.95 0.90 1.10 -0.9 0.98 0.96
## 29 6 Trav.00ab_O.. 3 1.851 NA 1.33 0.90 1.10 5.9 1.01 0.86
## W.High W.T
## 1 1.31 -0.1
## 2 1.09 -0.9
## 3 1.08 -0.3
## 4 1.12 0.2
## 5 1.28 0.3
## 6 1.21 -2.2
## 7 1.17 -0.1
## 8 1.08 -0.9
## 9 1.24 0.0
## 10 1.51 0.5
## 11 1.18 -2.6
## 12 1.07 -0.2
## 13 1.12 -0.4
## 14 1.16 -0.1
## 15 1.35 -0.3
## 16 1.14 -1.4
## 17 1.04 0.0
## 18 1.09 0.3
## 19 1.13 0.2
## 20 1.17 0.6
## 21 1.18 -0.6
## 22 1.09 -0.4
## 23 1.05 2.3
## 24 1.06 4.2
## 25 1.33 0.6
## 26 1.19 -2.4
## 27 1.05 0.3
## 28 1.04 -0.9
## 29 1.14 0.14.6 Thurstonian thresholds
For partial credit items, to achieve a score of 2, students would generally need to accomplish more tasks than for achieving a score of 1. To reflect this “cumulative achievement”, the Thurstonian thresholds are sometimes used as indicators of “score difficulties”. The Thurstonian threshold for a score category is defined as the ability at which the probability of achieving that score or higher reaches 0.50.
## [,1] [,2] [,3] [,4]
## Deliv.00ab_OEOE -2.581 -0.319 0.813 2.248
## Elev.00ab_OE2 -1.681 -0.853 1.818 2.759
## Elev.02abc_OE3 -1.590 0.225 1.098 2.367
## Mark.01abc_OE -1.922 0.286 1.001 1.803
## Tick.02_OE -2.021 -0.718 0.565 3.179
## Trav.00ab_OEOE -2.417 0.027 1.832 NA## DCS1 DCS2 DCS3 DCS4
## Deliv.00ab_OEOE -2.581 -0.319 0.813 2.248
## Elev.00ab_OE2 -1.681 -0.853 1.818 2.759
## Elev.02abc_OE3 -1.590 0.225 1.098 2.367
## Mark.01abc_OE -1.922 0.286 1.001 1.803
## Tick.02_OE -2.021 -0.718 0.565 3.179
## Trav.00ab_OEOE -2.417 0.027 1.832 NA## DCS1 DCS2 DCS3 DCS4
## Min. :-2.581 Min. :-0.8530 Min. :0.565 Min. :1.803
## 1st Qu.:-2.318 1st Qu.:-0.6182 1st Qu.:0.860 1st Qu.:2.248
## Median :-1.972 Median :-0.1460 Median :1.050 Median :2.367
## Mean :-2.035 Mean :-0.2253 Mean :1.188 Mean :2.471
## 3rd Qu.:-1.741 3rd Qu.: 0.1755 3rd Qu.:1.638 3rd Qu.:2.759
## Max. :-1.590 Max. : 0.2860 Max. :1.832 Max. :3.179
## NA's :15 Links to CQ files (please copy/paste these links)
5.2 CTT Item Analysis (ITN) file
5.4 Command file
https://www.dropbox.com/s/rs0zc0xu99vt29u/command_DCS.txt?dl=0
5.8 Parameter file
https://www.dropbox.com/s/5gbw5q2wfeoj3xd/DCS_PARMS.txt?dl=0
6 Item fit graph
7 Wright Map
8 Category Characteristic Curves (CCC)
9 Calibration summary from TAM
## ------------------------------------------------------------
## TAM 3.6-45 (2021-04-22 14:35:52)
## R version 4.0.5 (2021-03-31) x86_64, darwin17.0 | nodename=Permans-MacBook-Pro-4.local | login=root
##
## Date of Analysis: 2023-02-08 13:08:00
## Time difference of 0.09872603 secs
## Computation time: 0.09872603
##
## Multidimensional Item Response Model in TAM
##
## IRT Model: PCM2
## Call:
## TAM::tam.mml(resp = data2[, c(2:7)], irtmodel = "PCM2", verbose = FALSE)
##
## ------------------------------------------------------------
## Number of iterations = 35
## Numeric integration with 21 integration points
##
## Deviance = 8881.03
## Log likelihood = -4440.51
## Number of persons = 876
## Number of persons used = 849
## Number of items = 6
## Number of estimated parameters = 24
## Item threshold parameters = 23
## Item slope parameters = 0
## Regression parameters = 0
## Variance/covariance parameters = 1
##
## AIC = 8929 | penalty=48 | AIC=-2*LL + 2*p
## AIC3 = 8953 | penalty=72 | AIC3=-2*LL + 3*p
## BIC = 9043 | penalty=161.86 | BIC=-2*LL + log(n)*p
## aBIC = 8967 | penalty=85.53 | aBIC=-2*LL + log((n-2)/24)*p (adjusted BIC)
## CAIC = 9067 | penalty=185.86 | CAIC=-2*LL + [log(n)+1]*p (consistent AIC)
## AICc = 8930 | penalty=49.46 | AICc=-2*LL + 2*p + 2*p*(p+1)/(n-p-1) (bias corrected AIC)
## GHP = 1.27923 | GHP=( -LL + p ) / (#Persons * #Items) (Gilula-Haberman log penalty)
##
## ------------------------------------------------------------
## EAP Reliability
## [1] 0.685
## ------------------------------------------------------------
## Covariances and Variances
## [,1]
## [1,] 1.035
## ------------------------------------------------------------
## Correlations and Standard Deviations (in the diagonal)
## [,1]
## [1,] 1.017
## ------------------------------------------------------------
## Regression Coefficients
## [,1]
## [1,] 0
## ------------------------------------------------------------
## Item Parameters -A*Xsi
## item N M xsi.item AXsi_.Cat1 AXsi_.Cat2 AXsi_.Cat3 AXsi_.Cat4
## 1 Del.00ab.OEOE 399 1.922 0.043 -2.496 -2.592 -1.866 0.171
## 2 Elev.00ab.OE2 395 1.711 0.514 -1.117 -2.491 -0.222 2.057
## 3 Elev.02abc.OE3 380 1.545 0.528 -1.462 -0.957 -0.015 2.111
## 4 Mark.01abc.OE 779 1.646 0.292 -1.842 -1.138 -0.182 1.169
## 5 Tick.02.OE 764 2.009 0.255 -1.757 -2.481 -2.100 1.021
## 6 Trav.00ab.OEOE 773 1.558 -0.177 -2.321 -2.195 -0.530 NA
## B.Cat1.Dim1 B.Cat2.Dim1 B.Cat3.Dim1 B.Cat4.Dim1
## 1 1 2 3 4
## 2 1 2 3 4
## 3 1 2 3 4
## 4 1 2 3 4
## 5 1 2 3 4
## 6 1 2 3 0
##
## Item Parameters Xsi
## xsi se.xsi
## Del.00ab.OEOE 0.043 0.061
## Elev.00ab.OE2 0.514 0.070
## Elev.02abc.OE3 0.528 0.061
## Mark.01abc.OE 0.292 0.041
## Tick.02.OE 0.255 0.046
## Trav.00ab.OEOE -0.177 0.053
## Del.00ab.OEOE_step1 -2.539 0.153
## Del.00ab.OEOE_step2 -0.139 0.109
## Del.00ab.OEOE_step3 0.684 0.129
## Elev.00ab.OE2_step1 -1.632 0.155
## Elev.00ab.OE2_step2 -1.888 0.123
## Elev.00ab.OE2_step3 1.755 0.178
## Elev.02abc.OE3_step1 -1.990 0.134
## Elev.02abc.OE3_step2 -0.022 0.117
## Elev.02abc.OE3_step3 0.415 0.151
## Mark.01abc.OE_step1 -2.134 0.092
## Mark.01abc.OE_step2 0.411 0.081
## Mark.01abc.OE_step3 0.664 0.109
## Tick.02.OE_step1 -2.012 0.119
## Tick.02.OE_step2 -0.979 0.088
## Tick.02.OE_step3 0.125 0.085
## Trav.00ab.OEOE_step1 -2.145 0.091
## Trav.00ab.OEOE_step2 0.303 0.078
##
## Item Parameters in IRT parameterization
## item alpha beta tau.Cat1 tau.Cat2 tau.Cat3 tau.Cat4
## 1 Del.00ab.OEOE 1 0.043 -2.539 -0.139 0.684 1.994
## 2 Elev.00ab.OE2 1 0.514 -1.631 -1.888 1.755 1.765
## 3 Elev.02abc.OE3 1 0.528 -1.990 -0.022 0.415 1.598
## 4 Mark.01abc.OE 1 0.292 -2.134 0.411 0.664 1.059
## 5 Tick.02.OE 1 0.255 -2.012 -0.979 0.125 2.866
## 6 Trav.00ab.OEOE 1 -0.177 -2.144 0.303 1.841 NA