CoT Standard Setting

Session 1: AEI Construct

Analyze, Evaluate, and Iterate Computational Solutions:

Critically examine a computational solution and iteratively improve it by debugging, considering multiple criteria (e.g., accuracy, efficiency)

AEI construct (public URL):

https://cr4cr.bear-apps.com/constructs/public/u1JW9pXGW4QlVnjYGwnaF1PzwKMXNr

Fall 2021 report (requires login):

https://cr4cr.bear-apps.com/table_of_reports?scheduled_activities=141

Fall 2021 (AFE) report (requires login):

https://cr4cr.bear-apps.com/table_of_reports?scheduled_activities=219

Spring 2022 report (requires login):

https://cr4cr.bear-apps.com/table_of_reports?scheduled_activities=267

BASS page:

https://cr4cr.bear-apps.com/

1 Construct levels

AEI6: Step Beyond/Strategic:

AEI5: Integrated Relational - Complex:

AEI4: Integrated Relational - Simple:

AEI3: Multi-step Solutions:

AEI2: One-step Solutions:

AEI1: Attempting/Partial:

AEI0: Not Evident:

2 Analyses

Concurrent calibration with latent regression

1. we stacked three datasets (by matching on common items), which are: (dataset 1) pre, (dataset 2) mid (AFE) and (dataset 3) post.
2. we added indicator variables for mid and post as the predictors in the latent regression to allow separate means for pre (reference, mean=0), mid, and post timepoints.

3 Datasets (raw files)

There are n=876 respondents at pre, n=632 respondents at mid (AFE) and n=296 respondents at post. Raw data files are available here:

3.0.1 Pre

https://www.dropbox.com/s/ghdsyznb4u85dvn/raw_CoT%20%28computational%20thinking%29%20Fall%202021%2009_20_21%20-%2010_15_21%20analyze_evaluate_and_iterate_computational_solutions.csv?dl=0

3.0.2 Pre (AFE)

https://www.dropbox.com/s/j5k9rwhegogk33h/raw_CoT%20%28computational%20thinking%29%20AFE%20Spring%202022%2001_25_22%20-%2002_28_22%20analyze_evaluate_and_iterate_computational_solutions.csv?dl=0

3.0.3 Post

https://www.dropbox.com/s/rq39zph4cyvkn9y/raw_CoT%20%28computational%20thinking%29%20Spr%2022%20pretested%2004_06_22%20-%2005_27_22%20analyze_evaluate_and_iterate_computational_solutions.csv?dl=0

4 Items

Item 1: Delivery.03ab_MCMC

Scoring

Item 2: Delivery.03d_NU

Scoring

Item 3: Elevator.04c_OE

Scoring

Item 4: Park.01b_MC

Scoring

Item 5: Shipping.03abc_MCOE2

Scoring

Item 6: SignUp.01ab_MCRO

Scoring

Item 7: Sorting.03b_MC

Scoring

Item 8: Sorting.03c_MC

Scoring

Item 9: TicTac.01ab_MCOEv2

Scoring

Item 10: Travel.01_NU

Scoring

Item 11: SignUp.01a_MC

Scoring

Item 12: Delivery.01b_MC

Scoring

5 Calibration

5.1 Model specification

## [1] "item+item*step"

5.2 Latent regression coefficient (gain from pre to post)

##                  mid  post
## Main dimension 0.177 0.735
## S. errors      0.028 0.037

5.3 Reliability

## EAP/PV RELIABILITY: 0.504

5.4 Variance

##            errors
## [1,] 0.262  0.009

5.5 Item parameters

The parameter difficulty parameters may be thought of as a kind of “average” item difficulty for a partial credit item. This may be useful, if one wishes to have one indicative difficulty parameter for a partial credit item as a whole. Otherwise, to describe the difficulty of a partial credit item, one needs to describe the difficulties of individual score categories within the item, such as using the Thurstonian thresholds (Wu et al., 2016).

##    n_item          item    est error U.fit U.Low U.High  U.T W.fit W.Low W.High
## 1       1 Delivery.03..  0.200 0.047  0.92  0.91   1.09 -1.8  0.92  0.93   1.07
## 2       2 Delivery.03..  1.320 0.061  0.93  0.91   1.09 -1.6  0.93  0.93   1.07
## 3       3 Elevator.04..  1.084 0.060  0.88  0.85   1.15 -1.7  0.88  0.87   1.13
## 4       4   Park.01b_MC  1.050 0.055  0.90  0.91   1.09 -2.3  0.90  0.92   1.08
## 5       5 Shipping.03..  0.479 0.047  0.89  0.86   1.14 -1.5  0.91  0.89   1.11
## 6       6 SignUp.01ab.. -0.644 0.144  0.96  0.86   1.14 -0.6  0.98  0.75   1.25
## 7       7 Sorting.03b..  1.035 0.081  1.06  0.86   1.14  0.8  1.03  0.88   1.12
## 8       8 Sorting.03c.. -0.125 0.061  1.04  0.86   1.14  0.6  1.04  0.91   1.09
## 9       9 TicTac.01ab.. -0.102 0.032  1.27  0.92   1.08  6.5  1.24  0.94   1.06
## 10     10 Travel.01_N.. -0.072 0.052  0.99  0.93   1.07 -0.4  0.99  0.97   1.03
## 11     11 SignUp.01a_..  0.796 0.090  0.95  0.88   1.12 -0.9  0.96  0.94   1.06
## 12     12 Delivery.01..  0.005 0.145  0.90  0.70   1.30 -0.6  0.95  0.72   1.28
##     W.T
## 1  -2.2
## 2  -1.9
## 3  -1.9
## 4  -2.5
## 5  -1.8
## 6  -0.2
## 7   0.6
## 8   0.8
## 9   7.1
## 10 -1.0
## 11 -1.4
## 12 -0.3

## W.fit: Weighted fit (Infit)

##       [,1]     [,2]                  
##  [1,] "item1"  "Delivery.03ab_MCMC"  
##  [2,] "item2"  "Delivery.03d_NU"     
##  [3,] "item3"  "Elevator.04c_OE"     
##  [4,] "item4"  "Park.01b_MC"         
##  [5,] "item5"  "Shipping.03abc_MCOE2"
##  [6,] "item6"  "SignUp.01ab_MCRO"    
##  [7,] "item7"  "Sorting.03b_MC"      
##  [8,] "item8"  "Sorting.03c_MC"      
##  [9,] "item9"  "TicTac.01ab_MCOEv2"  
## [10,] "item10" "Travel.01_NU"        
## [11,] "item11" "SignUp.01a_MC"       
## [12,] "item12" "Delivery.01b_MC"

5.6 Step parameters

Delta is the point at which the probability of being in category k − 1 and category k is equals. This mathematical fact provides an interpretation for the delta (d) parameters (Wu et al., 2016)

##    n_item          item step    est error U.fit U.Low U.High  U.T W.fit W.Low
## 1       1 Delivery.03..    0     NA    NA  0.88  0.91   1.09 -2.9  0.93  0.93
## 2       1 Delivery.03..    1 -0.856 0.062  0.99  0.91   1.09 -0.2  0.99  0.99
## 3       1 Delivery.03..    2  0.856    NA  0.93  0.91   1.09 -1.7  0.97  0.92
## 4       2 Delivery.03..    0     NA    NA  0.92  0.91   1.09 -1.8  0.93  0.96
## 5       2 Delivery.03..    1 -1.680 0.066  0.95  0.91   1.09 -1.2  0.95  0.97
## 6       2 Delivery.03..    2  1.680    NA  0.88  0.91   1.09 -2.8  0.99  0.69
## 7       3 Elevator.04..    0     NA    NA  0.88  0.85   1.15 -1.7  0.98  0.84
## 8       3 Elevator.04..    1 -1.670 0.137  0.95  0.85   1.15 -0.6  0.98  0.92
## 9       3 Elevator.04..    2 -0.923 0.112  0.96  0.85   1.15 -0.5  0.98  0.93
## 10      3 Elevator.04..    3  0.291 0.162  0.73  0.85   1.15 -4.1  0.92  0.79
## 11      3 Elevator.04..    4  2.302    NA  0.92  0.85   1.15 -1.1  0.96  0.00
## 12      4   Park.01b_MC    0     NA    NA  0.89  0.91   1.09 -2.4  0.91  0.96
## 13      4   Park.01b_MC    1 -1.163 0.066  0.94  0.91   1.09 -1.3  0.95  0.98
## 14      4   Park.01b_MC    2  1.163    NA  0.85  0.91   1.09 -3.4  0.98  0.79
## 15      5 Shipping.03..    0     NA    NA  0.93  0.86   1.14 -0.9  0.95  0.91
## 16      5 Shipping.03..    1  2.033 0.114  0.91  0.86   1.14 -1.2  0.99  0.53
## 17      5 Shipping.03..    2 -2.155 0.119  0.96  0.86   1.14 -0.5  0.98  0.90
## 18      5 Shipping.03..    3  0.122    NA  0.83  0.86   1.14 -2.5  0.90  0.87
## 19      6 SignUp.01ab..    0     NA    NA  0.94  0.86   1.14 -0.8  0.98  0.52
## 20      6 SignUp.01ab..    1 -2.837 0.151  0.97  0.86   1.14 -0.4  0.99  0.78
## 21      6 SignUp.01ab..    2  2.837    NA  0.88  0.86   1.14 -1.8  0.98  0.73
## 22      7 Sorting.03b..    0     NA    NA  1.01  0.86   1.14  0.1  1.00  0.94
## 23      7 Sorting.03b..    1 -0.775 0.104  0.98  0.86   1.14 -0.3  0.98  0.96
## 24      7 Sorting.03b..    2  0.775    NA  1.25  0.86   1.14  3.2  1.05  0.75
## 25      8 Sorting.03c..    0     NA    NA  1.09  0.86   1.14  1.3  1.08  0.90
## 26      8 Sorting.03c..    1  1.025 0.149  0.98  0.86   1.14 -0.3  1.00  0.79
## 27      8 Sorting.03c..    2 -1.025    NA  0.99  0.86   1.14 -0.2  0.99  0.93
## 28      9 TicTac.01ab..    0     NA    NA  0.97  0.92   1.08 -0.9  0.99  0.88
## 29      9 TicTac.01ab..    1 -1.215 0.059  1.23  0.92   1.08  5.5  1.16  0.97
## 30      9 TicTac.01ab..    2  0.422 0.060  1.13  0.92   1.08  3.4  1.07  0.95
## 31      9 TicTac.01ab..    3  0.792    NA  1.31  0.92   1.08  7.3  1.11  0.92
## 32     12 Delivery.01..    0     NA    NA  0.88  0.70   1.30 -0.7  0.95  0.67
## 33     12 Delivery.01..    1  1.037 0.338  0.98  0.70   1.30 -0.1  1.01  0.49
## 34     12 Delivery.01..    2 -1.037    NA  0.93  0.70   1.30 -0.4  0.97  0.78
##    W.High  W.T
## 1    1.07 -2.2
## 2    1.01 -1.7
## 3    1.08 -0.7
## 4    1.04 -3.5
## 5    1.03 -3.3
## 6    1.31  0.0
## 7    1.16 -0.3
## 8    1.08 -0.6
## 9    1.07 -0.6
## 10   1.21 -0.8
## 11   2.07  0.1
## 12   1.04 -5.4
## 13   1.02 -4.9
## 14   1.21 -0.1
## 15   1.09 -1.0
## 16   1.47  0.1
## 17   1.10 -0.3
## 18   1.13 -1.5
## 19   1.48  0.0
## 20   1.22  0.0
## 21   1.27 -0.1
## 22   1.06  0.1
## 23   1.04 -1.0
## 24   1.25  0.4
## 25   1.10  1.6
## 26   1.21  0.0
## 27   1.07 -0.2
## 28   1.12 -0.1
## 29   1.03  9.6
## 30   1.05  2.8
## 31   1.08  2.8
## 32   1.33 -0.3
## 33   1.51  0.1
## 34   1.22 -0.3

5.7 Thurstonian thresholds

For partial credit items, to achieve a score of 2, students would generally need to accomplish more tasks than for achieving a score of 1. To reflect this “cumulative achievement”, the Thurstonian thresholds are sometimes used as indicators of “score difficulties”. The Thurstonian threshold for a score category is defined as the ability at which the probability of achieving that score or higher reaches 0.50.

##                        [,1]  [,2]  [,3]  [,4]
## Delivery.03ab_MCMC   -0.801 1.201    NA    NA
## Delivery.03d_NU      -0.393 3.033    NA    NA
## Elevator.04c_OE      -0.907 0.237 1.494 3.503
## Park.01b_MC          -0.199 2.299    NA    NA
## Shipping.03abc_MCOE2  0.127 0.207 0.950    NA
## SignUp.01ab_MCRO     -3.484 2.197    NA    NA
## Sorting.03b_MC        0.094 1.975    NA    NA
## Sorting.03c_MC       -0.304 0.053    NA    NA
## TicTac.01ab_MCOEv2   -1.484 0.098 1.097    NA
## Travel.01_NU         -0.071    NA    NA    NA
## SignUp.01a_MC         0.796    NA    NA    NA
## Delivery.01b_MC      -0.172 0.181    NA    NA

##                        AEI1   AEI2  AEI3  AEI4
## Delivery.03ab_MCMC   -0.801     NA 1.201    NA
## Delivery.03d_NU      -0.393     NA    NA 3.033
## Elevator.04c_OE      -0.907  0.237 1.494 3.503
## Park.01b_MC          -0.199     NA    NA 2.299
## Shipping.03abc_MCOE2  0.127  0.207 0.950    NA
## SignUp.01ab_MCRO     -3.484     NA 2.197    NA
## Sorting.03b_MC           NA  0.094 1.975    NA
## Sorting.03c_MC           NA -0.304 0.053    NA
## TicTac.01ab_MCOEv2   -1.484  0.098 1.097    NA
## Travel.01_NU             NA -0.071    NA    NA
## SignUp.01a_MC            NA  0.796    NA    NA
## Delivery.01b_MC      -0.172  0.181    NA    NA

##       AEI1              AEI2               AEI3            AEI4      
##  Min.   :-3.4840   Min.   :-0.30400   Min.   :0.053   Min.   :2.299  
##  1st Qu.:-1.0513   1st Qu.: 0.05275   1st Qu.:1.024   1st Qu.:2.666  
##  Median :-0.5970   Median : 0.13950   Median :1.201   Median :3.033  
##  Mean   :-0.9141   Mean   : 0.15475   Mean   :1.281   Mean   :2.945  
##  3rd Qu.:-0.1923   3rd Qu.: 0.21450   3rd Qu.:1.734   3rd Qu.:3.268  
##  Max.   : 0.1270   Max.   : 0.79600   Max.   :2.197   Max.   :3.503  
##  NA's   :4         NA's   :4          NA's   :5       NA's   :9

6 Links to CQ files

6.1 Data file

https://www.dropbox.com/s/k0qh8co2bxhd39e/AEI_data.txt?dl=0

6.2 Command file

https://www.dropbox.com/s/rbt1yt4dtrtzbmw/command_AEI.txt?dl=0

6.3 Label file

https://www.dropbox.com/s/k40ujtzbr7b0xbc/AEI_labels.txt?dl=0

6.4 CTT Item Analysis (ITN) file

https://www.dropbox.com/s/ylkvg1zr9xy04j6/AEI_ITN.txt?dl=0

6.5 SHOW file

https://www.dropbox.com/s/d86sj739mhttwwz/AEI_SHOW.txt?dl=0

6.6 EAP file

https://www.dropbox.com/s/bff6zxjn7l8fxwn/AEI_EAP.txt?dl=0

6.7 WLE file

https://www.dropbox.com/s/lqs3axpe32w5p4u/AEI_WLE.txt?dl=0

6.8 Parameter file

https://www.dropbox.com/s/lltcd25ggxmj02s/AEI_PARMS.txt?dl=0

7 Item fit graph

8 Wright Map

##                           l1         l2        l3       l4
## Sign.01ab.MCRO   -3.48275757 2.19680786        NA       NA
## TicT.01ab.MCOEv2 -1.48379517 0.09805298 1.0963440       NA
## Elev.04c.OE      -0.90756226 0.23574829 1.4924011 3.501251
## Deliv.03ab.MCMC  -0.80044556 1.20089722        NA       NA
## Deliv.03d.NU     -0.39193726 3.03250122        NA       NA
## Sort.03c.MC      -0.30422974 0.05245972        NA       NA
## Park.01b_MC      -0.19821167 2.29843140        NA       NA
## Deliv.01b.MC     -0.17184448 0.18081665        NA       NA
## Trav.01.NU       -0.07150269         NA        NA       NA
## Sort.03b.MC       0.09292603 1.97341919        NA       NA
## Ship.03abc.MCOE2  0.12606812 0.20626831 0.9480286       NA
## Sign.01a.MC       0.79568481         NA        NA       NA

9 Category Characteristic Curves (CCC)

10 Calibration summary from TAM

## ------------------------------------------------------------
## TAM 3.6-45 (2021-04-22 14:35:52) 
## R version 4.0.5 (2021-03-31) x86_64, darwin17.0 | nodename=Permans-MacBook-Pro-4.local | login=root 
## 
## Date of Analysis: 2023-02-08 13:11:16 
## Time difference of 0.3720419 secs
## Computation time: 0.3720419 
## 
## Multidimensional Item Response Model in TAM 
## 
## IRT Model: PCM2
## Call:
## TAM::tam.mml(resp = data2[, c(2:13)], Y = data2[, 14:15], irtmodel = "PCM2", 
##     verbose = FALSE)
## 
## ------------------------------------------------------------
## Number of iterations = 76 
## Numeric integration with 21 integration points
## 
## Deviance = 15552.14 
## Log likelihood = -7776.07 
## Number of persons = 1804 
## Number of persons used = 1684 
## Number of items = 12 
## Number of estimated parameters = 29 
##     Item threshold parameters = 26 
##     Item slope parameters = 0 
##     Regression parameters = 2 
##     Variance/covariance parameters = 1 
## 
## AIC = 15610  | penalty=58    | AIC=-2*LL + 2*p 
## AIC3 = 15639  | penalty=87    | AIC3=-2*LL + 3*p 
## BIC = 15768  | penalty=215.44    | BIC=-2*LL + log(n)*p 
## aBIC = 15675  | penalty=123.24    | aBIC=-2*LL + log((n-2)/24)*p  (adjusted BIC) 
## CAIC = 15797  | penalty=244.44    | CAIC=-2*LL + [log(n)+1]*p  (consistent AIC) 
## AICc = 15611  | penalty=59.05    | AICc=-2*LL + 2*p + 2*p*(p+1)/(n-p-1)  (bias corrected AIC) 
## GHP = 0.91911     | GHP=( -LL + p ) / (#Persons * #Items)  (Gilula-Haberman log penalty) 
## 
## ------------------------------------------------------------
## EAP Reliability
## [1] 0.461
## ------------------------------------------------------------
## Covariances and Variances
##      [,1]
## [1,] 0.26
## ------------------------------------------------------------
## Correlations and Standard Deviations (in the diagonal)
##      [,1]
## [1,] 0.51
## ------------------------------------------------------------
## Regression Coefficients
##         [,1]
## [1,] 0.00000
## [2,] 0.17718
## [3,] 0.73412
## ------------------------------------------------------------
## Standardized Coefficients
##        parm dim    est  StdYX   StdX   StdY
## 1 Intercept   1 0.0000     NA     NA     NA
## 2       mid   1 0.1772 0.1480 0.0846 0.3101
## 3      post   1 0.7341 0.4759 0.2720 1.2847
## 
## ** Explained Variance R^2
## [1] 0.2026
## ** SD Theta
## [1] 0.5714
## ** SD Predictors
## Intercept       mid      post 
##    0.0000    0.4772    0.3705 
## ------------------------------------------------------------
## Item Parameters -A*Xsi
##                    item    N     M xsi.item AXsi_.Cat1 AXsi_.Cat2 AXsi_.Cat3
## 1    Delivery.03ab_MCMC 1052 0.955    0.200     -0.655      0.400         NA
## 2       Delivery.03d_NU  963 0.642    1.320     -0.359      2.640         NA
## 3       Elevator.04c_OE  363 1.430    1.083     -0.586     -0.426      0.947
## 4           Park.01b_MC  962 0.634    1.050     -0.112      2.100         NA
## 5  Shipping.03abc_MCOE2  382 1.223    0.477      2.511      0.833      1.432
## 6      SignUp.01ab_MCRO  396 1.058   -0.643     -3.479     -1.286         NA
## 7        Sorting.03b_MC  387 0.623    1.033      0.259      2.066         NA
## 8        Sorting.03c_MC  387 1.204   -0.126      0.900     -0.252         NA
## 9    TicTac.01ab_MCOEv2 1354 1.572   -0.102     -1.316     -0.995     -0.305
## 10         Travel.01_NU 1589 0.558   -0.072     -0.072         NA         NA
## 11        SignUp.01a_MC  571 0.357    0.796      0.796         NA         NA
## 12      Delivery.01b_MC   86 1.488    0.004      1.042      0.009         NA
##    AXsi_.Cat4 B.Cat1.Dim1 B.Cat2.Dim1 B.Cat3.Dim1 B.Cat4.Dim1
## 1          NA           1           2           0           0
## 2          NA           1           2           0           0
## 3       4.332           1           2           3           4
## 4          NA           1           2           0           0
## 5          NA           1           2           3           0
## 6          NA           1           2           0           0
## 7          NA           1           2           0           0
## 8          NA           1           2           0           0
## 9          NA           1           2           3           0
## 10         NA           1           0           0           0
## 11         NA           1           0           0           0
## 12         NA           1           2           0           0
## 
## Item Parameters Xsi
##                               xsi se.xsi
## Delivery.03ab_MCMC          0.200  0.047
## Delivery.03d_NU             1.320  0.061
## Elevator.04c_OE             1.083  0.060
## Park.01b_MC                 1.050  0.055
## Shipping.03abc_MCOE2        0.477  0.047
## SignUp.01ab_MCRO           -0.643  0.144
## Sorting.03b_MC              1.033  0.081
## Sorting.03c_MC             -0.126  0.061
## TicTac.01ab_MCOEv2         -0.102  0.032
## Travel.01_NU               -0.072  0.052
## SignUp.01a_MC               0.796  0.090
## Delivery.01b_MC             0.005  0.145
## Delivery.03ab_MCMC_step1   -0.855  0.062
## Delivery.03d_NU_step1      -1.679  0.066
## Elevator.04c_OE_step1      -1.669  0.137
## Elevator.04c_OE_step2      -0.923  0.112
## Elevator.04c_OE_step3       0.290  0.162
## Park.01b_MC_step1          -1.162  0.066
## Shipping.03abc_MCOE2_step1  2.033  0.114
## Shipping.03abc_MCOE2_step2 -2.155  0.119
## SignUp.01ab_MCRO_step1     -2.836  0.151
## Sorting.03b_MC_step1       -0.775  0.104
## Sorting.03c_MC_step1        1.026  0.149
## TicTac.01ab_MCOEv2_step1   -1.214  0.059
## TicTac.01ab_MCOEv2_step2    0.422  0.060
## Delivery.01b_MC_step1       1.037  0.338
## 
## Item Parameters in IRT parameterization
##                    item alpha   beta tau.Cat1 tau.Cat2 tau.Cat3 tau.Cat4
## 1    Delivery.03ab_MCMC     1  0.200   -0.855    0.855       NA       NA
## 2       Delivery.03d_NU     1  1.320   -1.679    1.679       NA       NA
## 3       Elevator.04c_OE     1  1.083   -1.669   -0.923    0.290    2.302
## 4           Park.01b_MC     1  1.050   -1.162    1.162       NA       NA
## 5  Shipping.03abc_MCOE2     1  0.477    2.033   -2.155    0.122       NA
## 6      SignUp.01ab_MCRO     1 -0.643   -2.836    2.836       NA       NA
## 7        Sorting.03b_MC     1  1.033   -0.775    0.775       NA       NA
## 8        Sorting.03c_MC     1 -0.126    1.026   -1.026       NA       NA
## 9    TicTac.01ab_MCOEv2     1 -0.102   -1.214    0.422    0.791       NA
## 10         Travel.01_NU     1 -0.072       NA       NA       NA       NA
## 11        SignUp.01a_MC     1  0.796       NA       NA       NA       NA
## 12      Delivery.01b_MC     1  0.004    1.037   -1.037       NA       NA