Basic Analyses

ICC calculations

Phase 0

Per Institution

ICC Model Description
ICC Model Type and Description
# of APCS Programs coded # of APCS Programs with no DEIJ Model Type Description Subjects # of Subjects
6 0 2-way random effects model Raters randomly drawn from population, all raters code all subjects DEIJ variables per institution 117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

Sample Suject and Rater Description  

ICC Results
ICC for Each Institution in Phase 0
University Inter-Rater Reliability Intra-Rater Reliability Total # of Coders
Duke University 0.6103062 0.6103062 9
University of Arizona 0.7075858 0.7113262 9
University of Denver 0.4455760 0.4504906 9
University of Kentucky 0.6662142 0.6732197 8
University of Massachusetts- Amherst 0.5848952 0.5984738 8
University of Virginia 0.7557958 0.7561288 9

Composite (across all institutions)

ICC Model Description
ICC Model Type and Description
# of APCS Programs coded # of APCS Programs with no DEIJ Model Type Description Subjects # of Subjects
6 0 2-way random effects model Raters randomly drawn from population, all raters code all subjects DEIJ variables combined by institution (117 variables x 6 institutions) 702

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]

 

Sample Suject and Rater Description  

ICC Results
Combined ICC (Subject & Rater Treated as Random Factors)
Inter-Rater Reliability Intra-Rater Reliability Total # of Coders
0.669455 0.6756377 9

Phase 1

Per Institution

ICC Model Description
ICC Model Type and Description
# of APCS Programs coded # of APCS Programs with no DEIJ Model Type Description Subjects # of Subjects
24 5 2-way random effects model Raters randomly drawn from population, all raters code all subjects DEIJ variables per institution 117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

Sample Subject and Rater Description  

ICC Results
ICC table for Phase 1 (only institutions with DEIJ statements)
University Inter-Rater Reliability Intra-Rater Reliability Total # of Coders
Boston University 0.9101183 0.9101183 3
Florida State University 0.9053165 0.9053165 3
Michigan State University 0.8434732 0.8470519 3
Northwestern University 0.8994698 0.8996737 3
Ohio State University 0.7998931 0.8024199 3
Pennsylvania State University 0.7392276 0.7392276 3
Temple University 0.7020730 0.7108420 3
University of Delaware 0.9290791 0.9290791 3
University of Hawaii 0.8505783 0.8513343 3
University of Illinois- Urbana Champaign 0.7766073 0.7768048 3
University of Kansas (adult) 0.9610869 0.9619255 3
University of Maryland 0.7684887 0.7705954 3
University of Michigan 1.0000000 1.0000000 3
University of Pittsburgh 0.9792746 0.9792746 3
University of Rochester 0.8303459 0.8303459 3
University of Southern California 0.8465284 0.8529848 3
University of Utah 0.9827405 0.9827405 3
University of Wisconsin- Milwaukee 0.9385244 0.9391680 3
Virginia Tech 0.7089391 0.7089391 3

Composite (across all institutions)

ICC Model Description
ICC Model Type and Description
# of APCS Programs coded # of APCS Programs with no DEIJ Model Type Description Subjects # of Subjects
24 5 1-way random effects model Not all subjects are rated by the same roster of raters DEIJ variables combined by institution (117 variables x 24 institutions) 2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

 

Sample Suject and Rater Description  

ICC Results
ICC (One-way Effects) Across Institutions(Subjects)
Inter-Rater Reliability Error Variance Total # of Coders
0.8667913 0.0994777 9

Phase 2

ICC Results

Per Institution
ICC table for Phase 2 (only institutions with DEIJ statements)
University Inter-Rater Reliability Intra-Rater Reliability Total # of Coders
Stony 0.9198967 0.9207535 3
MN 0.9651233 0.9652435 3
Miami 0.7335764 0.7353263 3
UIC 0.2832246 0.2832246 3
WUSTL 0.8352501 0.8369821 3
OklahomaS 0.8404088 0.8409142 3
Oregon 0.7986111 0.7986111 3
MarylandIntern 0.7727835 0.7788660 3
SDSU 0.8936447 0.8936447 3
UT 0.6439077 0.6495750 3
UCLA 0.7498350 0.7510233 3
Indiana 0.8998206 0.9001528 3
UW 0.7409095 0.7420709 3
Georgia 0.8614480 0.8654906 3
Nevada 0.7553223 0.7580202 3
SemelIntern 0.8833500 0.8833500 3
MUSCIntern 0.7217781 0.7272146 3
GMU 0.8424809 0.8424809 3
WPIntern 0.5901932 0.5959394 3
FIU 0.9900561 0.9904613 3
Composite (across all institutions)
ICC (One-way Effects) Across Institutions(Subjects)
Inter-Rater Reliability Error Variance Total # of Coders
0.7771125 0.1968186 9

Analyses

Planned Actions (toward DEIJ)

Planned Action Steps along Dimensions of Diversity-1 Planned Action Steps along Dimensions of Diversity-2

Enacted Actions (toward DEIJ)

Enacted Action Steps along Dimensions of Diversity-1 Enacted Action Steps along Dimensions of Diversity-2 Enacted Action Steps along Dimensions of Diversity-3

The Why for DEIJ

Moral Rationales

Rationales supported by intrinsic values or principles (Starck, Sinclair, & Shelton, 2021)

Indicators of Moral Framing-1 Indicators of Moral Framing-2 Indicators of Moral Framing-3 Indicators of Moral Framing-4 Indicators of Moral Framing-5 Indicators of Moral Framing-6

Instrumental Rationales

Indicators of Instrumental Framing-1 Indicators of Instrumental Framing-2

Moral and Instrumental Rationales - Combined