DEIJ Program Statements

Basic Analyses

ICC calculations

Phase 0

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
6	0	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

 

ICC Results

ICC for Each Institution in Phase 0

University	Inter-Rater Reliability	Intra-Rater Reliability	Total # of Coders
Duke University	0.6103062	0.6103062	9
University of Arizona	0.6923521	0.6975842	9
University of Denver	0.4455760	0.4504906	9
University of Kentucky	0.6662142	0.6732197	8
University of Massachusetts- Amherst	0.5848952	0.5984738	8
University of Virginia	0.7557958	0.7561288	9

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
6	0	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables combined by institution (117 variables x 6 institutions)	702

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]

 

 

ICC Results

Combined ICC (Subject & Rater Treated as Random Factors)

Inter-Rater Reliability	Intra-Rater Reliability	Total # of Coders
0.6680542	0.674315	9

Phase 1

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	5	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

 

ICC Results

ICC table for Phase 1 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Intra-Rater Reliability	Total # of Coders
Boston University	0.9101183	0.9101183	3
Florida State University	0.9053165	0.9053165	3
Michigan State University	0.8434732	0.8470519	3
Northwestern University	0.8994698	0.8996737	3
Ohio State University	0.7998931	0.8024199	3
Pennsylvania State University	0.7392276	0.7392276	3
Temple University	0.7020730	0.7108420	3
University of Delaware	0.9290791	0.9290791	3
University of Hawaii	0.8505783	0.8513343	3
University of Illinois- Urbana Champaign	0.7766073	0.7768048	3
University of Kansas (adult)	0.9610869	0.9619255	3
University of Maryland	0.7677293	0.7703150	3
University of Michigan	1.0000000	1.0000000	3
University of Pittsburgh	0.9792746	0.9792746	3
University of Rochester	0.8303459	0.8303459	3
University of Southern California	0.8465284	0.8529848	3
University of Utah	0.9827405	0.9827405	3
University of Wisconsin- Milwaukee	0.9385244	0.9391680	3
Virginia Tech	0.7089391	0.7089391	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	5	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

 

 

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.8666875	0.0995964	9

Analyses

Planned Actions (toward DEIJ)

Enacted Actions (toward DEIJ)

References

• https://cran.r-project.org/web/packages/irrICC/vignettes/UserGuide.pdf

• https://stackoverflow.com/questions/45625887/r-iccirr-attempting-one-way-model-with-multiple-ratings-per-rater-per-subjec

• https://stackoverflow.com/questions/38164218/intraclass-correlation-coefficient-in-r-how-to-deal-with-nas-and-not-much-overl

• https://www.statology.org/intraclass-correlation-coefficient/

• https://aosmith.rbind.io/2019/11/27/split-example/

• https://stackoverflow.com/questions/21878974/wrap-long-axis-labels-via-labeller-label-wrap-in-ggplot2