DEIJ Program Statements

Basic Analyses

ICC calculations

Phase 0

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
6	0	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

 

ICC Results

ICC for Each Institution in Phase 0

University	Inter-Rater Reliability	Total # of Coders
Duke University	0.7073360	9
University of Arizona	0.7083053	9
University of Denver	0.7069354	9
University of Kentucky	0.7042582	8
University of Massachusetts- Amherst	0.7126281	8
University of Virginia	0.7588116	8

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
6	0	1-way random effects model	Not all progams are rated by the same roster of coders	DEIJ variables combined by institution (117 variables x 6 institutions)	702

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

 

ICC Results

Combined ICC (Subject & Rater Treated as Random Factors)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.7241006	0.1898306	9

Phase 1

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	5	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

 

ICC Results

ICC table for Phase 1 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Total # of Coders
Boston University	0.9101183	3
Florida State University	0.9053165	3
Michigan State University	0.8434732	3
Northwestern University	0.8994698	3
Ohio State University	0.7998931	3
Pennsylvania State University	0.7392276	3
Temple University	0.7020730	3
University of Delaware	0.9290791	3
University of Hawaii	0.8505783	3
University of Illinois- Urbana Champaign	0.7766073	3
University of Kansas (adult)	0.9610869	3
University of Maryland	0.7684887	3
University of Michigan	1.0000000	3
University of Pittsburgh	0.9792746	3
University of Rochester	0.8303459	3
University of Southern California	0.8465284	3
University of Utah	0.9827405	3
University of Wisconsin- Milwaukee	0.9385244	3
Virginia Tech	0.7089391	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	5	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

 

 

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.8667913	0.0994777	9

Phase 2

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	4	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC table for Phase 2 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Total # of Coders
Stony Brook University (State University of New York)	0.9198967	3
University of Minnesota	0.9651233	3
University of Miami	0.7335764	3
University of Illinois at Chicago	0.7586501	3
Washington University in St. Louis	0.8352501	3
Oklahoma State University	0.8404088	3
University of Oregon	0.7986111	3
VA Maryland Health Care System / University of Maryland Internship Consortium	0.7727835	3
San Diego State University/University of California San Diego Joint Doctoral Program	0.8936447	3
University of Texas	0.7180318	3
University of California- Los Angeles	0.7498350	3
Indiana University	0.8998206	3
University of Washington	0.7409095	3
University of Georgia	0.8614480	3
University of Nevada- Reno	0.7553223	3
UCLA, Semel Institute for Neuroscience and Human Behavior	0.8833500	3
Medical University of South Carolina	0.7217781	3
George Mason University	0.8424809	3
Western Psychiatric Institute and Clinic	0.7438884	3
Florida International University	0.9900561	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	4	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.8268194	0.1622745	9

Phase 3

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	2	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC table for Phase 3 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Total # of Coders
Virginia Commonwealth University	1.0000000	3
Yale University	0.8997568	3
University of Iowa	0.9287678	3
University of Memphis	0.8958450	3
University of Buffalo (State University of New York)	0.8207362	3
University of Wisconsin Department of Psychiatry	0.5195157	3
University of Pennsylvania	0.6841760	3
New York Presbyterian-Weill Cornell Medical Center	0.7446958	3
Arizona State University	0.6901666	3
University of Wisconsin	0.9089905	3
VA Boston Health Care System / Psychology Internship Training Program	0.8466676	3
University of California- Berkeley	0.7740773	3
Vanderbilt University	0.7300740	3
Purdue University	0.9302905	3
University of North Carolina at Chapel Hill	0.5669715	3
University of Missouri	0.9191087	3
Southern Methodist University	0.6669548	3
Rutgers University	0.7592262	3
McGill University	0.6641208	3
University of New Mexico	0.7070837	3
Harvard University	0.8964032	3
Brown University Medical School Consortium	0.8151281	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	2	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.7899184	0.1793685	9

Analyses

Exclusion

Mention

Acknowledgements

Planned Actions

Enacted Actions

The Why

Moral Rationales

Rationales supported by intrinsic values or principles (Starck, Sinclair, & Shelton, 2021)

Instrumental Rationales

Moral and Instrumental Rationales - Combined

	Moral Rationale
Predictors	Estimates	CI	p
(Intercept)	1.55	0.92 – 2.18	<0.001
Instrumental Rationale	0.16	-0.12 – 0.43	0.257
Observations	63
R2 / R2 adjusted	0.021 / 0.005

	Moral Rationale
Predictors	Odds Ratios	CI	p
(Intercept)	0.41	0.20 – 0.79	0.010
Instrumental Rationale [1]	1.38	0.45 – 4.14	0.565
Observations	63
R2 Tjur	0.005

Programs with high instrumental rationales have (e^.2187) = 1.24 times the odds of having high moral rationales versus programs with low instrumental rationales. Put differently, programs with high instrumental rationales have 24% greater odds of endorsing high moral rationales than do the programs with low instrumental rationales. However, this effect was not significant for an N=63 (i.e., excluding programs where it was unclear whether the program was providing a moral or instrumental rationale, and excluding programs with no DEIJ statement). We also note that the Tjur R-squared is 0.002, suggesting that the model has poor discriminating power between programs with high moral rationales and those with low moral rationales.

References

• https://cran.r-project.org/web/packages/irrICC/vignettes/UserGuide.pdf

• https://stackoverflow.com/questions/45625887/r-iccirr-attempting-one-way-model-with-multiple-ratings-per-rater-per-subjec

• https://stackoverflow.com/questions/38164218/intraclass-correlation-coefficient-in-r-how-to-deal-with-nas-and-not-much-overl

• https://www.statology.org/intraclass-correlation-coefficient/

• https://aosmith.rbind.io/2019/11/27/split-example/

• https://stackoverflow.com/questions/21878974/wrap-long-axis-labels-via-labeller-label-wrap-in-ggplot2

• https://stackoverflow.com/questions/4560459/all-levels-of-a-factor-in-a-model-matrix-in-r

• https://stackoverflow.com/questions/52554336/plot-the-equivalent-of-correlation-matrix-for-factors-categorical-data-and-mi

• https://stats.stackexchange.com/questions/93680/treating-ordinal-variables-as-continuous-for-regression-problems

• https://www.theanalysisfactor.com/pros-and-cons-of-treating-ordinal-variables-as-nominal-or-continuous/

• https://www.analyticsvidhya.com/blog/2016/02/multinomial-ordinal-logistic-regression/

• https://stats.stackexchange.com/questions/7720/how-to-understand-output-from-rs-polr-function-ordered-logistic-regression