DEIJ Program Statements

Basic Analyses

ICC calculations

Phase 0

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
6	0	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

 

ICC Results

ICC for Each Institution in Phase 0

University	Inter-Rater Reliability	Total # of Coders
Duke University	0.7073360	9
University of Arizona	0.7083053	9
University of Denver	0.7069354	9
University of Kentucky	0.7042582	8
University of Massachusetts- Amherst	0.7126281	8
University of Virginia	0.7588116	8

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
6	0	1-way random effects model	Not all progams are rated by the same roster of coders	DEIJ variables combined by institution (117 variables x 6 institutions)	702

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

 

ICC Results

Combined ICC (Subject & Rater Treated as Random Factors)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.7241006	0.1898306	9

Phase 1

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	5	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]  

 

ICC Results

ICC table for Phase 1 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Total # of Coders
Boston University	0.9101183	3
Florida State University	0.9053165	3
Michigan State University	0.8434732	3
Northwestern University	0.8994698	3
Ohio State University	0.7998931	3
Pennsylvania State University	0.7392276	3
Temple University	0.7020730	3
University of Delaware	0.9290791	3
University of Hawaii	0.8505783	3
University of Illinois- Urbana Champaign	0.7766073	3
University of Kansas (adult)	0.9610869	3
University of Maryland	0.7684887	3
University of Michigan	1.0000000	3
University of Pittsburgh	0.9792746	3
University of Rochester	0.8303459	3
University of Southern California	0.8465284	3
University of Utah	0.9827405	3
University of Wisconsin- Milwaukee	0.9385244	3
Virginia Tech	0.7089391	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	5	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

 

 

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.8667913	0.0994777	9

Phase 2

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	4	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC table for Phase 2 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Total # of Coders
Stony Brook University (State University of New York)	0.9198967	3
University of Minnesota	0.9651233	3
University of Miami	0.7335764	3
University of Illinois at Chicago	0.7586501	3
Washington University in St. Louis	0.8352501	3
Oklahoma State University	0.8404088	3
University of Oregon	0.7986111	3
VA Maryland Health Care System / University of Maryland Internship Consortium	0.7727835	3
San Diego State University/University of California San Diego Joint Doctoral Program	0.8936447	3
University of Texas	0.7180318	3
University of California- Los Angeles	0.7498350	3
Indiana University	0.8998206	3
University of Washington	0.7409095	3
University of Georgia	0.8614480	3
University of Nevada- Reno	0.7553223	3
UCLA, Semel Institute for Neuroscience and Human Behavior	0.8833500	3
Medical University of South Carolina	0.7217781	3
George Mason University	0.8424809	3
Western Psychiatric Institute and Clinic	0.7438884	3
Florida International University	0.9900561	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	4	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.8268194	0.1622745	9

Phase 3

Per Institution

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	2	2-way random effects model	Raters randomly drawn from population, all raters code all subjects	DEIJ variables per institution	117

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\r_{j}\text{ is rater } j \text{'s effect, } \\(sr)_{ij}\text{ is the subject-rater interaction effect associated with subject }i \text{ and rater }j\text{, and } \\ \text{ takes into the account that the effect of bias may not be the same for all subjects,} \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC table for Phase 3 (only institutions with DEIJ statements)

University	Inter-Rater Reliability	Total # of Coders
Virginia Commonwealth University	1.0000000	3
Yale University	0.8997568	3
University of Iowa	0.9287678	3
University of Memphis	0.8958450	3
University of Buffalo (State University of New York)	0.8207362	3
University of Wisconsin Department of Psychiatry	1.0000000	3
University of Pennsylvania	0.6841760	3
New York Presbyterian-Weill Cornell Medical Center	0.7446958	3
Arizona State University	0.6901666	3
University of Wisconsin	0.9089905	3
VA Boston Health Care System / Psychology Internship Training Program	0.8466676	3
University of California- Berkeley	0.7740773	3
Vanderbilt University	0.7300740	3
Purdue University	0.9302905	3
University of North Carolina at Chapel Hill	0.9881091	3
University of Missouri	0.9191087	3
Southern Methodist University	0.7615766	3
Rutgers University	0.7592262	3
McGill University	0.6641208	3
University of New Mexico	0.7070837	3
Harvard University	0.8964032	3
Brown University Medical School Consortium	0.8151281	3

Composite (across all institutions)

ICC Model Description

ICC Model Type and Description

# of APCS Programs coded	# of APCS Programs with no DEIJ	Model Type	Description	Subjects	# of Subjects
24	2	1-way random effects model	Not all subjects are rated by the same roster of raters	DEIJ variables combined by institution (117 variables x 24 institutions)	2808

Model Equation \[ Y_{ij} \sim \mu + s_{i} + r_{j} + (sr)_{ij} + \epsilon_{ij} \\\text{ } \\where\text{ } \mu\text{ is the average rating, } \\s_{i}\text{ is subject } i \text{'s effect, } \\ \epsilon_{ij}\text{ is the error effect} \]

ICC Results

ICC (One-way Effects) Across Institutions(Subjects)

Inter-Rater Reliability	Error Variance	Total # of Coders
0.8702351	0.1162156	9

Analyses

Exclusion

Mention

Acknowledgements

Planned Actions

Enacted Actions

The Why

Moral Rationales

Rationales supported by intrinsic values or principles (Starck, Sinclair, & Shelton, 2021)

  Description of moral rationales with specific examples from APCS programs. Moral rationales are broadly grouped, based on the coding manual formulated as part of this study.  

Instrumental Rationales

Indicators of Instrumental Framing-1

Moral and Instrumental Rationales - Combined

The correlation coefficient as assessed by Kendall’s tau was 0.18 (p=0.08). Although not significant, we postulate that with a higher sample size, the correlation would reach significance.

 

	Moral Rationale
Predictors	Estimates	CI	p
(Intercept)	1.33	0.72 – 1.95	<0.001
Instrumental Rationale	0.23	-0.03 – 0.50	0.084
Observations	63
R2 / R2 adjusted	0.048 / 0.032

Regression output with Moral Rationale as the response variable (moral and instrumental rationales treated as continuous for this exploratory analyses). These results suggest that a unit increase in endorsement of instrumental rationale by a program is associated with a positive (0.23) increase in moral rationale provided by the program. We also note that instrumental variables account for 4.6% of the variance toward moral rationale as endorsed by a program.However, we treat these findings with caution, since the beta-estimate for Instrumental Rationale is not significant at an alpha level of 0.05 for an N=64. Further, the variables Moral Rationale and Instrumental Rationale are coded with nominal values (0-4), rendering a 0.23 increase in moral rationale difficult to interpret.

 

	Moral Rationale
Predictors	Odds Ratios	CI	p
(Intercept)	0.37	0.18 – 0.71	0.004
Instrumental Rationale [1]	1.56	0.50 – 4.74	0.433
Observations	63
R2 Tjur	0.010

Logistic regression output with Moral Rationale as the response variable. Given the nominal/ordinal nature of the moral and instrumental frames, we differentiated moral and instrumental frames by low and high rationales. A low moral rationale was assigned to programs whose DEIJ statements were coded as “Not at all”, “A little bit”, or “Somewhat”. A high moral rationale was assigned to programs whose DEIJ statements were coded as “Quite a lot” or “Extremely”. We adopted a similar approach when coding low and high instrumental rationales for programs. These results suggest that programs with high instrumental rationales have (e^1.61) = 5 times the odds of having high moral rationales than do programs with low instrumental rationales. However, we treat this interpretation with caution, since this effect is not significant with an N=64 (alpha level=0.05). We also note that the Tjur R-squared is 0.01, suggesting that the model has poor discriminating power between programs with high moral rationales and those with low moral rationales.

References

• https://cran.r-project.org/web/packages/irrICC/vignettes/UserGuide.pdf

• https://stackoverflow.com/questions/45625887/r-iccirr-attempting-one-way-model-with-multiple-ratings-per-rater-per-subjec

• https://stackoverflow.com/questions/38164218/intraclass-correlation-coefficient-in-r-how-to-deal-with-nas-and-not-much-overl

• https://www.statology.org/intraclass-correlation-coefficient/

• https://aosmith.rbind.io/2019/11/27/split-example/

• https://stackoverflow.com/questions/21878974/wrap-long-axis-labels-via-labeller-label-wrap-in-ggplot2

• https://stackoverflow.com/questions/4560459/all-levels-of-a-factor-in-a-model-matrix-in-r

• https://stackoverflow.com/questions/52554336/plot-the-equivalent-of-correlation-matrix-for-factors-categorical-data-and-mi

• https://stats.stackexchange.com/questions/93680/treating-ordinal-variables-as-continuous-for-regression-problems

• https://www.theanalysisfactor.com/pros-and-cons-of-treating-ordinal-variables-as-nominal-or-continuous/

• https://www.analyticsvidhya.com/blog/2016/02/multinomial-ordinal-logistic-regression/

• https://stats.stackexchange.com/questions/7720/how-to-understand-output-from-rs-polr-function-ordered-logistic-regression