CARPS Reproducibility Report
Report Details
articleID <- "16-2-2015_PS" # insert the article ID code here e.g., "10-3-2015_PS"
reportType <- 'final' # specify whether this is the 'pilot' report or 'final' report
pilotNames <- 'Emily Hembacher' # insert the pilot's name here e.g., "Tom Hardwicke". If there are multiple cpilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
copilotNames <- 'Michael Frank, Tom Hardwicke' # # insert the co-pilot's name here e.g., "Michael Frank". If there are multiple co-pilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
pilotTTC <- 300 # insert the pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
copilotTTC <- 120 # insert the co-pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
pilotStartDate <- as.Date("06/13/18", format = "%m/%d/%y") # insert the pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
copilotStartDate <- as.Date("10/19/18", format = "%m/%d/%y") # insert the co-pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
completionDate <- as.Date("08/24/18", format = "%m/%d/%y") # copilot insert the date of final report completion (after any necessary rounds of author assistance) in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")Methods summary:
The authors attempted to replicate previous findings that conception risk is related to women’s racial bias. Specifically, previous research had found that women who had a higher likelihood of becoming pregnant given the day in their menstrual cycle exhibited stronger racial biases favoring White men over Black men on a variety of measures. The authors of the present study attempted to replicate this in a high-powered design. In Study 1 (which we attempt to reproduce in this report), 867 women were recruited. Participants’ current menstrual cycle day was used to estimate conception-risk. Participants’ explicit racial biases (explicit racial preference, explicit warmth towards Whites, explicit warmth towards Blacks) were estimated from self-report measures and implicit racial bias were estimated from Implicit Association Tests. Correlations with 95% confidence intervals were reported between conception risk and the explicit and implicit bias measures.
Target outcomes:
We will attempt to reproduce all findings related to Study 1 in the quoted text:
For all racial measures, higher scores represented pro-White bias. The mean score for implicit racial bias was 0.34 (SD = 0.43) in Study 1, 0.33 (SD = 0.42) in Study 2, 0.27 (SD = 0.41) in Study 3, and 0.18 (SD = 0.37) in Study 4. The mean score for explicit racial bias was 0.30 (SD = 1.09) in Study 1, 0.30 (SD = 1.05) in Study 2, 0.27 (SD = 0.83) in Study 3, and 0.53 (SD = 1.05) in Study 4. Mean rated warmth toward European Americans was 7.18 (SD = 2.07) in Study 1 and 7.15 (SD = 2.00) in Study 2, and mean rated warmth toward African Americans was 6.69 (SD = 2.06) in Study 1 and 6.78 (SD = 2.12) in Study 2. The mean score for explicit physicality stereotypes was 0.80 (SD = 0.83) in Study 3 and 0.81 (SD = 0.85) in Study 4, and the mean score for implicit physicality stereotypes was 0.26 (SD = 0.36) in Study 3 and 0.21 (SD = 0.38) in Study 4. As in Navarrete et al. (2009), we created composite bias scores in each study by standardizing and then averaging all racial-bias measures (warmth toward African Americans was subtracted from warmth toward European Americans to create a warmth difference score prior to standardizing). Mean conception risk was 0.03 (SD = 0.03) in Study 1, 0.03 (SD = 0.03) in Study 2, 0.04 (SD = 0.03) in Study 3, and 0.03 (SD = 0.03) in Study 4. Correlations were computed between conception risk and each race variable, including the composite score. To best match the existing literature, we computed correlations for White women only (see the Supplemental Material for additional analyses). Conception risk did not significantly correlate with any of the individual measures of racial bias (see Table S4 in the Supplemental Material) or with the composite score in Study 1, r = .00, 95% confidence interval (CI) = [−.09, .08]; Study 2, r = .00, 95% CI = [−.09, .08]; Study 3, r = .03, 95% CI = [−.17, .22]; or Study 4, r = .05, 95% CI = [−.11, .21]. (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
Step 1: Load packages and prepare report object
# load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(ReproReports) # custom report functions
library(magrittr)
#library(Hmisc)# Prepare report object. This will be updated automatically by the reproCheck function each time values are compared
reportObject <- data.frame(dummyRow = TRUE, reportedValue = NA, obtainedValue = NA, valueType = NA, percentageError = NA, comparisonOutcome = NA, eyeballCheck = NA)Step 2: Load data
Load full raw data. Authors also included a pre-processed data file with excluded participants and standardized variables, but we will attempt to work from the raw data.
d_raw <-read_excel('data/study1/Study_1_full.xls')Step 3: Tidy data
Add variable for subject ID; otherwise data is already in tidy format
d_raw$sid <- 1:length(d_raw$session_status)Step 4: Run analysis
Pre-processing
…we excluded women who: (a) reported contraceptive use, that they were currently pregnant or uncertain of current pregnancy status, as well as women whose current cycle was day 41 or higher or had a previous cycle of irregular length - less than 20 days or more than 40 days; or (b) women who reported infertility due to menopause or some other health reason, or if they reported that they were uncertain of reproductive status, preferred not to say, or they left the reproductive question blank. After these exclusions, Study 1 contained 867 women and Study 2 contained 878 women with a valid conception risk score. (Supplementary Materials for Hawkins, Fitzgerald, & Nosek (2014) p. 1)
d <- d_raw %>%
filter(repstatus == "fertile",
currentcycle_day < 41,
currentcycle_day > 0,
cycle_length >= 20,
cycle_length <= 40,
pregnow == "no")…we created composite bias scores in each study by standardizing and then averaging all racial-bias measures (warmth toward African Americans was subtracted from warmth toward European Americans to create a warmth difference score prior to standardizing).(Hawkins, Fitzgerald & Nosek, 2014, p. 250)
d %<>%
mutate(zIAT = scale(IAT, center = TRUE, scale = TRUE), #calculate z scores for variables
zattitude = scale(attitude, center = TRUE, scale = TRUE),
zwarmth_black = scale(warmth_black, center = TRUE, scale = TRUE),
zwarmth_white = scale(warmth_white, center = TRUE, scale = TRUE),
zrisk = scale(risk, center = TRUE, scale = TRUE),
zwarmth_diff = scale(warmth_white - warmth_black, center = TRUE, scale = TRUE),
#calculate warmth difference score
zatt_composite = (zIAT + zattitude + zwarmth_diff)/3) #calculate composite score by averaging bias scoresCheck on descriptives for the filtered data. We at least have the same number of participants included.
nrow(d) == 867## [1] TRUEDescriptive statistics
The mean score for implicit racial bias was 0.34 (SD = 0.43) in Study 1 (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
ms_implicit_bias <- d%>%
filter(!is.na(IAT))%>%
summarise(mean = mean(IAT), sd = sd(IAT))
reportObject <- reproCheck(reportedValue = '.34',
obtainedValue = ms_implicit_bias$mean, valueType = 'mean')## [1] "MATCH for mean. The reported value (0.34) and the obtained value (0.34) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.43',
obtainedValue = ms_implicit_bias$sd, valueType = 'sd')## [1] "MATCH for sd. The reported value (0.43) and the obtained value (0.43) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."The mean score for explicit racial bias was 0.30 (SD = 1.09) in Study 1 (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
ms_explicit_bias <- d%>%
filter(!is.na(attitude))%>%
summarise(mean = mean(attitude), sd = sd(attitude))
reportObject <- reproCheck(reportedValue = '.30',
obtainedValue = ms_explicit_bias$mean, valueType = 'mean')## [1] "MATCH for mean. The reported value (0.3) and the obtained value (0.3) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '1.09',
obtainedValue = ms_explicit_bias$sd, valueType = 'sd')## [1] "MATCH for sd. The reported value (1.09) and the obtained value (1.09) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Mean rated warmth toward European Americans was 7.18 (SD = 2.07) in Study 1 (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
ms_warmth_white <- d %>%
filter(!is.na(warmth_white)) %>%
summarise(mean = mean(warmth_white), sd = sd(warmth_white))
reportObject <- reproCheck(reportedValue = '7.18', obtainedValue = ms_warmth_white$mean, valueType = 'mean')## [1] "MATCH for mean. The reported value (7.18) and the obtained value (7.18) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '2.07', obtainedValue = ms_warmth_white$sd, valueType = 'sd')## [1] "MATCH for sd. The reported value (2.07) and the obtained value (2.07) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."mean rated warmth toward African Americans was 6.69 (SD = 2.06) in Study 1 (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
ms_warmth_black <- d %>%
filter(!is.na(warmth_black))%>%
summarise(mean = mean(warmth_black), sd = sd(warmth_black))
reportObject <- reproCheck(reportedValue = '6.69', obtainedValue = ms_warmth_black$mean, valueType = 'mean')## [1] "MATCH for mean. The reported value (6.69) and the obtained value (6.69) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '2.06', obtainedValue = ms_warmth_black$sd, valueType = 'sd')## [1] "MATCH for sd. The reported value (2.06) and the obtained value (2.06) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Mean conception risk was 0.03 (SD = 0.03) in Study 1 (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
ms_risk <- d %>%
filter(!is.na(risk)) %>%
summarise(mean = mean(risk), sd = sd(risk))
reportObject <- reproCheck(reportedValue = '.03', obtainedValue = ms_risk$mean, valueType = 'mean')## [1] "MATCH for mean. The reported value (0.03) and the obtained value (0.03) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.03', obtainedValue = ms_risk$sd, valueType = 'sd')## [1] "MATCH for sd. The reported value (0.03) and the obtained value (0.03) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Inferential statistics
Correlations were computed between conception risk and each race variable, including the composite score. To best match the existing literature, we computed correlations for White women only…
Conception risk did not significantly correlate with any of the individual measures of racial bias (see Table S4 in the Supplemental Material) or with the composite score in Study 1, r = .00, 95% confidence interval (CI) = [−.09, .08] (Hawkins, Fitzgerald & Nosek, 2014, p. 250)
In the following section, we attempt to reproduce the correlations (with 95% confidence intervals) reported in Table S4 for Study 1.
Conception risk and Implicit racial bias
cortab <- d %>%
filter(race == "6") #filter for White women
risk_implicit_cor <- cor.test(cortab$zrisk, cortab$zIAT,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.03',
obtainedValue = unname(risk_implicit_cor$estimate),
valueType = 'r')## [1] "MAJOR_ERROR for r. The reported value (0.03) and the obtained value (0.05) differed by 66.67%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.05',
obtainedValue = unname(risk_implicit_cor$conf.int[1]),
valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (-0.05) and the obtained value (-0.04) differed by 20%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.12',
obtainedValue = unname(risk_implicit_cor$conf.int[2]),
valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.12) and the obtained value (0.13) differed by 8.33%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Conception risk and Explicit preference
risk_explicit_cor <- cor.test(cortab$zrisk, cortab$zattitude,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '-.01',
obtainedValue = unname(risk_explicit_cor$estimate),
valueType = 'r')## [1] "MAJOR_ERROR for r. The reported value (-0.01) and the obtained value (0) differed by 100%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.09',
obtainedValue = unname(risk_explicit_cor$conf.int[1]),
valueType = 'ci')## [1] "MATCH for ci. The reported value (-0.09) and the obtained value (-0.09) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.08',
obtainedValue = unname(risk_explicit_cor$conf.int[2]),
valueType = 'ci')## [1] "MATCH for ci. The reported value (0.08) and the obtained value (0.08) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Conception risk and Explicit warmth toward Blacks
risk_warmth_blacks_cor <- cor.test(cortab$zrisk, cortab$zwarmth_black,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '-.01',
obtainedValue = unname(risk_warmth_blacks_cor$estimate),
valueType = 'r')## [1] "MATCH for r. The reported value (-0.01) and the obtained value (-0.01) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.10',
obtainedValue = unname(risk_warmth_blacks_cor$conf.int[1]),
valueType = 'ci')## [1] "MATCH for ci. The reported value (-0.1) and the obtained value (-0.1) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.07',
obtainedValue = unname(risk_warmth_blacks_cor$conf.int[2]),
valueType = 'ci')## [1] "MATCH for ci. The reported value (0.07) and the obtained value (0.07) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Conception risk and Explicit warmth towards Whites
risk_warmth_whites_cor<- cor.test(cortab$zrisk, cortab$zwarmth_white,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '-.03',
obtainedValue = unname(risk_warmth_whites_cor$estimate),
valueType = 'r')## [1] "MAJOR_ERROR for r. The reported value (-0.03) and the obtained value (-0.02) differed by 33.33%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.11',
obtainedValue = unname(risk_warmth_whites_cor$conf.int[1]),
valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (-0.11) and the obtained value (-0.1) differed by 9.09%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.06',
obtainedValue = unname(risk_warmth_whites_cor$conf.int[2]),
valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (0.06) and the obtained value (0.07) differed by 16.67%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Conception risk and Race attitude composite
risk_atts_comp_cor <- cor.test(cortab$zrisk, cortab$zatt_composite,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.00', obtainedValue = unname(risk_atts_comp_cor$estimate), valueType = 'r')## [1] "MAJOR_ERROR for r. The reported value (0) and the obtained value (0.03) differed by Inf%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.09', obtainedValue = unname(risk_atts_comp_cor$conf.int[1]), valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (-0.09) and the obtained value (-0.05) differed by 44.44%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.08', obtainedValue = unname(risk_atts_comp_cor$conf.int[2]), valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (0.08) and the obtained value (0.12) differed by 50%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Implicit racial bias and Explicit Preference
implicit_explicit_cor <- cor.test(cortab$zIAT,cortab$zattitude,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.17', obtainedValue = unname(implicit_explicit_cor$estimate), valueType = 'r')## [1] "MATCH for r. The reported value (0.17) and the obtained value (0.17) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.08', obtainedValue = unname(implicit_explicit_cor$conf.int[1]), valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (0.08) and the obtained value (0.09) differed by 12.5%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.25', obtainedValue = unname(implicit_explicit_cor$conf.int[2]), valueType = 'ci')## [1] "MATCH for ci. The reported value (0.25) and the obtained value (0.25) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Implicit racial bias and Explicit Warmth towards Blacks
implicit_warmth_black_cor <- cor.test(cortab$zIAT, cortab$zwarmth_black,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '-.06', obtainedValue = unname(implicit_warmth_black_cor$estimate), valueType = 'r')## [1] "MATCH for r. The reported value (-0.06) and the obtained value (-0.06) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.15', obtainedValue = unname(implicit_warmth_black_cor$conf.int[1]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (-0.15) and the obtained value (-0.14) differed by 6.67%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.02', obtainedValue = unname(implicit_warmth_black_cor$conf.int[2]), valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (0.02) and the obtained value (0.03) differed by 50%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Implicit racial bias and Explicit Warmth towards Whites
implicit_warmth_white_cor <- cor.test(cortab$zIAT, cortab$zwarmth_white,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.08', obtainedValue = unname(implicit_warmth_white_cor$estimate), valueType = 'r')## [1] "MAJOR_ERROR for r. The reported value (0.08) and the obtained value (0.1) differed by 25%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.01', obtainedValue = unname(implicit_warmth_white_cor$conf.int[1]), valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (-0.01) and the obtained value (0.01) differed by 200%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.17', obtainedValue = unname(implicit_warmth_white_cor$conf.int[2]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.17) and the obtained value (0.18) differed by 5.88%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Implicit racial bias and Race attitude composite
implicit_attscomp_cor <- cor.test(cortab$zIAT, cortab$zatt_composite,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.66',
obtainedValue = unname(implicit_attscomp_cor$estimate),
valueType = 'r')## [1] "MINOR_ERROR for r. The reported value (0.66) and the obtained value (0.63) differed by 4.55%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.61',
obtainedValue = unname(implicit_attscomp_cor$conf.int[1]),
valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.61) and the obtained value (0.57) differed by 6.56%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.71',
obtainedValue = unname(implicit_attscomp_cor$conf.int[2]),
valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.71) and the obtained value (0.68) differed by 4.23%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Explicit Preference and Explicit Warmth towards Blacks
explicit_warmth_black_cor <- cor.test(cortab$zattitude, cortab$zwarmth_black,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '-.33',
obtainedValue = unname(explicit_warmth_black_cor$estimate),
valueType = 'r')## [1] "MATCH for r. The reported value (-0.33) and the obtained value (-0.33) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.41',
obtainedValue = unname(explicit_warmth_black_cor$conf.int[1]),
valueType = 'ci')## [1] "MATCH for ci. The reported value (-0.41) and the obtained value (-0.41) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.25',
obtainedValue = unname(explicit_warmth_black_cor$conf.int[2]),
valueType = 'ci')## [1] "MATCH for ci. The reported value (-0.25) and the obtained value (-0.25) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Explicit Preference and Explicit Warmth towards Whites
explicit_warmth_white_cor <- cor.test(cortab$zattitude, cortab$zwarmth_white,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.25', obtainedValue = unname(explicit_warmth_white_cor$estimate), valueType = 'r')## [1] "MINOR_ERROR for r. The reported value (0.25) and the obtained value (0.26) differed by 4%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.17', obtainedValue = unname(explicit_warmth_white_cor$conf.int[1]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.17) and the obtained value (0.18) differed by 5.88%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.33', obtainedValue = unname(explicit_warmth_white_cor$conf.int[2]), valueType = 'ci')## [1] "MATCH for ci. The reported value (0.33) and the obtained value (0.33) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Explicit Preference and Race attitude composite
explicit_attcomp_cor <- cor.test(cortab$zattitude, cortab$zatt_composite,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.81', obtainedValue = unname(explicit_attcomp_cor$estimate), valueType = 'r')## [1] "MINOR_ERROR for r. The reported value (0.81) and the obtained value (0.8) differed by 1.23%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.77', obtainedValue = unname(explicit_attcomp_cor$conf.int[1]), valueType = 'ci')## [1] "MATCH for ci. The reported value (0.77) and the obtained value (0.77) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.84', obtainedValue = unname(explicit_attcomp_cor$conf.int[2]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.84) and the obtained value (0.83) differed by 1.19%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Explicit Warmth toward Blacks and Explicit Warmth towards Whites
warmth_black_warmth_white_cor <- cor.test(cortab$zwarmth_black, cortab$zwarmth_white,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.65', obtainedValue = unname(warmth_black_warmth_white_cor$estimate), valueType = 'r')## [1] "MATCH for r. The reported value (0.65) and the obtained value (0.65) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.59', obtainedValue = unname(warmth_black_warmth_white_cor$conf.int[1]), valueType = 'ci')## [1] "MATCH for ci. The reported value (0.59) and the obtained value (0.59) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.70', obtainedValue = unname(warmth_black_warmth_white_cor$conf.int[2]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.7) and the obtained value (0.69) differed by 1.43%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Explicit Warmth toward Blacks and Race attitude composite
warmth_black_attscomp <- cor.test(cortab$zwarmth_black, cortab$zatt_composite,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '-.37', obtainedValue = unname(warmth_black_attscomp$estimate), valueType = 'r')## [1] "MINOR_ERROR for r. The reported value (-0.37) and the obtained value (-0.35) differed by 5.41%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.44', obtainedValue = unname(warmth_black_attscomp$conf.int[1]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (-0.44) and the obtained value (-0.42) differed by 4.55%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.29', obtainedValue = unname(warmth_black_attscomp$conf.int[2]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (-0.29) and the obtained value (-0.27) differed by 6.9%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Explicit Warmth toward Whites and Race attitude composite
warmth_white_attscomp <- cor.test(cortab$zwarmth_white, cortab$zatt_composite,
method = c("pearson"),
conf.level = 0.95)
reportObject <- reproCheck(reportedValue = '.32', obtainedValue = unname(warmth_white_attscomp$estimate), valueType = 'r')## [1] "MINOR_ERROR for r. The reported value (0.32) and the obtained value (0.34) differed by 6.25%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.24', obtainedValue = unname(warmth_white_attscomp$conf.int[1]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.24) and the obtained value (0.26) differed by 8.33%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.39', obtainedValue = unname(warmth_white_attscomp$conf.int[2]), valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (0.39) and the obtained value (0.41) differed by 5.13%. Note that the obtained value was rounded to 2 decimal places to match the reported value."Step 5: Conclusion
We found several small discrepencies between values reported in the text and our obtained values. The authors included a table of correlations with 95% confidence intervals between the different dependent measures (available in Supplemental Materials).
We noted several “major errors” in correlation coefficients (\(r\) values) and confidence intervals. All of these are very small in magnitude (all < .03 correlation units) but reach the 10% threshold. It is not clear to us what the source of the discrepency is, although it could be that different software packages (the authors used SAS and we are using R) give different values for these tests. (One possibility is minor numerical differences in how standardization of variables is performed).
We attempted to run the SAS program provided by the authors along with the article. However, the program is trying to read in the following text files: ‘explicit.txt’ ‘sessions.txt’, ‘sessionTasks.txt’, ‘iat.txt’, which, as far as we can tell, are not shared along with the article. We contacted the corresponding author on April 23, 2019, to request these files. We initially received a response from the corresponding author on Apr 23 (2019) which offered to help in principle but noted that this could take some time. We sent follow-up e-mails on April 25, May 22, and July 2, but received no further response. We ultimately decided to close the reproducibility check on August 24th.
Note that although we made contact with an author, no substantive assistance was provided - this is reflected in the categorisation of this case as “Failure without author assistance”.
None of the reproducibility issues we identified are likely to substantially change the authors’ original conclusions and we could successfully reproduce all reported descriptive statistics for the target outcomes.
Author_Assistance = FALSE # was author assistance provided? (if so, enter TRUE)
Insufficient_Information_Errors <- 0 # how many discrete insufficient information issues did you encounter?
# Assess the causal locus (discrete reproducibility issues) of any reproducibility errors. Note that there doesn't necessarily have to be a one-to-one correspondance between discrete reproducibility issues and reproducibility errors. For example, it could be that the original article neglects to mention that a Greenhouse-Geisser correct was applied to ANOVA outcomes. This might result in multiple reproducibility errors, but there is a single causal locus (discrete reproducibility issue).
locus_typo <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification <- 0 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data <- 0 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified <- 1 # how many discrete issues were there for which you could not identify the cause
# How many of the above issues were resolved through author assistance?
locus_typo_resolved <- NA # how many discrete issues did you encounter that related to typographical errors?
locus_specification_resolved <- NA # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis_resolved <- NA # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data_resolved <- NA # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified_resolved <- NA # how many discrete issues were there for which you could not identify the cause
Affects_Conclusion <- FALSE # Do any reproducibility issues encounter appear to affect the conclusions made in the original article? TRUE, FALSE, or NA. This is a subjective judgement, but you should taking into account multiple factors, such as the presence/absence of decision errors, the number of target outcomes that could not be reproduced, the type of outcomes that could or could not be reproduced, the difference in magnitude of effect sizes, and the predictions of the specific hypothesis under scrutiny.reportObject <- reportObject %>%
filter(dummyRow == FALSE) %>% # remove the dummy row
select(-dummyRow) %>% # remove dummy row designation
mutate(articleID = articleID) %>% # add variables to report
select(articleID, everything()) # make articleID first column
# decide on final outcome
if(any(!(reportObject$comparisonOutcome %in% c("MATCH", "MINOR_ERROR"))) | Insufficient_Information_Errors > 0){
finalOutcome <- "Failure without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Failure despite author assistance"
}
}else{
finalOutcome <- "Success without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Success with author assistance"
}
}
# collate report extra details
reportExtras <- data.frame(articleID, pilotNames, copilotNames, pilotTTC, copilotTTC, pilotStartDate, copilotStartDate, completionDate, Author_Assistance, finalOutcome, Insufficient_Information_Errors, locus_typo, locus_specification, locus_analysis, locus_data, locus_unidentified, locus_typo_resolved, locus_specification_resolved, locus_analysis_resolved, locus_data_resolved, locus_unidentified_resolved)
# save report objects
if(reportType == "pilot"){
write_csv(reportObject, "pilotReportDetailed.csv")
write_csv(reportExtras, "pilotReportExtras.csv")
}
if(reportType == "final"){
write_csv(reportObject, "finalReportDetailed.csv")
write_csv(reportExtras, "finalReportExtras.csv")
}Session information
devtools::session_info()## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.0.0 (2020-04-24)
## os macOS Catalina 10.15.4
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz Europe/London
## date 2020-05-19
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date lib
## assertthat 0.2.1 2019-03-21 [1]
## backports 1.1.6 2020-04-05 [1]
## broom 0.5.6 2020-04-20 [1]
## callr 3.4.3 2020-03-28 [1]
## cellranger 1.1.0 2016-07-27 [1]
## cli 2.0.2 2020-02-28 [1]
## colorspace 1.4-1 2019-03-18 [1]
## crayon 1.3.4 2017-09-16 [1]
## DBI 1.1.0 2019-12-15 [1]
## dbplyr 1.4.3 2020-04-19 [1]
## desc 1.2.0 2018-05-01 [1]
## devtools 2.3.0 2020-04-10 [1]
## digest 0.6.25 2020-02-23 [1]
## dplyr * 0.8.5 2020-03-07 [1]
## ellipsis 0.3.0 2019-09-20 [1]
## evaluate 0.14 2019-05-28 [1]
## fansi 0.4.1 2020-01-08 [1]
## forcats * 0.5.0 2020-03-01 [1]
## fs 1.4.1 2020-04-04 [1]
## generics 0.0.2 2018-11-29 [1]
## ggplot2 * 3.3.0 2020-03-05 [1]
## glue 1.4.0 2020-04-03 [1]
## gtable 0.3.0 2019-03-25 [1]
## haven * 2.2.0 2019-11-08 [1]
## hms 0.5.3 2020-01-08 [1]
## htmltools 0.4.0 2019-10-04 [1]
## httr 1.4.1 2019-08-05 [1]
## jsonlite 1.6.1 2020-02-02 [1]
## knitr * 1.28 2020-02-06 [1]
## lattice 0.20-41 2020-04-02 [1]
## lifecycle 0.2.0 2020-03-06 [1]
## lubridate 1.7.8 2020-04-06 [1]
## magrittr * 1.5 2014-11-22 [1]
## memoise 1.1.0 2017-04-21 [1]
## modelr 0.1.7 2020-04-30 [1]
## munsell 0.5.0 2018-06-12 [1]
## nlme 3.1-147 2020-04-13 [1]
## pillar 1.4.4 2020-05-05 [1]
## pkgbuild 1.0.7 2020-04-25 [1]
## pkgconfig 2.0.3 2019-09-22 [1]
## pkgload 1.0.2 2018-10-29 [1]
## prettyunits 1.1.1 2020-01-24 [1]
## processx 3.4.2 2020-02-09 [1]
## ps 1.3.2 2020-02-13 [1]
## purrr * 0.3.4 2020-04-17 [1]
## R6 2.4.1 2019-11-12 [1]
## Rcpp 1.0.4.6 2020-04-09 [1]
## readr * 1.3.1 2018-12-21 [1]
## readxl * 1.3.1 2019-03-13 [1]
## remotes 2.1.1 2020-02-15 [1]
## reprex 0.3.0 2019-05-16 [1]
## ReproReports * 0.1 2020-05-06 [1]
## rlang 0.4.6 2020-05-02 [1]
## rmarkdown 2.1 2020-01-20 [1]
## rprojroot 1.3-2 2018-01-03 [1]
## rstudioapi 0.11 2020-02-07 [1]
## rvest 0.3.5 2019-11-08 [1]
## scales 1.1.0 2019-11-18 [1]
## sessioninfo 1.1.1 2018-11-05 [1]
## stringi 1.4.6 2020-02-17 [1]
## stringr * 1.4.0 2019-02-10 [1]
## testthat 2.3.2 2020-03-02 [1]
## tibble * 3.0.1 2020-04-20 [1]
## tidyr * 1.0.2 2020-01-24 [1]
## tidyselect 1.0.0 2020-01-27 [1]
## tidyverse * 1.3.0 2019-11-21 [1]
## usethis 1.6.1 2020-04-29 [1]
## vctrs 0.2.4 2020-03-10 [1]
## withr 2.2.0 2020-04-20 [1]
## xfun 0.13 2020-04-13 [1]
## xml2 1.3.2 2020-04-23 [1]
## yaml 2.2.1 2020-02-01 [1]
## source
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## Github (METRICS-CARPS/CARPSreports@3277f85)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
##
## [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library