1- pbinom(0.6*144, 144, 0.5)[1] 0.007685257Replication of Does Seeing Faces of Young Black Boys Facilitate the Identification of Threatening Stimuli? by Todd et al. (2016, Psychological Science)
Introduction
After reviewing the possible rescue projects available to students, a decision was made to attempt to rescue the replication project from Todd et al. (2016) for mainly two reasons. First, the topic is highly relevant and has strong implications. The paper deals with anti-Black racism toward young boys, and the finding that the perception of young Black faces facilitates the recognition of certain objects or words says a lot about how stereotyping and racism function at a perceptive level. Second, the experimental paradigm itself seemed straightforward and easy to reproduce, but other aspects of the experiment (e.g., sample size, data analysis) seem not to have been done in the best way.
The present project aims to rescue the replication of Experiment 1, which used photos of Black and White boys as primes for the categorization of objects (gun vs. toy). Specifically, the rescue will focus on attempting to replicate the results concerning reaction times and error rates. For that, we will need access to the stimuli utilized in the experiment (i.e., images of faces of boys and a set of images of objects to be categorized). In addition, the experiment will have to be programmed in the appropriate form (e.g., PsychoPy .
Challenges involve recreating the experimental paradigm in line with the original and analyzing the data using the same statistical techniques. Adding other forms to guarantee the high quality of the experiment (e.g., exclusion criteria) might also be challenging.
Summary of prior replication attempt
The prior replication attempt had a substantially smaller sample size (effective N=26) than the original study (N=63). This decision was based on a a priori power analysis indicating a sample size of N = 24 for 95% power. Additionally, the original study was conducted in an in-person lab setting, while the replication attempt was conducted online using Mechanical Turk. All other relevant criteria seem to be equal or comparable to the original study.
Methods
“White participants completed sequential priming tasks in which they categorized threatening and non-threatening objects (toys and guns) and words after brief presentations of Black and White children’s faces.”
The present rescue attempt will keep the same methodological paradigm and recode the original experiment in PsychoPy. Additionally, this rescue attempt will collect data online using Prolific as opposed to Mechanical Turk.
Power Analysis
Original ηp2 = .22 which translates into an effect size of 0.53. Power analysis shows that I need 16 participants for 80% power, 20 for 90%, and 24 for 95% power. This effect size is for the 2 (race of prime: Black, White) × 2 (target object: gun, toy) repeated measures analysis of variance (ANOVA). Attaining this number of participants is easily feasible with Prolific.
Considering the initial replication attempt failed to find an effect, I will propose a sample closer to the original sample of the study, aiming for effective N = 60. According to G*Power, a sample of N = 60 would have a power of near 1 to find the original effect size.
Planned Sample
We will recruit 66 participants to be close to the N of the original study. We added 6, which is 10% of 60 to ensure we will still have enough participants after the exclusion criteria. We will exclude non-White participants in the analysis. We will also exclude participants who have less than 60% level of accuracy. The original study excluded based on 50% accuracy or lower (chance-level), meaning that a participant who guesses every single trial might still have a reasonable chance of not being excluded. The chance of a participant reaching 60% accuracy over 144 trials is:
Materials
“The primes were 12 photos of boys (6 Black, 6 White) taken from the Child Affective Facial Expression set (LoBue & Thrasher, 2015). We selected these photos using the following criteria: The faces had to be easily categorized by race, to have a neutral expression, to have no idiosyncrasies (e.g., facial scars), and to be similar in actual age (mean age for Black faces = 4.98 years; mean age for White faces = 5.01 years; p > .250). Each photo was cropped so that it included only the head and was standardized in size. The target objects were 6 gun images taken from Payne (2001) and 6 toy images (e.g., a rattle) taken from online sources. The toy images were converted to grayscale and sized to match the gun images.”
Procedure
Instead of beginning with a blank screen (500 ms) like in the original article, we began with a screen that instructed participants to “Press the spacebar to continue” after each trial. The rest of the procedures were the same. After feedback from the original author, we edited visual features of the experimental paradigm to the best of our ability to match the original experiment.
“Each trial sequence began [..] followed by a face prime (200 ms), then a target object (200 ms), and finally a pattern mask (which remained on screen until participants responded). If participants did not respond within 500 ms, a message (“Please respond faster!”) appeared for 1 s. Each of the 12 face primes was paired once with each of the 12 target objects, which resulted in 144 randomly ordered experimental trials. Eight practice trials preceded the experimental trials.”
This replication included 12 practice trials instead of 8.
Additionally, following the recommendation in a commentary in the previous replication attempt report, we will include a cover page for the study, as recommended by the author: “This task is investigating object recognition under distracting conditions. You will be asked to quickly identify objects will being distracted by other stimuli.”
Controls
The experiment already contains built-in attention checks as it alerts the participant to pay attention if they are too slow to react to the image. Additionally, the rescue replication will exclude participants with less than 50% accuracy rate, virtually eliminating participants who do not complete the experiment in good faith. We will also exclude non-white participants, as did the original study and the 1st replication attempt.
Analysis Plan
“Before analysis, we excluded errors and RTs less than 100 ms. RTs exceeding the 500-ms deadline were excluded from analyses, and responses on those trials were treated as errors. We then subjected the remaining RTs to a log transformation (Payne, 2001); however, for interpretive ease, we report raw RTs. A 2 (race of prime: Black, White) × 2 (target object: gun, toy) repeated measures analysis of variance (ANOVA) revealed a significant interaction.” The key test will be the interaction effect of this 2 X 2 ANOVA.
We will only analyze data from people who indicated their race as White.
Differences from Original Study and 1st replication
The only difference between the original study and the replication attempt in terms of procedure is that the replication attempt was made online and the original experiment was conducted in-person.
The present rescue attempt is going to try to get closer to the original study by aiming at a sample size closer to the original. Additionally, this rescue will use Prolific as opposed to MTurk. Since evidence has shown that Profilic consistently delivers higher quality data, the differences to an in-person setting could be further reduced. We will also raise the chance-exclusion cut-off to 60%.
No difference in claims is expected to occur from the original study.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Results
Data preparation
Data was be imported and wrangled into a tidy format, renaming variables for better interpretability. Categorical variables were factored and relabeled.
Results of control measures
X people were excluded based on not meeting screener criteria (e.g., identifying as White). An additional X trials were excluded for being too slow (>500 ms) or too quick (<100 ms).
Confirmatory analysis
A repeated measures ANOVA will be computed to probe for the significance of the Race of Prime X Object Classification interaction. RT logs will be used as per the original study and replication attempt. Box plots were generated for visualization. Additionally, bar plots with with 95% error bars were computed using ggplot2.
# Checking accuracy
acc_dat <- processed_dat %>%
group_by(ID, StimRace, StimObj) %>%
summarise(mean_accuracy = mean(Correct)) #%>% `summarise()` has grouped output by 'ID', 'StimRace'. You can override using
the `.groups` argument. #filter(mean_accuracy > 0.599)
acc_dat # One prolifi pilot participant was clearly just pressing one key throughout the whole thing! LOL# A tibble: 12 × 4
# Groups: ID, StimRace [6]
ID StimRace StimObj mean_accuracy
<chr> <chr> <chr> <dbl>
1 61033208fb1f00f2b7780527 Black Gun 0.829
2 61033208fb1f00f2b7780527 Black Toy 0.8
3 61033208fb1f00f2b7780527 White Gun 0.543
4 61033208fb1f00f2b7780527 White Toy 0.914
5 636ebfc7b2ea0004549eb06b Black Gun 0.970
6 636ebfc7b2ea0004549eb06b Black Toy 0
7 636ebfc7b2ea0004549eb06b White Gun 1
8 636ebfc7b2ea0004549eb06b White Toy 0.0588
9 655120cb1d74c5f776476489 Black Gun 0.824
10 655120cb1d74c5f776476489 Black Toy 0.743
11 655120cb1d74c5f776476489 White Gun 0.917
12 655120cb1d74c5f776476489 White Toy 0.857 # Turning into factors for ANOVA
processed_dat$ID <- as_factor(processed_dat$ID)
processed_dat$StimRace <- as_factor(processed_dat$StimRace)
processed_dat$StimObj <- as_factor(processed_dat$StimObj)
#Looking at summarized data
sum_dat <- processed_dat %>%
# FILTER BY ACCURACY LATER
group_by(StimRace, StimObj, ID) %>%
get_summary_stats(LogRT, type = "mean_sd")
sum_dat# A tibble: 12 × 7
ID StimRace StimObj variable n mean sd
<fct> <fct> <fct> <fct> <dbl> <dbl> <dbl>
1 61033208fb1f00f2b7780527 White Gun LogRT 35 -1.86 0.316
2 636ebfc7b2ea0004549eb06b White Gun LogRT 34 -2.65 0.962
3 655120cb1d74c5f776476489 White Gun LogRT 36 -1.73 0.34
4 61033208fb1f00f2b7780527 White Toy LogRT 35 -1.78 0.527
5 636ebfc7b2ea0004549eb06b White Toy LogRT 34 -2.66 1.03
6 655120cb1d74c5f776476489 White Toy LogRT 35 -1.69 0.388
7 61033208fb1f00f2b7780527 Black Gun LogRT 35 -1.86 0.427
8 636ebfc7b2ea0004549eb06b Black Gun LogRT 33 -2.71 0.937
9 655120cb1d74c5f776476489 Black Gun LogRT 34 -1.70 0.394
10 61033208fb1f00f2b7780527 Black Toy LogRT 35 -1.77 0.524
11 636ebfc7b2ea0004549eb06b Black Toy LogRT 32 -2.87 1.42
12 655120cb1d74c5f776476489 Black Toy LogRT 35 -1.75 0.99 Plots
# Boxplot for visualization
bxp <- processed_dat %>%
ggboxplot( x = "StimObj", y = "ReactionTime",
color = "StimRace", palette = "jco"
)
bxp
#Barplot with CI bars
sum_dat_plot <- processed_dat %>%
group_by(StimObj, StimRace) %>%
summarise(
n=n(),
mean=mean(ReactionTime),
sd=sd(ReactionTime)
) %>%
mutate( se=sd/sqrt(n)) %>%
mutate( ci=se * qt((1-0.05)/2 + .5, n-1))`summarise()` has grouped output by 'StimObj'. You can override using the
`.groups` argument.barplot <- sum_dat_plot %>%
ggplot(aes(x=StimObj, y=mean, fill=StimRace)) +
geom_bar(stat = "identity", position=position_dodge()) +
geom_errorbar(aes(ymin=mean-ci, ymax=mean+ci), width=.2,
position=position_dodge(.9)) +
ggthemes::theme_few()
barplot + scale_fill_brewer(palette="Paired")
# running 2x2 repeated measures ANOVA
aov_model <- aov( data = processed_dat,
LogRT ~ StimObj*StimRace + Error(ID/(StimObj*StimRace)))
summary(aov_model)
Error: ID
Df Sum Sq Mean Sq
StimObj 1 81.66 81.66
StimRace 1 0.72 0.72
Error: ID:StimObj
Df Sum Sq Mean Sq
StimObj 1 0.0001 0.0001
StimRace 1 0.0522 0.0522
StimObj:StimRace 1 0.4190 0.4190
Error: ID:StimRace
Df Sum Sq Mean Sq F value Pr(>F)
StimRace 1 0.2302 0.2302 16.71 0.153
StimObj:StimRace 1 0.3878 0.3878 28.14 0.119
Residuals 1 0.0138 0.0138
Error: ID:StimObj:StimRace
Df Sum Sq Mean Sq F value Pr(>F)
StimObj:StimRace 1 0.1678 0.16776 3.281 0.212
Residuals 2 0.1023 0.05114
Error: Within
Df Sum Sq Mean Sq F value Pr(>F)
Residuals 401 232.5 0.5798 ### ALTERNATIVE
# Perform the repeated measures ANOVA
anova_result <- ezANOVA(data = processed_dat,
dv = .(LogRT),
wid = .(ID),
within = .(StimRace, StimObj),
detailed = TRUE)Warning: Collapsing data to cell means. *IF* the requested effects are a subset
of the full design, you must use the "within_full" argument, else results may
be inaccurate.print(anova_result)$ANOVA
Effect DFn DFd SSn SSd F p
1 (Intercept) 1 2 5.221444e+01 2.447288353 4.267126e+01 0.02264209
2 StimRace 1 2 7.180445e-03 0.011978399 1.198899e+00 0.38780324
3 StimObj 1 2 5.321055e-06 0.014283757 7.450498e-04 0.98070269
4 StimRace:StimObj 1 2 5.022467e-03 0.002985155 3.364963e+00 0.20803358
p<.05 ges
1 * 9.547177e-01
2 2.891009e-03
3 2.148583e-06
4 2.023917e-03Three-panel graph with original, 1st replication, and your replication is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
# Consider this model
lmer( data = processed_dat,
LogRT ~ StimObj*StimRace + (1|ID))Linear mixed model fit by REML ['lmerMod']
Formula: LogRT ~ StimObj * StimRace + (1 | ID)
Data: processed_dat
REML criterion at convergence: 960.5307
Random effects:
Groups Name Std.Dev.
ID (Intercept) 0.5482
Residual 0.7574
Number of obs: 413, groups: ID, 3
Fixed Effects:
(Intercept) StimObjToy StimRaceBlack
-2.081990 0.040200 -0.007664
StimObjToy:StimRaceBlack
-0.078729 Discussion
Mini meta-analysis
Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.