Replication of Does Seeing Faces of Young Black Boys Facilitate the Identification of Threatening Stimuli? by Todd et al. (2016, Psychological Science)
Introduction
After reviewing the possible rescue projects available to students, a decision was made to attempt to rescue the replication project from Todd et al. (2016) for mainly two reasons. First, the topic is highly relevant and has strong implications. The paper deals with anti-Black racism toward young boys, and the finding that the perception of young Black faces facilitates the recognition of certain objects or words says a lot about how stereotyping and racism function at a perceptive level. Second, the experimental paradigm itself seemed straightforward and easy to reproduce, but other aspects of the experiment (e.g., sample size, data analysis) seem not to have been done in the best way.
The present project aims to rescue the replication of Experiment 1, which used photos of Black and White boys as primes for the categorization of objects (gun vs. toy). Specifically, the rescue will focus on attempting to replicate the results concerning reaction times and error rates. For that, we will need access to the stimuli utilized in the experiment (i.e., images of faces of boys and a set of images of objects to be categorized). In addition, the experiment will have to be programmed in the appropriate form (e.g., PsychoPy or Qualtrics).
Challenges involve recreating the experimental paradigm in line with the original and analyzing the data using the same statistical techniques. Adding other forms to guarantee the high quality of the experiment (e.g., attention checks, control) might also be challenging.
Summary of prior replication attempt
The prior replication attempt had a substantially smaller sample size (effective N=26) than the original study (N=63). This decision was based on a a priori power analysis indicating a sample size of N = 24 for 95% power. Additionally, the original study was conducted in an in-person lab setting, while the replication attempt was conducted online using Mechanical Turk. All other relevant criteria seem to be equal or comparable to the original study.
Methods
“White participants completed sequential priming tasks in which they categorized threatening and non-threatening objects (toys and guns) and words after brief presentations of Black and White children’s faces.”
The present rescue attempt will keep the same methodological paradigm and recycle the JavaScript code utilized by the first replication attempt. Additionally, this rescue attempt will collect data online using Prolific as opposed to Mechanical Turk.
Power Analysis
Original ηp2 = .22 which translates into an effect size of 0.53. Power analysis shows that I need 16 participants for 80% power, 20 for 90%, and 24 for 95% power. This effect size is for the 2 (race of prime: Black, White) × 2 (target object: gun, toy) repeated measures analysis of variance (ANOVA). Attaining this number of participants is easily feasible with MTurk.
Considering the initial replication attempt failed to find an effect, I will propose a sample closer to the original sample of the study, aiming for effective N = 60. According to GPower, a sample of N = 60 would have a power of near 1 to find the original effect size.
Planned Sample
We will recruit 66 participants to be close to the N of the original study. We added 6, which is 10% of 60 to ensure we will still have enough participants after the exclusion criteria. We will exclude non-White participants in the analysis. We will also exclude participants who have less than 50% level of accuracy, as the original study did.
Materials
“The primes were 12 photos of boys (6 Black, 6 White) taken from the Child Affective Facial Expression set (LoBue & Thrasher, 2015). We selected these photos using the following criteria: The faces had to be easily categorized by race, to have a neutral expression, to have no idiosyncrasies (e.g., facial scars), and to be similar in actual age (mean age for Black faces = 4.98 years; mean age for White faces = 5.01 years; p > .250). Each photo was cropped so that it included only the head and was standardized in size. The target objects were 6 gun images taken from Payne (2001) and 6 toy images (e.g., a rattle) taken from online sources. The toy images were converted to grayscale and sized to match the gun images.”
Procedure
Instead of beginning with a blank screen (500 ms) like in the original article, we began with a screen that instructed participants to “Press the spacebar to continue” after each trial. The rest of the procedures were the same.
“Each trial sequence began [..] followed by a face prime (200 ms), then a target object (200 ms), and finally a pattern mask (which remained on screen until participants responded). If participants did not respond within 500 ms, a message (“Please respond faster!”) appeared for 1 s. Each of the 12 face primes was paired once with each of the 12 target objects, which resulted in 144 randomly ordered experimental trials. Eight practice trials preceded the experimental trials.”
Additionally, following the recommendation in a commentary in the previous replication attempt report, we will include a cover page for the study, as recommended by the author: “This task is investigating object recognition under distracting conditions. You will be asked to quickly identify objects will being distracted by other stimuli.”
Controls
The experiment already contains built-in attention checks as it alerts the participant to pay attention if they are too slow to react to the image. Additionally, the study protocol requires that those with less than 50% accuracy rate be excluded. We will also exclude non-white participants as the original study and the replication attempt only had white participants.
Analysis Plan
“We decided a priori to exclude data from participants with below-chance accuracy (i.e., errors on > 50% of trials) on the weapon identification task.”
“Before analysis, we excluded errors and RTs less than 100 ms. RTs exceeding the 500-ms deadline were excluded from analyses, and responses on those trials were treated as errors. We then subjected the remaining RTs to a log transformation (Payne, 2001); however, for interpretive ease, we report raw RTs. A 2 (race of prime: Black, White) × 2 (target object: gun, toy) repeated measures analysis of variance (ANOVA) revealed a significant interaction.” The key test will be the interaction effect of this 2 X 2 ANOVA.
We will only analyze data from people who indicated their race as White.
Differences from Original Study and 1st replication
The only difference between the original study and the replication attempt in terms of procedure is that the replication attempt was made online and the original experiment was conducted in-person.
The present rescue attempt is going to try to get closer to the original study by aiming at a sample size closer to the original. Additionally, this rescue will use Prolific as opposed to MTurk. Since evidence has shown that Profilic consistently delivers higher quality data, the differences to an in-person setting could be further reduced.
No difference in claims is expected to occur from the original study.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
None
Results
Data preparation
Data was be imported and wrangled into a tidy format, renaming variables for better interpretability. Categorical variables were factored and relabeled.
Results of control measures
X people were excluded based on not meeting screener criteria (e.g., identifying as White). An additional X trials were excluded for being too slow (>500 ms) or too quick (<100 ms).
Confirmatory analysis
The mean reaction time for each condition was calculated for each participant, then the mean reaction time for each condition across all participants was calculated. A repeated measures ANOVA will be computed to probe for the significance of the Race of Prime X Object Classification interaction. RT logs will be used as per the original study and replication attempt. Bar plots with 95% error bars were computed using ggplot2.
Three-panel graph with original, 1st replication, and your replication is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Mini meta-analysis
Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.