Replication of Study 3a by Craig & Richeson (2014, Psychological Science)

Introduction

GitHub Repository: https://github.com/psych251/craig2014_rescue/tree/main

Craig & Richeson (2014) PDF: https://github.com/psych251/craig2014_rescue/blob/main/original_paper/craig-richeson-2014-on-the-precipice-of-a-majority-minority-america-perceived-status-threat-from-the-racial-demographic.pdf

I am planning to replicate Study 3a from the Craig & Richeson (2014) paper examining the impact of racial status threat on White Americans’ political ideologies. This is a topic directly related to my research interests in that I am interested broadly in looking at ways to improve understanding of systemic roots of racism and subsequently leveraging those understandings (1) to improve political support for racially equitable social policy and (2) to develop tools to foster a greater sense of agency for people who identify with marginalized identities (e.g., people of color, people with disabilities). Therefore, this project presents an opportunity to delve deeper into the underlying psychological processes which might explain why status threat might lead people to resist embracing equity-minded policy.

In this replication of Study 3a, I will seek to maintain many of the same stimuli and procedures utilized in the original paper. Specifically, following a pre-test of participants’ political ideologies, I will randomly assign participants to one of three conditions – either a control condition, a status-threat condition, or a assuaged threat condition. The stimuli (simulated news articles) will also remain the same across the three condition groups. However, if possible, I would like to pilot/pre-test these stimuli to ensure that (1) they are sufficiently different and that (2) the manipulation is clear and that status threat as a manipulation is operationalized correctly.

Summary of prior replication attempt

Differences in methods

Methodological differences between the original study and the replication are minor. First, the replication of this study doesn’t specify the materials used, particularly in terms of the news article (or “press release”), but I am assuming that the replication author used the original materials. Second, the replication study pared down some of the demographic materials in an effort to avoid a priming effect. Third, the replication collected gender demographic information, while the original study captured self-reported sex. Lastly, the replicating author added an item to flag participants who may have already read the manipulation article or who may otherwise be familiar with the goals of the study.

Based on the prior write-up, describe any differences between the original and 1st replication in terms of methods, sample, sample size, and analysis. Note any potential problems such as exclusion rates, noisy data, or issues with analysis.

Methods

Power Analysis

Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.

How much power does your planned sample have for original effect? For an attenuated effect that is half the size of the original?

(If power analysis is not possible or precise, discuss more fully how you determined a sample size that would be sufficient for rescue.)

Planned Sample

Mirroring the original paper, I plan to recruit one sample of 180 White U.S. citizens from an online survey platform (Prolific). Note: this number may change depending on subsequent power analyses. Further, while the original sample was comprised predominantly of male participants (160 men: 20 women), I hope to reach a more representative sample. Once I have reached my target n-size (as determined by a power analysis), I will close the survey.

Materials

I will follow the materials exactly where available for this experiment. This includes the two-item baseline ideology measure, the condition materials (i.e., the news articles, which are linked here: https://osf.io/vdg46), and the eight-item policy preferences list to assess participants’ policy preferences.

Procedure

The procedure for this replication will be followed precisely. As such, participants will, upon providing informed consent, report “their demographic characteristics, including political ideology”. Participants will then be randomly assigned “to read an article about the growing eographic mobility of the United States (control condition), an article about the projected majority-minority shift in the United States (status-threat condition), or the latter article with an extra paragraph designed to reduce status threat (assuaged-threat condition).” Finally, participants will indicate their policy preferences on an eight-item list of policies (Craig & Richeson, 2014).

Controls

As in the original study, I will conduct an ANOVA to confirm “that participants in the status-threat condition expressed greater perceived group-status threat than did participants in the assuaged-threat or control conditions” (Craig & Richeson, 2014). Furthermore, I will also add an attention check item in the policy preferences scale asking participants to “select ‘strongly support’”.

Analysis Plan

Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.

Differences from Original Study and 1st replication

Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

First, I plan to clean the Qualtrics datafile of any identifiable or metadata columns. I will focus my dataset on the participants’ self-reported demographic and political ideology variables, a variable denoting each participants’ randomly assigned condition, and the columns for each individual policy preference item. Participants who incorrectly respond to the attention-check item will be excluded from analysis, as will be participants who did not finish the entire activity.

Results of control measures

As in the original study, I will conduct an ANOVA to confirm “that participants in the status-threat condition expressed greater perceived group-status threat than did participants in the assuaged-threat or control conditions” (Craig & Richeson, 2014). Furthermore, I will also add an attention check item in the policy preferences scale asking participants to “select ‘strongly support’”. I will also to check to ensure that all participants correctly answered the attention check item.

Confirmatory analysis

In analysis, I plan to replicate the Craig & Richeson (2014) original analysis strategy: “Unless otherwise noted, reported analyses for Studies 3a and 3b are ANCOVAs with experimental condition as the independent variable and demographic characteristics and baseline political ideology as covariates.” I plan to replicate the ANCOVA analysis to examine the reported role of group status threat (the condition assignment) on policy preferences for White Americans, incorporating participants’ baseline political ideologies into the model. I think I could potentially use a regression to model the relationship between group status threat (condition manipulation) and the policy preferences, but I’m not sure why to prefer one over the other.

Three-panel graph with original, 1st replication, and your replication is ideal here

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Mini meta analysis

Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.