I am planning to replicate Study 3a from the Craig & Richeson (2014) paper examining the impact of racial status threat on White Americans’ political ideologies. This is a topic directly related to my research interests in that I am interested broadly in looking at ways to improve understanding of systemic roots of racism and subsequently leveraging those understandings (1) to improve political support for racially equitable social policy and (2) to develop tools to foster a greater sense of agency for people who identify with marginalized identities (e.g., people of color, people with disabilities). Therefore, this project presents an opportunity to delve deeper into the underlying psychological processes which might explain why status threat might lead people to resist embracing equity-minded policy.
In this replication of Study 3a, I will seek to maintain many of the same stimuli and procedures utilized in the original paper. Specifically, following a pre-test of participants’ political ideologies, I will randomly assign participants to one of three conditions – either a control condition, a status-threat condition, or a assuaged threat condition. The stimuli (simulated news articles) will also remain the same across the three condition groups. However, if possible, I would like to pilot/pre-test these stimuli to ensure that (1) they are sufficiently different and that (2) the manipulation is clear and that status threat as a manipulation is operationalized correctly.
Summary of prior replication attempt
Differences in methods
Methodological differences between the original study and the replication are minor. First, the replication of this study doesn’t specify the materials used, particularly in terms of the news article (or “press release”), but I am assuming that the replication author used the original materials. Second, the replication study pared down some of the demographic materials in an effort to avoid a priming effect. Third, the replication collected gender demographic information, while the original study captured self-reported sex. Lastly, the replicating author added an item to flag participants who may have already read the manipulation article or who may otherwise be familiar with the goals of the study.
Based on the prior write-up, describe any differences between the original and 1st replication in terms of methods, sample, sample size, and analysis. Note any potential problems such as exclusion rates, noisy data, or issues with analysis.
Methods
Power Analysis
Using the pwr.t.test() function within the “pwr” R package, I calculated that I need 85 participants/condition to achieve 80% power. Rounding up to 90/condition, I am planning to sample 180 participants.
Planned Sample
Mirroring the original paper, I plan to recruit one sample of 180 White U.S. citizens from an online survey platform (Prolific). Further, while the original sample was comprised predominantly of male participants (160 men: 20 women), I hope to reach a more representative sample. Once I have reached my target n-size (as determined by a power analysis), I will close the survey. While this sample will not provide a 2.5x replication of the original study, it is significantly larger than the first replication (n=111).
Materials
I will follow the materials exactly where available for this experiment. This includes the two-item baseline ideology measure, the condition materials (i.e., the news articles, which are linked here: https://osf.io/vdg46), and the eight-item policy preferences list to assess participants’ policy preferences.
Procedure
The procedure for this replication will be followed precisely. As such, participants will, upon providing informed consent, report “their demographic characteristics, including political ideology”. Participants will then be randomly assigned “to read an article about the growing eographic mobility of the United States (control condition), an article about the projected majority-minority shift in the United States (status-threat condition), or the latter article with an extra paragraph designed to reduce status threat (assuaged-threat condition).” Finally, participants will indicate their policy preferences on an eight-item list of policies (Craig & Richeson, 2014).
Controls
As in the original study, I will conduct an ANOVA to confirm “that participants in the status-threat condition expressed greater perceived group-status threat than did participants in the assuaged-threat or control conditions” (Craig & Richeson, 2014). Furthermore, I will also add an attention check item in the policy preferences scale asking participants to “select ‘strongly support’”.
Analysis Plan
My key analysis of interest will be a two condition t-test. This differs from the original study and the first replication, in that both earlier iterations used ANOVAs to compare the three (assuaged threat, status threat, control) conditions, while I only have 2 conditions in my replication. More specifically, my analysis is contrasting the difference in “conservative policy score” or the standardized total score of individuals’ support of different conservative policies between the “assuaged threat” and the “status threat” conditions. In the original study (Craig & Richeson, 2014), this difference was statistically significant (F(1, 161) = 7.63, p = .006, ηp2 = .05). In the first replication, the comparison between the assuaged threat and status threat conditions was not reported, but the overall ANOVA analyzing the effect of condition on policy endorsement was not significant (F(2, 111) = 0.151, p = .86).
Differences from Original Study and 1st replication
Differences in sample: The original study sampled 170 participants across three conditions, with 121 participants combined in the two conditions of interest (62 in status-threat condition; 59 in assuaged-threat condition). The first replication sampled 114 participants across all three conditions (condition breakdown was not reported). In comparison, this replication will sample 180 participants total, only across the status-threat and the assuaged-threat conditions (90 in each condition).
Differences in materials: There are several minor changes in the materials of the study, which are largely the same. One change was to the article manipulation, changing the presented year of when the US would reach “majority-minority” status. The original article, published in 2014, wrote that the US would reach majority-minority status in 2023. To maintain the same time period/future setting, I moved the date to 2033. In addition, one of the policy outcome measures was around support for same-sex marriage, which, since the original study was run, has already become legal in the US. Further, the original study contained several duplicated demographic measures (e.g., age, race, political orientation), which I removed for brevity and time.
Differences in analysis: As described previously, I am focusing on the comparison between the assuaged- and the status-threat conditions. As such, I will utilize a t-test rather than an ANOVA for my main analyses.
As these are largely material differences or differences to contextualize the manipulation to 2023, I do not expect these differences alone to drive any differences in any observed effects between this replication, the first replication, or the original article.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
First, I plan to clean the Qualtrics datafile of any identifiable or metadata columns. I will focus my dataset on the participants’ self-reported demographic and political ideology variables, a variable denoting each participants’ randomly assigned condition, and the columns for each individual policy preference item. Participants who incorrectly respond to the attention-check item will be excluded from analysis, as will be participants who did not finish the entire activity.
Load Relevant Libraries and Functions
library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(psych)
Attaching package: 'psych'
The following objects are masked from 'package:ggplot2':
%+%, alpha
Import data
#import Pilot B datasetRawPilotB <-read.csv("/Users/ganmol/Documents/Psych 251_Experimental Methods/craig2014_rescue/data/Pilot B/251 Replication Project - v2_December 5, 2023_19.02.csv")RawPilotB <- RawPilotB[-c(1), ] #delete top two row of metadataRawPilotB <- RawPilotB[-c(1), ] #repeat the previous line to delete top two rows of metadata
Data exclusion / filtering
PilotB_filter = RawPilotB |>filter( Consent ==1, # Gave consent Citizen ==1, # US Citizen Residency_1 ==1, #only select US residents#Race1 == 3, #only select White participants ** this is commented out b/c I forgot to select my sample in Pilot B to only be White Americans ** primeRepeat ==2, #Filter out any participants who have read the stimuli before groupGrowth =="4", #Both of the following are attention checks whitePopChange !=1| whitePopChange !=2|is.na(whitePopChange))#!(Condition == "Status Threat" & groupPower == 1), #!(Condition == "Assuaged Threat" & groupPower == 2)
#creating "baseline political ideology" scorePilotB_cleaned = PilotB_cleaned |>mutate(BaselinePolIdeo = (ConservativeIdeo + LiberalIdeo)/2)#create standardized and composite policy preferences scorePilotB_cleaned = PilotB_cleaned %>%mutate(standardImmCiti = (ImmigrantCitizenship-mean(ImmigrantCitizenship))/sd(ImmigrantCitizenship),standardImmNum = (ImmigrationNumber-mean(ImmigrationNumber))/sd(ImmigrationNumber), standardMilitary = (MilitaryFund -mean(MilitaryFund))/sd(MilitaryFund), standardAffAct = (AffirmativeAction -mean(AffirmativeAction))/sd(AffirmativeAction), # standardHealth = (Healthcare - mean(Healthcare))/sd(Healthcare), # commented out because all respondents in Pilot B said "1", so standard deviation is 0standardEngLang = (EnglishNationalLang -mean(EnglishNationalLang))/sd(EnglishNationalLang), standardPrisons = (Prisons -mean(Prisons))/sd(Prisons) )#create variable for overall conservative policy endorsement scorePilotB_cleaned = PilotB_cleaned |>mutate(ConsPolicyScore = (standardImmCiti + standardImmNum + standardMilitary + standardAffAct +standardEngLang + standardPrisons)) # don't forget to add standardHealth back in here -- removed for Pilot B
Results of control measures
In this replication, I will use a t-test to confirm “that participants in the status-threat condition expressed greater perceived group-status threat than did participants in the assuaged-threat condition” (Craig & Richeson, 2014). This differs from the original study and the first replication in that I only have two conditions (no control), which thus calls for a t-test approach rather than an ANOVA.
# Perform a t-test + print resultresult <-t.test(statusThreatPercept ~ Condition, data = PilotB_cleaned)print(result)
Welch Two Sample t-test
data: statusThreatPercept by Condition
t = 0.70711, df = 2, p-value = 0.5528
alternative hypothesis: true difference in means between group Assuaged Threat and group Status Threat is not equal to 0
95 percent confidence interval:
-5.08487 7.08487
sample estimates:
mean in group Assuaged Threat mean in group Status Threat
6 5
#calculate mean of perceived threatPilotB_cleaned %>%group_by(Condition) %>%summarize(mean_statusthreat =mean(statusThreatPercept),sd_statusthreat =sd(statusThreatPercept) ) -> summarised_data# Create bar chart with error barsggplot(summarised_data, aes(x = Condition, y = mean_statusthreat)) +geom_bar(stat ="identity", fill ="lightblue") +geom_errorbar(aes(ymin = mean_statusthreat - sd_statusthreat,ymax = mean_statusthreat + sd_statusthreat),width =0.2,position =position_dodge(0.9),color ="black") +labs(title ="Mean Status Threat by Condition - Pilot B",x ="Condition",y ="Status Threat") +theme_minimal()
View(PilotB_cleaned)
Confirmatory analysis
In analysis, I plan to use a linear model for confirmatory analyses, as I have two conditions and need to control for various covariates (demographics and baseline ideologies). The original paper and replication utilized ANCOVAs for analyses: “Unless otherwise noted, reported analyses for Studies 3a and 3b are ANCOVAs with experimental condition as the independent variable and demographic characteristics and baseline political ideology as covariates.” I used a t-test to examine the reported role of group status threat (the condition assignment) on policy preferences for White Americans, incorporating participants’ baseline political ideologies into the model.
model <-lm(ConsPolicyScore ~ Condition + Age + BaselinePolIdeo + Gender1 + Education, data = PilotB_cleaned)summary(model)
Call:
lm(formula = ConsPolicyScore ~ Condition + Age + BaselinePolIdeo +
Gender1 + Education, data = PilotB_cleaned)
Residuals:
ALL 4 residuals are 0: no residual degrees of freedom!
Coefficients: (2 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.1634 NaN NaN NaN
ConditionStatus Threat 5.1713 NaN NaN NaN
Age -0.3843 NaN NaN NaN
BaselinePolIdeo 3.3812 NaN NaN NaN
Gender1 NA NA NA NA
Education NA NA NA NA
Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared: 1, Adjusted R-squared: NaN
F-statistic: NaN on 3 and 0 DF, p-value: NA
# box plot of policy preferences; more positive scores = more support for conservative policyggplot(PilotB_cleaned, aes(x = Condition, y = ConsPolicyScore)) +geom_boxplot(fill ="gray", color ="black", width =0.7) +labs(title ="Policy Score by Condition",x ="Condition",y ="PolicyScore") +theme_minimal()
Three-panel graph with original, 1st replication, and your replication is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Mini meta analysis
Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.
