Replication of Study 3 by Porter, Rheinschmidt-Same, and Richeson (2016, Psychological Science)
Cai Guo (caiguo@stanford.edu)
March 07, 2017
Introduction
This is a replication project of the Study 3 of Porter, Rheinschmidt-Same, and Richeson (2016) titled “Inferring identity from language: Linguistic intergroup bias informs social categorization” published in Psychological Science. The original study examined 1) whether people could infer a communicator’s political group membership based on the kind of language bias in the communicator’s description of another individual whose political group membership was known, and 2) whether the kind of language bias used by the communicator could influence participants’ intention to be friends with the communicator and whether this intention was qualified by the interaction between participants’ and the target’s political group memberships.
The original study found that 1) regardless of participants’ own political affiliations and the target’s political group membership, favorable language bias, compared to unfavorable langauge bias, in the communicator’s description led participants to infer that the communicator and the target shared a political group membership, and that 2) for the friendship measure, there was a significant interaction between LIB and target’s political affiliation, which was qualified by a three-way interaction between LIB, target political affiliation and participants’ own political affiliation. Subsequent analyses showed that democrat-identified participants were more likely to be friends with the communicator used favorable language bias, compared to unfavorable language bias, to describe a democrat – this pattern did not show for Republican or Independent participants.
Methods
Power Analysis
The original effect size for the key test (three-way interaction for the friendship measure) was 0.24. To achieve 80% power, the planned sample size should be 168; 90% should be 220; 95% should be 268.
Planned Sample
Planned sample size is 168. Termination rule is that the study will finish as soon as it has reached this planned sample size. Sample method would be convenience sample on MTurk.
Materials
“As in Study 1a, participants were asked to read a passage and then respond to questions. In the Republican-target condition, the passage indicated that Peter had voted for John McCain; in the Democratic-target condition, Peter had voted for Barack Obama. In the second part of the passage, participants were again provided with an unknown communicator’s description of Peter’s helpful and rude behaviors. Following Wigboldus et.al. (2000), we included a description of one discrete episode, expressed in the present tense, for each type of behavior (for the full descriptions, see Table S1 in the Supplemental Material available online). For example, the description of helpful behavior in the favorable-LIB condition was written in abstract language and read as follows: “On one occasion, there is a person in a wheelchair who needs assistance getting up a ramp. Peter reaches for the handles of the wheelchair. Peter is helpful.” In the unfavorable-LIB condition, helpful behavior was described concretely: “On one occasion, there is a person in a wheelchair who needs assistance getting up a ramp. Peter reaches for the handles of the wheelchair. Peter pushes the wheelchair up the ramp.” After reading the passage, participants indicated the likely group membership of the communicator on an 8-point scale anchored by 1, definitely a Democrat, and 8, definitely a Republican. They then rated the likelihood that they would be friends with the communicator, using a 5-point scale ranging from 1, it is not at all likely, to 5, it is extremely likely. Finally, participants completed the manipulation-check items and a demographic questionnaire on which they reported their political-party affiliation and political ideology.”
Procedure
It is combined with materials – see above.
Analysis Plan
Conduct a two-way ANOVA (LIB x Target’s political affiliation) to examine the manipulation check question – whether favorable LIB (unfavorable LIB) led participants to conclude that the target was more likely to be helpful (rude) in the future and whether this pattern differed depending on the target’s political affiliation.
Conduct a 2 (LIB condition) × 2 (target’s political affiliation) × 3 (participant’s political affiliation: Democrat vs. Republican vs. Independent) analysis of variance (ANOVA) to examine whether participants can infer the communicator’s political group identity based on the kind of language bias and the target’s political affiliation as well as whether participants’ own political affiliation moderated their judgment of the communicator’s political identity. Subsequent analyses (two-way ANOVAs for two-way interactions across the levels of the 3rd variable and pairwise t-tests for simple effects) will be conducted depending on the results of the previous ANOVA to explain the details of the interactions.
Conduct a 2 (LIB condition) × 2 (target’s political affiliation) × 3 (participant’s political affiliation: Democrat vs. Republican vs. Independent) analysis of variance of participants’ ratings of their likelihood of becoming friends with the communicator. Subsequent analyses (two-way ANOVAs for two-way interactions across the levels of the 3rd variable and pairwise t-tests for simple effects) will be conducted depending on the results of the previous ANOVA to explain the details of the interactions.
Key Analysis
Key analysis is the three-way interaction in the ANOVA of the friendship measure in the previous section.
Differences from Original Study
Since we failed to obtain the specific wording of questions and instructions from the authors, we came up with our own specific wording for the questions and instructions. We do not expect the difference to create significant deviation from the original study.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
clean data
d <- read_csv("~/Downloads/Pillot B.csv")
d_tidy = d %>%
mutate(index=rownames(d)) %>%
gather(variable, value, Communicator.F.D:Rude.U.R) %>%
group_by(index) %>%
separate(variable, into=c("variable.n", "LIB", "Party"))
ds = spread(d_tidy, variable.n, value)
ds$Party = as.factor(ds$Party)
ds$Political = as.factor(ds$Political)
ds$LIB = as.factor(ds$LIB)Set up functions to clean regression outputs
#print with 2 digits of precision
fixed_digits <- function(xs, n = 2) {
formatC(xs, digits = n, format = "f")
}
# Print three digits of a p-value, but use
# the "< .001" notation on tiny values.
format_pval <- function(ps, html = FALSE) {
tiny <- ifelse(html, "< .001", "< .001")
ps_chr <- ps %>% fixed_digits(3)
ps_chr[ps < 0.001] <- tiny
ps_chr
}
#fix names
fix_names <- . %>%
str_replace(".Intercept.", "Intercept") %>%
str_replace("LIBU", "Unfavorable LIB") %>%
str_replace("LIBF", "Favorable LIB") %>%
str_replace("PartyD", "Democrat Target") %>%
str_replace("PartyR", "Republican Target") %>%
str_replace("PoliticalDemocrat", "Democrat Participant") %>%
str_replace("PoliticalRepublican", "Republican Participant") %>%
str_replace("PoliticalIndependent", "Independent Participant")
#formatting pipeline
two_digits <- . %>% fixed_digits(2)
table_names <- c("Parameter", "Estimate", "SE",
"_t_", "_p_")
format_model_table <- . %>%
mutate_each(funs(two_digits),
-term, -p.value) %>%
mutate(term=fix_names(term), p.value = format_pval(p.value)) %>%
set_colnames(table_names)Descriptive Stats
table(d$Gender)##
## Man Woman
## 11 4table(d$Race)##
## White
## 4summary(d$Age)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 23.00 25.00 28.00 31.53 33.00 60.00Manipulation Check
#main effect of LIB on estimation of helpfulness
lm(Helpful~LIB, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 57.37 | 6.04 | 9.50 | < .001 |
| Unfavorable LIB | -0.09 | 8.84 | -0.01 | 0.992 |
f = summary(lm(Helpful~LIB, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.00" "1.00" "13.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.992"#interaction between LIB and target's political group
lm(Helpful~LIB*Party, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 59.67 | 10.68 | 5.58 | < .001 |
| Unfavorable LIB | -2.92 | 14.13 | -0.21 | 0.840 |
| Republican Target | -3.67 | 13.52 | -0.27 | 0.791 |
| Unfavorable LIB:Republican Target | 4.92 | 19.56 | 0.25 | 0.806 |
f = summary(lm(Helpful~LIB*Party, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.03" "3.00" "11.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.994"#main effect of LIB on estimation of rudeness
lm(Rude~LIB, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 42.37 | 7.18 | 5.91 | < .001 |
| Unfavorable LIB | 16.91 | 10.50 | 1.61 | 0.131 |
f = summary(lm(Rude~LIB, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "2.59" "1.00" "13.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.131"#interaction between LIB and target's political group
lm(Rude~LIB*Party, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 45.33 | 12.51 | 3.62 | 0.004 |
| Unfavorable LIB | 17.92 | 16.55 | 1.08 | 0.302 |
| Republican Target | -4.73 | 15.83 | -0.30 | 0.770 |
| Unfavorable LIB:Republican Target | -4.52 | 22.90 | -0.20 | 0.847 |
f = summary(lm(Rude~LIB*Party, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.89" "3.00" "11.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.476"Friendship Measure
#main effect of LIB
lm(Friend~LIB, ds)%>%
tidy %>%
format_model_table %>%
kable(align = "r") #LIB| Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 2.25 | 0.52 | 4.31 | < .001 |
| Unfavorable LIB | 0.89 | 0.76 | 1.17 | 0.264 |
f = summary(lm(Friend~LIB, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "1.36" "1.00" "13.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.264"#main effect of target's party
lm(Friend~Party, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") #Target's group membership| Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 2.86 | 0.58 | 4.91 | < .001 |
| Republican Target | -0.36 | 0.80 | -0.45 | 0.662 |
f = summary(lm(Friend~Party, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.20" "1.00" "13.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.662"#main effect of participant's party
lm(Friend~Political, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") #Participants' own group membership| Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 2.73 | 0.48 | 5.64 | < .001 |
| Independent Participant | -0.73 | 1.67 | -0.43 | 0.672 |
| Republican Participant | -0.06 | 1.04 | -0.06 | 0.955 |
f = summary(lm(Friend~Political, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.09" "2.00" "12.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.911"#Contrast 1 -- When Favorable LIB, Democrat (communicator), and Democrat (participant) are the defaults
lm(Friend~LIB*Party*Political, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 3.00 | 1.17 | 2.56 | 0.034 |
| Unfavorable LIB | 1.00 | 2.03 | 0.49 | 0.635 |
| Republican Target | -1.20 | 1.39 | -0.87 | 0.412 |
| Independent Participant | -2.00 | 2.34 | -0.85 | 0.418 |
| Republican Participant | -0.00 | 2.03 | -0.00 | 1.000 |
| Unfavorable LIB:Republican Target | 0.87 | 2.36 | 0.37 | 0.723 |
| Unfavorable LIB:Republican Participant | -1.50 | 2.87 | -0.52 | 0.615 |
f = summary(lm(Friend~LIB*Party*Political, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.57" "6.00" "8.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.746"#########
lm(Friend~LIB*Party, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 3.00 | 0.86 | 3.50 | 0.005 |
| Unfavorable LIB | -0.25 | 1.13 | -0.22 | 0.829 |
| Republican Target | -1.20 | 1.08 | -1.11 | 0.292 |
| Unfavorable LIB:Republican Target | 2.12 | 1.57 | 1.35 | 0.204 |
f = summary(lm(Friend~LIB*Party, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "1.08" "3.00" "11.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.399"##########
lm(Friend~Party*Political, ds) %>%
tidy %>%
format_model_table %>%
kable(align = "r") | Parameter | Estimate | SE | t | p |
|---|---|---|---|---|
| Intercept | 3.33 | 0.94 | 3.54 | 0.005 |
| Republican Target | -0.83 | 1.11 | -0.75 | 0.467 |
| Independent Participant | -1.33 | 1.89 | -0.71 | 0.494 |
| Republican Participant | -0.67 | 1.33 | -0.50 | 0.627 |
f = summary(lm(Friend~Party*Political, ds))$fstatistic
f %>%
two_digits## value numdf dendf
## "0.25" "3.00" "11.00"pf(f[1], f[2], f[3], lower=FALSE) %>%
format_pval## value
## "0.860"#The exact contrasts to test simple effects will be detailed after all data have been collected and the significant interactions have been known.Confirmatory analysis
The analyses as specified in the analysis plan.
Social Inference Graph in the original study
Table in the original study
Social Inference Graph w/o facets in the current study
Social Inference Graph (faceted by participants’ political affiliation) in the current study
Friendship Graph in the current study
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.
Social Identity Measure