Introduction

This is a replication project of the Study 3 of Porter, Rheinschmidt-Same, and Richeson (2016) titled “Inferring identity from language: Linguistic intergroup bias informs social categorization” published in Psychological Science. The original study examined 1) whether people could infer a communicator’s political group membership based on the kind of language bias in the communicator’s description of another individual whose political group membership was known and 2) whether the effect in (1) could be moderated by the target’s and participants’ own political group memberships. The original study also examined 1) whether the kind of language bias used by the communicator could influence participants’ intention to be friends with the communicator and 2) whether this intention was qualified by the interaction between participants’ and the target’s political group memberships.

The original study found that 1) regardless of participants’ own political affiliations and the target’s political group membership, favorable language bias, compared to unfavorable langauge bias, in the communicator’s description led participants to infer that the communicator and the target shared a political group membership (no moderating effects of the target’s or participants’ own group memberships), and that 2) for the friendship measure, there was a significant interaction between LIB and target’s political affiliation, which was qualified by a three-way interaction between LIB, target political affiliation, and participants’ own political affiliation. Subsequent analyses showed that democrat-identified participants were more likely to be friends with the communicator who used favorable language bias, compared to unfavorable language bias, to describe a democrat – this pattern did not show for Republican or Independent participants.

Methods

Power Analysis

The original effect size for the key test (three-way interaction for the friendship measure) was 0.24. To achieve 80% power, the planned sample size should be 168; 90% should be 220; 95% should be 268.

Planned Sample

Planned sample size is 168. Termination rule is that the study will finish as soon as it has reached this planned sample size. Sample method would be convenience sample on MTurk.

Materials

“As in Study 1a, participants were asked to read a passage and then respond to questions. In the Republican-target condition, the passage indicated that Peter had voted for John McCain; in the Democratic-target condition, Peter had voted for Barack Obama. In the second part of the passage, participants were again provided with an unknown communicator’s description of Peter’s helpful and rude behaviors. Following Wigboldus et.al. (2000), we included a description of one discrete episode, expressed in the present tense, for each type of behavior (for the full descriptions, see Table S1 in the Supplemental Material available online). For example, the description of helpful behavior in the favorable-LIB condition was written in abstract language and read as follows: “On one occasion, there is a person in a wheelchair who needs assistance getting up a ramp. Peter reaches for the handles of the wheelchair. Peter is helpful.” In the unfavorable-LIB condition, helpful behavior was described concretely: “On one occasion, there is a person in a wheelchair who needs assistance getting up a ramp. Peter reaches for the handles of the wheelchair. Peter pushes the wheelchair up the ramp.” After reading the passage, participants indicated the likely group membership of the communicator on an 8-point scale anchored by 1, definitely a Democrat, and 8, definitely a Republican. They then rated the likelihood that they would be friends with the communicator, using a 5-point scale ranging from 1, it is not at all likely, to 5, it is extremely likely. Finally, participants completed the manipulation-check items and a demographic questionnaire on which they reported their political-party affiliation and political ideology.”

Procedure

It is combined with materials – see above.

Analysis Plan

  1. Conduct a 2 (Favorable vs. Unfavorable LIB) x 2 (Democrat vs. Republican Target) ANOVA to examine the manipulation check question – whether favorable LIB (unfavorable LIB) led participants to conclude that the target was more likely to be helpful (rude) in the future and whether this pattern differed depending on the target’s political affiliation.

  2. Conduct a 2 (Favorable vs. Unfavorable LIB) × 2 (Democrat vs. Republican Target) × 3 (Democrat vs. Republican vs. Independent Participant) analysis of variance (ANOVA) to examine whether participants can infer the communicator’s political group identity based on the kind of language bias and the target’s political affiliation as well as whether participants’ own political affiliation moderated their judgment of the communicator’s political identity. Subsequent analyses (two-way ANOVAs for two-way interactions across the levels of the 3rd variable and pairwise t-tests for simple effects) will be conducted depending on the results of the previous ANOVA to explain the details of the interactions.

  3. Conduct a 2 (Favorable vs. Unfavorable LIB) × 2 (Democrat vs. Republican Target) × 3 ( Democrat vs. Republican vs. Independent Participant) analysis of variance of participants’ ratings of their likelihood of becoming friends with the communicator. Subsequent analyses (two-way ANOVAs for two-way interactions across the levels of the 3rd variable and pairwise t-tests for simple effects) will be conducted depending on the results of the previous ANOVA to explain the details of the interactions.

Key Analysis

Key analysis is the three-way interaction in the ANOVA of the friendship measure in the previous section.

Differences from Original Study

Since we failed to obtain the specific wording of questions (prompt) and instructions from the authors, we came up with our own specific wording for the questions and instructions. We do not expect the differences to create significant deviation from the original study. In particular, we used “How would you describe your political affiliation” with three responses “Democrat”, “Republican” and “Independent” (same response options as in the original paper) to ask about participants’ own political affiliations. We used the prompt “How likely would you personally be friends with Peter” to assess participants’ intention to be friends with the communicator. Likewise, we used the prompt “What do you think is the communicator’s political group membership” to ask about participants’ perceptions of the communicator’s political group affiliation given that of the target.

Methods Addendum (Post Data Collection)

Actual Sample

The current replication attempt included 168 participants, the same number as the planned sample size. Among the 168 participants, 106 identified as men, 62 as women, and one as non-baniry. Moreover, racial and ethnic composition of the final sample was such that 29 identified as Asian/Pacific Islander, 8 as Black, 11 as Latino/Hispanic, 3 as Native American, 116 as White, and 2 as Other. All participants finished all the portions of the study, and none of them was excluded from data analyses.

Differences from pre-data collection methods plan

We added descriptive statistics to present the means and SDs of the dependent variables. We also added ANOVA tables for the lm models to obtain F statistics as the original paper did. In addition, we modified the interaction analyses for manipulation checks to more precisely follow the original author’s analysis strategies. Specifically, for interactions in assessing manipulation checks, instead of directly making “Helpful” and “Rude” the DVs of the linear regression models, we combined “Helpful” and “Rude” as two levels of a single variable (IV) and created a placeholder DV named “Percentage” so that we could, in the same way as the original authors did, examine whether the interaction between LIB and the target’s group membership differed as a function of the valence of the target’s behavior (Helpful vs. Rude).

Results

Data preparation

Data preparation following the analysis plan.

clean data

library(readr)
d <- read_csv("~/GitHub/Porter2016/Study 3 Data.csv")

d_tidy = d %>%
  mutate(index=rownames(d)) %>%
  gather(variable, value, Communicator.F.D:Rude.U.R) %>%
  group_by(index) %>%
  separate(variable, into=c("variable.n", "LIB", "Party")) 

ds = spread(d_tidy, variable.n, value)
ds$Party = as.factor(ds$Party)
ds$Political = as.factor(ds$Political)
ds$LIB = as.factor(ds$LIB)

Set up functions to clean regression outputs

#print with 2 digits of precision
fixed_digits <- function(xs, n = 2) {
  formatC(xs, digits = n, format = "f")
}

# Print three digits of a p-value, but use
# the "< .001" notation on tiny values.
format_pval <- function(ps, html = FALSE) {
  tiny <- ifelse(html, "&lt;&nbsp;.001", "< .001")
  ps_chr <- ps %>% fixed_digits(3)
  ps_chr[ps < 0.001] <- tiny
  ps_chr
}
#fix names
fix_names <- . %>%
  str_replace(".Intercept.", "Intercept") %>%
  str_replace("LIBU", "Unfavorable LIB") %>%
  str_replace("LIBF", "Favorable LIB") %>%
  str_replace("PartyD", "Democrat Target") %>%
  str_replace("PartyR", "Republican Target") %>%
  str_replace("PoliticalDemocrat", "Democrat Participant") %>%
  str_replace("PoliticalRepublican", "Republican Participant") %>%
  str_replace("PoliticalIndependent", "Independent Participant")

#formatting pipeline for regression tables
two_digits <- . %>% fixed_digits(2)
table_names_1 <- c("Parameter", "Estimate", "SE",
                 "_t_", "_p_")

format_model_table_r <- . %>%
  mutate_each(funs(two_digits),
              -term, -p.value) %>%
  mutate(term=fix_names(term), p.value = format_pval(p.value)) %>%
  set_colnames(table_names_1)

#formatting pipeline for anova tables
table_names_2 <- c("Parameter", "DF", "Sum Sq", "Mean Sq",
                 "F", "_p_")
format_model_table_k <- . %>%
  mutate_each(funs(two_digits),
              -term, -p.value) %>%
  mutate(term=fix_names(term), p.value = format_pval(p.value)) %>% 
  set_colnames(table_names_2)

Descriptive Stats

table(d$Gender)
## 
##        Man Non-binary      Woman 
##        106          1         62
table(d$Race)
## 
##     Asian/Pacific Islander                      Black 
##                         29                          8 
##            Latino/Hispanic            Native American 
##                         11                          3 
## Prefer to self-describe as                      White 
##                          2                        116
summary(d$Age)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   19.00   26.00   31.00   33.41   37.00   66.00

Confirmatory analysis

The analyses as specified in the analysis plan.

Social Inference Graph in the original study

Table in the original study

Social Inference Graph w/o facets in the current study

Social Inference Graph (faceted by participants’ political affiliation) in the current study

Friendship Graph in the current study

Manipulation Check

Mean values and SDs of perception of future helpfulness grouped by LIB)
Helpfulness = ds %>%
  group_by(LIB) %>%
  summarise(mean=mean(Helpful, na.rm=T), sd=sd(Helpful, na.rm=T))
Helpfulness %>%
  mutate(mean=round(mean, digits=2)) %>%
   mutate(sd=round(sd, digits=2)) %>%
  set_colnames(c("Language Bias", "Mean", "SD")) %>%
    kable(align="r")
Language Bias Mean SD
F 60.51 17.35
U 53.69 15.77
Main effect of LIB on estimation of helpfulness
lm(Helpful~LIB, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 60.51 1.81 33.46 < .001
Unfavorable LIB -6.82 2.55 -2.67 0.008 
f = summary(lm(Helpful~LIB, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "7.15"   "1.00" "167.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.008"
Interaction between LIB and target’s political group
lm(Helpful~LIB*Party, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 60.72 2.54 23.89 < .001
Unfavorable LIB -6.35 3.59 -1.77 0.079
Republican Target -0.43 3.64 -0.12 0.906
Unfavorable LIB:Republican Target -0.94 5.13 -0.18 0.854 
f = summary(lm(Helpful~LIB*Party, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "2.41"   "3.00" "165.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.069"
aov(lm(Helpful~LIB*Party, ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 1963.81 1963.81 7.07 0.009
Party 1.00 34.44 34.44 0.12 0.725
LIB:Party 1.00 9.41 9.41 0.03 0.854
Residuals 165.00 45827.19 277.74 NA NA
Mean values and SDs of perception of future rudeness grouped by LIB
Rudeness = ds %>%
  group_by(LIB) %>%
  summarise(mean=mean(Rude, na.rm=T), sd=sd(Rude, na.rm=T))
Rudeness %>%
  mutate(mean=round(mean, digits=2)) %>%
   mutate(sd=round(sd, digits=2)) %>%
  set_colnames(c("Language Bias", "Mean", "SD")) %>%
    kable(align="r")
Language Bias Mean SD
F 43.98 21.99
U 46.04 21.12
Main effect of LIB on estimation of rudeness
lm(Rude~LIB, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 43.98 2.35 18.69 < .001
Unfavorable LIB 2.06 3.32 0.62 0.536 
f = summary(lm(Rude~LIB, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "0.39"   "1.00" "167.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.536"
Interaction between LIB and target’s political group
lm(Rude~LIB*Party, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 44.44 3.27 13.60 < .001
Unfavorable LIB -3.05 4.62 -0.66 0.511
Republican Target -0.95 4.68 -0.20 0.839
Unfavorable LIB:Republican Target 10.34 6.59 1.57 0.119 
f = summary(lm(Rude~LIB*Party, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "1.50"   "3.00" "165.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.215"
Test the differences between ratings of helpfulness and rudeness regardless of the target’s party
d_merge = ds %>%
  gather(variable, Percentage, Helpful, Rude) %>%
  group_by(index) %>%
  separate(variable, into=c("FutureBehavior")) 

lm(Percentage~LIB*FutureBehavior, d_merge) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 60.51 2.10 28.84 < .001
Unfavorable LIB -6.82 2.96 -2.30 0.022
FutureBehaviorRude -16.54 2.97 -5.57 < .001
Unfavorable LIB:FutureBehaviorRude 8.88 4.18 2.12 0.035 
aov(lm(Percentage~LIB*FutureBehavior, d_merge)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 478.36 478.36 1.29 0.256
FutureBehavior 1.00 12312.43 12312.43 33.30 < .001
LIB:FutureBehavior 1.00 1664.57 1664.57 4.50 0.035
Residuals 334.00 123497.88 369.75 NA NA 
#change the reference level of LIB to intrepret the interaction
d_merge$LIB = relevel(d_merge$LIB, ref = "U")
lm(Percentage~LIB*FutureBehavior, d_merge) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 53.69 2.09 25.74 < .001
Favorable LIB 6.82 2.96 2.30 0.022
FutureBehaviorRude -7.66 2.95 -2.60 0.010
Favorable LIB:FutureBehaviorRude -8.88 4.18 -2.12 0.035 
aov(lm(Percentage~LIB*FutureBehavior, d_merge)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 478.36 478.36 1.29 0.256
FutureBehavior 1.00 12312.43 12312.43 33.30 < .001
LIB:FutureBehavior 1.00 1664.57 1664.57 4.50 0.035
Residuals 334.00 123497.88 369.75 NA NA
Test whether the interaction between LIB and target’s party differs as a function of the type of future behavior assessed
lm(Percentage~LIB*Party*FutureBehavior, d_merge) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 54.37 2.93 18.58 < .001
Favorable LIB 6.35 4.14 1.53 0.126
Republican Target -1.37 4.16 -0.33 0.742
FutureBehaviorRude -12.98 4.14 -3.14 0.002
Favorable LIB:Republican Target 0.94 5.91 0.16 0.873
Favorable LIB:FutureBehaviorRude -3.30 5.85 -0.56 0.573
Republican Target:FutureBehaviorRude 10.76 5.89 1.83 0.068
Favorable LIB:Republican Target:FutureBehaviorRude -11.29 8.35 -1.35 0.178 
aov(lm(Percentage~LIB*Party*FutureBehavior, d_merge)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 478.36 478.36 1.30 0.255
Party 1.00 236.54 236.54 0.64 0.424
FutureBehavior 1.00 12312.43 12312.43 33.42 < .001
LIB:Party 1.00 466.53 466.53 1.27 0.261
LIB:FutureBehavior 1.00 1664.57 1664.57 4.52 0.034
Party:FutureBehavior 1.00 560.73 560.73 1.52 0.218
LIB:Party:FutureBehavior 1.00 672.70 672.70 1.83 0.178
Residuals 330.00 121561.38 368.37 NA NA
Test the difference between the helpful and rude models when the target is a democrat
d_D = d_merge %>%
  filter(Party=="D")

lm(Percentage~LIB*FutureBehavior, d_D) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 54.37 2.87 18.93 < .001
Favorable LIB 6.35 4.06 1.56 0.120
FutureBehaviorRude -12.98 4.06 -3.20 0.002
Favorable LIB:FutureBehaviorRude -3.30 5.74 -0.57 0.566 
aov(lm(Percentage~LIB*FutureBehavior, d_D)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 948.93 948.93 2.68 0.104
FutureBehavior 1.00 9200.95 9200.95 25.95 < .001
LIB:FutureBehavior 1.00 117.23 117.23 0.33 0.566
Residuals 168.00 59577.58 354.63 NA NA
Test the difference between the helpful and rude models when the target is a Republican
d_R = d_merge %>%
  filter(Party=="R")

#when favorable LIB is the refernce
lm(Percentage~LIB*FutureBehavior, d_R) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 53.00 3.02 17.56 < .001
Favorable LIB 7.29 4.29 1.70 0.091
FutureBehaviorRude -2.21 4.27 -0.52 0.605
Favorable LIB:FutureBehaviorRude -14.59 6.07 -2.40 0.017 
#when unfavorable LIB is the reference
d_R$LIB = relevel(d_R$LIB, ref="U")
lm(Percentage~LIB*FutureBehavior, d_R) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 53.00 3.02 17.56 < .001
Favorable LIB 7.29 4.29 1.70 0.091
FutureBehaviorRude -2.21 4.27 -0.52 0.605
Favorable LIB:FutureBehaviorRude -14.59 6.07 -2.40 0.017 
aov(lm(Percentage~LIB*FutureBehavior, d_R)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 0.00 0.00 0.00 0.999
FutureBehavior 1.00 3683.88 3683.88 9.63 0.002
LIB:FutureBehavior 1.00 2208.37 2208.37 5.77 0.017
Residuals 162.00 61983.80 382.62 NA NA

Social Identity Measure

Mean values of inferences of the communicator’s political membership grouped by the target’s political membership and language bias
Inference = ds %>%
  group_by(Party, LIB) %>%
  summarise(mean=mean(Communicator, na.rm=T), sd=sd(Communicator, na.rm=T))
Inference %>%
  mutate(mean=round(mean, digits=2)) %>%
   mutate(sd=round(sd, digits=2)) %>%
  set_colnames(c("Target Party", "Language Bias", "Mean", "SD")) %>%
    kable(align="r")
Target Party Language Bias Mean SD
D F 5.88 1.79
D U 6.07 2.09
R F 6.88 0.87
R U 7.02 0.87
Mean values of inferences of the communicator’s political membership grouped by the target’s political membership, language bias, and participants’ political membership
Inference_1 = ds %>%
  group_by(Party, LIB, Political) %>%
  summarise(mean=mean(Communicator, na.rm=T), sd=sd(Communicator, na.rm=T))

Inference_1 %>%
  mutate(mean=round(mean, digits=2)) %>%
  mutate(sd=round(sd, digits=2)) %>%
  set_colnames(c("Target Party", "Language Bias", "Participant Party", "Mean", "SD")) %>%
  kable(align="r")
Target Party Language Bias Participant Party Mean SD
D F Democrat 6.19 1.64
D F Independent 5.43 1.95
D F Republican 6.00 1.83
D U Democrat 6.61 1.04
D U Independent 6.35 2.21
D U Republican 4.25 2.76
R F Democrat 7.19 0.91
R F Independent 6.92 0.86
R F Republican 6.42 0.67
R U Democrat 6.90 0.91
R U Independent 7.25 0.86
R U Republican 6.83 0.75
Main effect of LIB on perceptions of the communicator’s social group membership
lm(Communicator~LIB, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")   #LIB
Parameter Estimate SE t p
Intercept 6.37 0.17 36.87 < .001
Unfavorable LIB 0.17 0.24 0.71 0.481 
f = summary(lm(Communicator~LIB, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "0.50"   "1.00" "167.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.481"
aov(lm(Communicator~LIB, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 1.25 1.25 0.50 0.481
Residuals 167.00 418.67 2.51 NA NA
Main effect of target’s party on perceptions of the communicator’s social group membership
lm(Communicator~Party, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")  #Target's group membership
Parameter Estimate SE t p
Intercept 5.98 0.16 36.76 < .001
Republican Target 0.98 0.23 4.20 < .001 
f = summary(lm(Communicator~Party, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##  "17.66"   "1.00" "167.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##    value 
## "< .001"
aov(lm(Communicator~Party, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
Party 1.00 40.16 40.16 17.66 < .001
Residuals 167.00 379.76 2.27 NA NA
Main effect of participant’s party
lm(Communicator~Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")   #Participants' own group membership
Parameter Estimate SE t p
Intercept 6.73 0.19 36.16 < .001
Independent Participant -0.23 0.27 -0.83 0.405
Republican Participant -0.83 0.31 -2.67 0.008 
f = summary(lm(Communicator~Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "3.61"   "2.00" "166.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.029"
aov(lm(Communicator~Political, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
Political 2.00 17.48 8.74 3.61 0.029
Residuals 166.00 402.43 2.42 NA NA
Three-way interaction Contrast 1 – When Favorable LIB, Democrat (communicator), and Democrat (participant) are the defaults
lm(Communicator~LIB*Party*Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 6.19 0.37 16.93 < .001
Unfavorable LIB 0.42 0.50 0.84 0.400
Republican Target 1.00 0.52 1.94 0.055
Independent Participant -0.76 0.53 -1.42 0.158
Republican Participant -0.19 0.55 -0.34 0.732
Unfavorable LIB:Republican Target -0.71 0.70 -1.01 0.313
Unfavorable LIB:Independent Participant 0.50 0.73 0.69 0.493
Unfavorable LIB:Republican Participant -2.17 0.83 -2.63 0.009
Republican Target:Independent Participant 0.49 0.76 0.65 0.519
Republican Target:Republican Participant -0.58 0.78 -0.75 0.456
Unfavorable LIB:Republican Target:Independent Participant 0.11 1.03 0.11 0.913
Unfavorable LIB:Republican Target:Republican Participant 2.88 1.21 2.38 0.018 
f = summary(lm(Communicator~LIB*Party*Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "3.59"  "11.00" "157.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##    value 
## "< .001"
aov(lm(Communicator~LIB*Party*Political, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 1.25 1.25 0.59 0.445
Party 1.00 40.07 40.07 18.76 < .001
Political 2.00 14.58 7.29 3.41 0.035
LIB:Party 1.00 0.05 0.05 0.02 0.881
LIB:Political 2.00 11.51 5.75 2.69 0.071
Party:Political 2.00 3.14 1.57 0.74 0.481
LIB:Party:Political 2.00 13.88 6.94 3.25 0.041
Residuals 157.00 335.44 2.14 NA NA 
##########
lm(Communicator~LIB*Party,ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 5.88 0.23 25.47 < .001
Unfavorable LIB 0.19 0.33 0.57 0.570
Republican Target 0.99 0.33 3.01 0.003
Unfavorable LIB:Republican Target -0.04 0.47 -0.09 0.931 
f = summary(lm(Communicator~LIB*Party,ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "6.01"   "3.00" "165.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##    value 
## "< .001"
aov(lm(Communicator~LIB*Party,ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 1.25 1.25 0.55 0.461
Party 1.00 40.07 40.07 17.47 < .001
LIB:Party 1.00 0.02 0.02 0.01 0.931
Residuals 165.00 378.58 2.29 NA NA 
############  
lm(Communicator~LIB*Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 6.69 0.27 24.46 < .001
Unfavorable LIB 0.08 0.37 0.20 0.839
Independent Participant -0.54 0.40 -1.33 0.184
Republican Participant -0.49 0.41 -1.18 0.239
Unfavorable LIB:Independent Participant 0.56 0.55 1.03 0.304
Unfavorable LIB:Republican Participant -0.92 0.64 -1.44 0.150 
f = summary(lm(Communicator~LIB*Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "2.51"   "5.00" "163.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.032"
aov(lm(Communicator~LIB*Political, ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 1.25 1.25 0.52 0.470
Political 2.00 16.49 8.24 3.45 0.034
LIB:Political 2.00 12.30 6.15 2.57 0.080
Residuals 163.00 389.88 2.39 NA NA 
############
lm(Communicator~Party*Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 6.41 0.26 25.13 < .001
Republican Target 0.62 0.36 1.73 0.085
Independent Participant -0.48 0.37 -1.29 0.199
Republican Participant -1.08 0.41 -2.61 0.010
Republican Target:Independent Participant 0.55 0.52 1.05 0.294
Republican Target:Republican Participant 0.61 0.60 1.02 0.310 
f = summary(lm(Communicator~Party*Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "5.33"   "5.00" "163.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##    value 
## "< .001"
aov(lm(Communicator~Party*Political, ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
Party 1.00 40.16 40.16 18.14 < .001
Political 2.00 15.50 7.75 3.50 0.032
Party:Political 2.00 3.38 1.69 0.76 0.468
Residuals 163.00 360.88 2.21 NA NA

Friendship Measure

Mean values and SDs of friendship grouped by LIB and Target’s political membership
Friendship = ds %>%
  group_by(Party, LIB) %>%
  summarise(mean=mean(Friend, na.rm=T), sd=sd(Friend, na.rm=T))
Friendship %>%
  mutate(mean=round(mean, digits=2)) %>%
   mutate(sd=round(sd, digits=2)) %>%
  set_colnames(c("Target Party", "Language Bias", "Mean", "SD")) %>%
    kable(align="r")
Target Party Language Bias Mean SD
D F 2.95 0.79
D U 2.53 0.80
R F 3.00 0.84
R U 2.79 0.84
Mean values and SDs of friendship grouped by LIB, Target’s political membership, and Participants’ political group membership
Friendship_1 = ds %>%
  group_by(Party, LIB, Political) %>%
  summarise(mean=mean(Friend, na.rm=T), sd=sd(Friend, na.rm=T))
Friendship_1 %>%
  mutate(mean=round(mean, digits=2)) %>%
   mutate(sd=round(sd, digits=2)) %>%
  set_colnames(c("Target Party", "Language Bias", "Participant Party", "Mean", "SD")) %>%
    kable(align="r")
Target Party Language Bias Participant Party Mean SD
D F Democrat 2.94 0.57
D F Independent 2.93 0.83
D F Republican 3.00 1.00
D U Democrat 2.50 0.62
D U Independent 2.76 0.90
D U Republican 2.12 0.83
R F Democrat 3.12 0.96
R F Independent 2.77 0.73
R F Republican 3.08 0.79
R U Democrat 2.75 0.91
R U Independent 2.75 0.77
R U Republican 3.00 0.89
Main effect of LIB on participants’ intention to be friends with the communicator
lm(Friend~LIB, ds)%>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")  #LIB
Parameter Estimate SE t p
Intercept 2.98 0.09 33.45 < .001
Unfavorable LIB -0.32 0.13 -2.53 0.012 
f = summary(lm(Friend~LIB, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "6.40"   "1.00" "167.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.012"
aov(lm(Friend~LIB,ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 4.26 4.26 6.40 0.012
Residuals 167.00 111.06 0.67 NA NA
Main effect of target’s party on participants’ intention to be friends with the communicator
lm(Friend~Party, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")   #Target's group membership
Parameter Estimate SE t p
Intercept 2.74 0.09 30.75 < .001
Republican Target 0.15 0.13 1.16 0.249 
f = summary(lm(Friend~Party, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "1.34"   "1.00" "167.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.249"
aov(lm(Friend~Party,ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
Party 1.00 0.92 0.92 1.34 0.249
Residuals 167.00 114.40 0.69 NA NA
Main effect of participants’ party on participants’ intention to be friends with the communicator
lm(Friend~Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")   #Participants' own group membership
Parameter Estimate SE t p
Intercept 2.81 0.10 28.26 < .001
Independent Participant -0.01 0.15 -0.10 0.922
Republican Participant 0.03 0.17 0.19 0.848 
f = summary(lm(Friend~Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "0.04"   "2.00" "166.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.964"
aov(lm(Friend~Political, ds)) %>%
  tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
Political 2.00 0.05 0.03 0.04 0.964
Residuals 166.00 115.26 0.69 NA NA
Three-way interaction Contrast 1 – When Favorable LIB, Democrat (communicator), and Democrat (participant) are the defaults
lm(Friend~LIB*Party*Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 2.94 0.21 14.30 < .001
Unfavorable LIB -0.44 0.28 -1.55 0.123
Republican Target 0.19 0.29 0.65 0.520
Independent Participant -0.01 0.30 -0.03 0.976
Republican Participant 0.06 0.31 0.20 0.839
Unfavorable LIB:Republican Target 0.06 0.39 0.16 0.874
Unfavorable LIB:Independent Participant 0.27 0.41 0.67 0.505
Unfavorable LIB:Republican Participant -0.44 0.46 -0.94 0.348
Republican Target:Independent Participant -0.35 0.43 -0.81 0.421
Republican Target:Republican Participant -0.10 0.44 -0.24 0.813
Unfavorable LIB:Republican Target:Independent Participant 0.08 0.58 0.14 0.888
Unfavorable LIB:Republican Target:Republican Participant 0.73 0.68 1.07 0.284 
f = summary(lm(Friend~LIB*Party*Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "1.25"  "11.00" "157.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.258"
aov(lm(Friend~LIB*Party*Political, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 4.26 4.26 6.30 0.013
Party 1.00 0.94 0.94 1.39 0.240
Political 2.00 0.01 0.00 0.01 0.993
LIB:Party 1.00 0.44 0.44 0.65 0.422
LIB:Political 2.00 1.28 0.64 0.94 0.391
Party:Political 2.00 1.53 0.76 1.13 0.325
LIB:Party:Political 2.00 0.84 0.42 0.62 0.537
Residuals 157.00 106.02 0.68 NA NA 
#########
lm(Friend~LIB*Party, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 2.95 0.12 23.76 < .001
Unfavorable LIB -0.42 0.18 -2.38 0.018
Republican Target 0.05 0.18 0.26 0.794
Unfavorable LIB:Republican Target 0.20 0.25 0.81 0.417 
f = summary(lm(Friend~LIB*Party, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "2.83"   "3.00" "165.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.040"
aov(lm(Friend~LIB*Party, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
LIB 1.00 4.26 4.26 6.40 0.012
Party 1.00 0.94 0.94 1.42 0.236
LIB:Party 1.00 0.44 0.44 0.66 0.417
Residuals 165.00 109.68 0.66 NA NA 
#########change reference level of Party to parse the two-way interaction of LIB*Party
ds$Party = relevel(ds$Party, ref = "R")
lm(Friend~LIB*Party, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 3.00 0.13 23.56 < .001
Unfavorable LIB -0.21 0.18 -1.20 0.233
Democrat Target -0.05 0.18 -0.26 0.794
Unfavorable LIB:Democrat Target -0.20 0.25 -0.81 0.417 
f = summary(lm(Friend~LIB*Party, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "2.83"   "3.00" "165.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.040"
##########
lm(Friend~Party*Political, ds) %>%
  tidy %>%
  format_model_table_r %>%
  kable(align = "r")
Parameter Estimate SE t p
Intercept 2.92 0.14 21.03 < .001
Democrat Target -0.21 0.20 -1.06 0.291
Independent Participant -0.16 0.21 -0.76 0.448
Republican Participant 0.14 0.24 0.58 0.564
Democrat Target:Independent Participant 0.29 0.29 0.99 0.322
Democrat Target:Republican Participant -0.18 0.33 -0.53 0.594 
f = summary(lm(Friend~Party*Political, ds))$fstatistic
f %>%
  two_digits
##    value    numdf    dendf 
##   "0.69"   "5.00" "163.00"
pf(f[1], f[2], f[3], lower=FALSE) %>%
  format_pval
##   value 
## "0.632"
aov(lm(Friend~Party*Political, ds)) %>%
   tidy %>%
  format_model_table_k %>%
  kable(align = "r")
Parameter DF Sum Sq Mean Sq F p
Party 1.00 0.92 0.92 1.32 0.252
Political 2.00 0.06 0.03 0.04 0.957
Party:Political 2.00 1.41 0.71 1.02 0.364
Residuals 163.00 112.92 0.69 NA NA

Exploratory Analyses

Chi-square test of goodness of fit comparing the compositions of participants’ political affiliation between the current replication attempt and the original study

#Proportions of political groups represented by participants in the current sample
prop.table(table(ds$Political))
## 
##    Democrat Independent  Republican 
##   0.4142012   0.3550296   0.2307692
#chi-square test of goodness of fit comparing the current sample with the original
chisq.test(x=prop.table(table(ds$Political)), p=c(0.414, 0.313, 0.172), rescale.p = T)
## 
##  Chi-squared test for given probabilities
## 
## data:  prop.table(table(ds$Political))
## X-squared = 0.012925, df = 2, p-value = 0.9936

Discussion

Summary of Replication Attempt

Results in the current replication attempt revealed that overall the original study failed to replicate in the current attempt.

Specifically, first of all, unlike the original study, the current replication attempt did not find a significant main effect of language bias on social inference, F(1, 167) = 0.50, p = .48. Moreover, while the original study did not find a significant main effect of the target’s group membership on social inference, the current replication attempt did find such a significant main effect – specifically, regardless of language bias, participants were more likely to perceive the communicator as sharing the same group membership with the target when the target was a Republican than when the target was a Democrat, F(1, 167) = 17.66, p <.001. Similarly, while the original study did not find a significant main effect of participant’s group membership on social inference, the current attemp did find such a significant main effect, F(2, 166) = 3.61, p = .029 – specifically, on average, participants who identified as Republican were less likely than those who identified as Democrat to perceive the communicator as having the same group membership as the target, t(166) = -2.67, p = .008, whereas those who identified as Independent did not differ significantly from those who identified as Democrat, t(166) = -0.83, p = .405. In addition, while the original study did not find a significant three-way interactive effect between target’s group membership, language bias, and participants’ group membership on social inference, the current attempt did find this interaction to be significant, F(2, 157) = 3.25, p = .041. Since the key measure of interest is whether language bias can reveal group membership (main effect of language bias), the fact that the current attemp did not find a significant main effect of language bias on judgement of the communicator’s membership suggests that the original results on the social inferences meaure failed to replicate.

Second, the current attempt also failed to find results consistent with the original ones on the friendship measure. Specifically, while the original study found a significant three-way interaction between the target’s group membership, language bias, and participants’ own group membership, the current attempt did not find this interaction to be significant, F(2, 166) = 0.62, p = .537. Moreover, the original study also found a significant two-way interaction between the target’s and participants’ group memberships, nested in the previous three-way interaction, but the current attempt failed to find such a significant interaction, F(2, 166) = 1.02, p = .364. Since the three-way interaction is the key result of interest, the fact that the current attempt did not find this interaction to be significant suggests that the original study failed to replicate on the friendship measure.

In sum, since Study 3 in the original paper focused on the friendship measure and whether participants’ own group membership could be a moderator, the fact that the current study did not find any significant interactions on the friendship measure suggests that Study 3 in Porter et al. (2016) failed to replication in the current attempt.

Commentary

The most plausible reason for the failure of replication in the current study is the ineffectiveness of the manipulation, which was reported as successful in the original study. Specifically, the analyses of the manipulation checks revealed that while the manipulation of language bias did influence participants’ perception of future helpfulness of the target, F(1, 167) = 7.15, p = .008, the manipulation failed to exert significant effects on participants’ perception of future rudeness of the target, F(1, 167) = 0.39, p = .536. Therefore, the failure to manipulate perceptions of the target’s future rudeness by language bias might have reduced the contrast between depictions of positive versus negative behavior that could have made in-group favoritism more salient for participants to infer the communicator’s group membership.

We also considered the possibility that the proportions of political group affiliations of participants in the current study might be different from those in the original study, which might have contributed to the replication failure. However, a chi-square test of goodness of fit revealed that the proportions of political groups represented by participants in the current attempt did not differ significantly from the proportions in the original study, X^2(2) = 0.01, p =.99.

Lastly, it is also possible that the survey design made it hard for participants to retain memory of the content of the vignette. Specifically, the vignette passage was displayed on a separage page from the questions, and each question was also displayed on a separage page. Between the pages there was no “go back” option, so if participants forgot what was in the vignette, they had no means of going back to read the vignette again. Should we have included an attention check in the end, we might have known whether this indeed contributed to the failure of replication.