Replication of Study 3 by Porter, Rheinschmidt-Same, and Richeson (2016, Psychological Science)
Cai Guo (caiguo@stanford.edu)
March 07, 2017
Introduction
This is a replication project of the Study 3 of Porter, Rheinschmidt-Same, and Richeson (2016) titled “Inferring identity from language: Linguistic intergroup bias informs social categorization” published in Psychological Science. The original study examined 1) whether people could infer a communicator’s political group membership based on the kind of language bias in the communicator’s description of another individual whose political group membership was known, and 2) whether the kind of language bias used by the communicator could influence participants’ intention to be friends with the communicator and whether this intention was qualified by the interaction between participants’ and the target’s political group memberships.
The original study found that 1) regardless of participants’ own political affiliations and the target’s political group membership, favorable language bias, compared to unfavorable langauge bias, in the communicator’s description led participants to infer that the communicator and the target shared a political group membership, and that 2) for the friendship measure, there was a significant interaction between LIB and target’s political affiliation, which was qualified by a three-way interaction between LIB, target political affiliation and participants’ own political affiliation. Subsequent analyses showed that democrat-identified participants were more likely to be friends with the communicator used favorable language bias, compared to unfavorable language bias, to describe a democrat – this pattern did not show for Republican or Independent participants.
Methods
Power Analysis
The original effect size for the key test (three-way interaction for the friendship measure) was 0.24. To achieve 80% power, the planned sample size should be 168; 90% should be 220; 95% should be 268.
Planned Sample
Planned sample size is 168. Termination rule is that the study will finish as soon as it has reached this planned sample size. Sample method would be convenience sample on MTurk.
Materials
“As in Study 1a, participants were asked to read a passage and then respond to questions. In the Republican-target condition, the passage indicated that Peter had voted for John McCain; in the Democratic-target condition, Peter had voted for Barack Obama. In the second part of the passage, participants were again provided with an unknown communicator’s description of Peter’s helpful and rude behaviors. Following Wigboldus et.al. (2000), we included a description of one discrete episode, expressed in the present tense, for each type of behavior (for the full descriptions, see Table S1 in the Supplemental Material available online). For example, the description of helpful behavior in the favorable-LIB condition was written in abstract language and read as follows: “On one occasion, there is a person in a wheelchair who needs assistance getting up a ramp. Peter reaches for the handles of the wheelchair. Peter is helpful.” In the unfavorable-LIB condition, helpful behavior was described concretely: “On one occasion, there is a person in a wheelchair who needs assistance getting up a ramp. Peter reaches for the handles of the wheelchair. Peter pushes the wheelchair up the ramp.” After reading the passage, participants indicated the likely group membership of the communicator on an 8-point scale anchored by 1, definitely a Democrat, and 8, definitely a Republican. They then rated the likelihood that they would be friends with the communicator, using a 5-point scale ranging from 1, it is not at all likely, to 5, it is extremely likely. Finally, participants completed the manipulation-check items and a demographic questionnaire on which they reported their political-party affiliation and political ideology.”
Procedure
It is combined with materials – see above
Analysis Plan
Conduct a two-way ANOVA (LIB x Target’s political affiliation) to examine the manipulation check question – whether favorable LIB (unfavorable LIB) led participants to conclude that the target was more likely to be helpful (rude) in the future and whether this pattern differed depending on the target’s political affiliation.
Conduct a 2 (LIB condition) × 2 (target’s political affiliation) × 3 (participant’s political affiliation: Democrat vs. Republican vs. Independent) analysis of variance (ANOVA) to examine whether participants can infer the communicator’s political group identity based on the kind of language bias and the target’s political affiliation as well as whether participants’ own political affiliation moderated their judgment of the communicator’s political identity. Subsequent analyses (two-way ANOVAs for two-way interactions across the levels of the 3rd variable and pairwise t-tests for simple effects) will be conducted depending on the results of the previous ANOVA to explain the details of the interactions.
Conduct a 2 (LIB condition) × 2 (target’s political affiliation) × 3 (participant’s political affiliation: Democrat vs. Republican vs. Independent) analysis of variance of participants’ ratings of their likelihood of becoming friends with the communicator. Subsequent analyses (two-way ANOVAs for two-way interactions across the levels of the 3rd variable and pairwise t-tests for simple effects) will be conducted depending on the results of the previous ANOVA to explain the details of the interactions.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Key Analysis
Key analysis is the three-way interaction in the ANOVA of the friendship measure in the previous section.
Differences from Original Study
Since we failed to obtain the specific wording of questions and instructions from the authors, we came up with our own specific wording for the questions and instructions. We do not expect the difference to create significant deviation from the original study.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
clean data
d <- read_csv("~/Downloads/Pillot B.csv")## Parsed with column specification:
## cols(
## .default = col_integer(),
## Gender = col_character(),
## Race = col_character(),
## Political = col_character(),
## Liberal = col_character(),
## Conservative = col_character()
## )## See spec(...) for full column specifications.d_tidy = d %>%
mutate(index=rownames(d)) %>%
gather(variable, value, Communicator.F.D:Rude.U.R) %>%
group_by(index) %>%
separate(variable, into=c("variable.n", "LIB", "Party"))
ds = spread(d_tidy, variable.n, value)
ds$Party = as.factor(ds$Party)
ds$Political = as.factor(ds$Political)
ds$LIB = as.factor(ds$LIB)Descriptive Stats
table(ds$Gender)##
## Man Woman
## 44 16table(ds$Race)##
## White
## 16summary(ds$Age)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 23.00 25.00 28.00 31.53 33.00 60.00manipulation check
summary(lm(Helpful~LIB, ds)) #main effect of LIB on estimation of helpfulness##
## Call:
## lm(formula = Helpful ~ LIB, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -42.375 -7.830 1.714 7.714 27.625
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 57.37500 6.04097 9.498 3.26e-07 ***
## LIBU -0.08929 8.84307 -0.010 0.992
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 17.09 on 13 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 7.842e-06, Adjusted R-squared: -0.07691
## F-statistic: 0.0001019 on 1 and 13 DF, p-value: 0.9921summary(lm(Helpful~LIB*Party, ds)) #interaction between LIB and target's political group##
## Call:
## lm(formula = Helpful ~ LIB * Party, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -41.000 -9.333 2.000 8.125 29.000
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 59.667 10.685 5.584 0.000164 ***
## LIBU -2.917 14.135 -0.206 0.840288
## PartyR -3.667 13.515 -0.271 0.791186
## LIBU:PartyR 4.917 19.556 0.251 0.806132
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 18.51 on 11 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.007356, Adjusted R-squared: -0.2634
## F-statistic: 0.02717 on 3 and 11 DF, p-value: 0.9936summary(lm(Rude~LIB, ds)) #main effect of LIB on estimation of rudeness##
## Call:
## lm(formula = Rude ~ LIB, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -35.375 -13.375 7.625 14.214 23.625
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 42.375 7.176 5.905 5.19e-05 ***
## LIBU 16.911 10.504 1.610 0.131
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 20.3 on 13 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.1662, Adjusted R-squared: 0.1021
## F-statistic: 2.592 on 1 and 13 DF, p-value: 0.1314summary(lm(Rude~LIB*Party, ds)) #interaction between LIB and target's political group##
## Call:
## lm(formula = Rude ~ LIB * Party, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -33.600 -13.967 6.667 10.250 25.400
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 45.333 12.513 3.623 0.00401 **
## LIBU 17.917 16.553 1.082 0.30223
## PartyR -4.733 15.827 -0.299 0.77047
## LIBU:PartyR -4.517 22.902 -0.197 0.84725
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 21.67 on 11 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.1956, Adjusted R-squared: -0.02378
## F-statistic: 0.8916 on 3 and 11 DF, p-value: 0.4758Friendship Measure
#main effects
summary(lm(Friend~LIB, ds)) #LIB##
## Call:
## lm(formula = Friend ~ LIB, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.1429 -1.2500 -0.1429 0.8571 2.7500
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.2500 0.5222 4.309 0.000849 ***
## LIBU 0.8929 0.7644 1.168 0.263753
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.477 on 13 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.09498, Adjusted R-squared: 0.02537
## F-statistic: 1.364 on 1 and 13 DF, p-value: 0.2638summary(lm(Friend~Party, ds)) #Target's group membership##
## Call:
## lm(formula = Friend ~ Party, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.8571 -1.5000 0.1429 1.1429 2.5000
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.8571 0.5823 4.907 0.000287 ***
## PartyR -0.3571 0.7974 -0.448 0.661596
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.541 on 13 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.0152, Adjusted R-squared: -0.06056
## F-statistic: 0.2006 on 1 and 13 DF, p-value: 0.6616summary(lm(Friend~Political, ds)) #Participants' own group membership##
## Call:
## lm(formula = Friend ~ Political, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.7273 -1.6970 0.2727 1.2727 2.2727
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.72727 0.48343 5.642 0.000109 ***
## PoliticalIndependent -0.72727 1.67464 -0.434 0.671786
## PoliticalRepublican -0.06061 1.04432 -0.058 0.954677
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.603 on 12 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.01547, Adjusted R-squared: -0.1486
## F-statistic: 0.0943 on 2 and 12 DF, p-value: 0.9107#Contrast 1 -- When Favorable LIB, Democrat (communicator), and Democrat (participant) are the defaults
summary(lm(Friend~LIB*Party*Political, ds))##
## Call:
## lm(formula = Friend ~ LIB * Party * Political, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.0000 -0.8000 0.0000 0.7667 2.2000
##
## Coefficients: (5 not defined because of singularities)
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.000e+00 1.172e+00 2.560 0.0336 *
## LIBU 1.000e+00 2.029e+00 0.493 0.6354
## PartyR -1.200e+00 1.386e+00 -0.866 0.4119
## PoliticalIndependent -2.000e+00 2.343e+00 -0.853 0.4182
## PoliticalRepublican -4.765e-15 2.029e+00 0.000 1.0000
## LIBU:PartyR 8.667e-01 2.363e+00 0.367 0.7233
## LIBU:PoliticalIndependent NA NA NA NA
## LIBU:PoliticalRepublican -1.500e+00 2.870e+00 -0.523 0.6154
## PartyR:PoliticalIndependent NA NA NA NA
## PartyR:PoliticalRepublican NA NA NA NA
## LIBU:PartyR:PoliticalIndependent NA NA NA NA
## LIBU:PartyR:PoliticalRepublican NA NA NA NA
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.657 on 8 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.2989, Adjusted R-squared: -0.2269
## F-statistic: 0.5685 on 6 and 8 DF, p-value: 0.7463summary(lm(Friend~LIB*Party, ds))##
## Call:
## lm(formula = Friend ~ LIB * Party, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.0000 -0.8000 -0.6667 1.2500 2.2000
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.0000 0.8566 3.502 0.00495 **
## LIBU -0.2500 1.1332 -0.221 0.82944
## PartyR -1.2000 1.0836 -1.107 0.29173
## LIBU:PartyR 2.1167 1.5679 1.350 0.20414
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.484 on 11 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.2271, Adjusted R-squared: 0.01634
## F-statistic: 1.078 on 3 and 11 DF, p-value: 0.3985summary(lm(Friend~Party*Political, ds))##
## Call:
## lm(formula = Friend ~ Party * Political, data = ds)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.3333 -1.5000 0.3333 1.0000 2.5000
##
## Coefficients: (2 not defined because of singularities)
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.3333 0.9428 3.536 0.00467 **
## PartyR -0.8333 1.1055 -0.754 0.46681
## PoliticalIndependent -1.3333 1.8856 -0.707 0.49421
## PoliticalRepublican -0.6667 1.3333 -0.500 0.62693
## PartyR:PoliticalIndependent NA NA NA NA
## PartyR:PoliticalRepublican NA NA NA NA
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.633 on 11 degrees of freedom
## (45 observations deleted due to missingness)
## Multiple R-squared: 0.06383, Adjusted R-squared: -0.1915
## F-statistic: 0.25 on 3 and 11 DF, p-value: 0.8597#The exact contrasts to test simple effects will be detailed after all data have been collected and the significant interactions have been known.Confirmatory analysis
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
#set up error bar function
sem <- function(x) {sd(x, na.rm=TRUE) / sqrt(length(x))}
ci95 <- function(x) {sem(x) * 1.96}
#social inference graph
agg <- ds %>%
group_by(Party,LIB, Political) %>%
summarise(mean = mean(Communicator,na.rm=T),
se = sem(Communicator),
upper = mean + se,
lower = mean - se)
plot1 = ggplot(agg,aes(x=Party, y=mean, fill=LIB)) +
geom_bar(position="dodge", stat="identity") +
geom_errorbar( aes( ymax=upper, ymin=lower ) ,
width =.25,
linetype="solid", position=position_dodge(.9)
) +
theme_bw() +
facet_grid(. ~Political)
plot1## Warning: Removed 5 rows containing missing values (geom_bar).## Warning: Removed 8 rows containing missing values (geom_errorbar).
#faceted by participants' own group membership
#friendship graph
agg <- ds %>%
group_by(Party,LIB, Political) %>%
summarise(mean = mean(Friend,na.rm=T),
se = sem(Friend),
upper = mean + se,
lower = mean - se)
plot2 = ggplot(agg,aes(x=Party, y=mean, fill=LIB)) +
geom_bar(position="dodge", stat="identity") +
geom_errorbar(aes( ymax=upper, ymin=lower ), width =.25,
linetype="solid", position=position_dodge(.9)) +
theme_bw() +
facet_grid(. ~Political)
plot2 ## Warning: Removed 5 rows containing missing values (geom_bar).
## Warning: Removed 8 rows containing missing values (geom_errorbar).
#faceted by participants' own group membershipExploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.
Social Identity Measure