Replication of Study 1 by Porter, Rheinschmidt-Same, & Richeson (2016, Psychological Science)
Melissa Mesinas (mmesinas@stanford.edu)
December 5, 2018
Introduction
One of my research interests entails the language use multilingual speaking communities use to communicate through their intergenerational teaching. I use ethnographic methods to collect my data, however, I find that the proposed study for replication brings a new perspective I can include in my research. The participants I conduct my research with are multilingual and speak a number of combinations that include Zapotec (an Indigenous language native to Mexico), Spanish, and English. One of my research interests includes the language preference my participants use among peers their age and adults who are experienced members in the culture-specific learning setting. I was intrigued by the findings of this article because the researchers examine how implicit messages vary in the degree to which the communicator’s language is concrete rather than abstract. Hence, research has demonstrated that subtle linguistic variations can have a substantial effect on what is conveyed about a target (Porter, Rheinschmidt-Same, & Richeson, 2016). As I design my dissertation, this study has helped me realize the importance of potential favorable or unfavorable LIB (defined in the next section) within the intragroup setting of my study.
The procedures of the study required participants to complete all tasks on a computer as they were asked to read a passage and then respond to questions. The beginning of the passage was the same across all conditions: “Imagine that someone is communicating with you about a man named Peter. Peter is American, has an interest in politics, and voted for Barack Obama.” This information was intended to subtly imply that Peter (the target) was a Democrat. The stimuli are introduced in the second part of the passage as participants were given the communicator’s description of Peter’s helpful and rude behaviors. In the favorable linguistic intergroup bias (favorable-LIB) condition, which is the use of abstract language to describe in-group members’ desirable behaviors and concrete language to describe their undesirable behaviors, Peter’s helping behavior was described abstractly (such as “Peter is someone who stands up for the interests of others”). In the unfavorable LIB condition, Peter’s helping behavior is described more concretely (such as “Peter helped another person, even if it did not benefit him”) and his rude behavior was described more abstractly (such as “Peter is cold.”).
Participants were asked to assess the likelihood that the communicator was a Republican or Democrat after reading the passage. Participants assessed their ratings on a 7-point scale, anchored by 1, definitely a Republican, and 7, definitely a Democrat. Next, in order to check the effectiveness of the LIB manipulation, the authors asked participants to estimate the percentage of future situation in which Peter was likely to be helpful, along with the percentage of future situations in which he was likely to be rude. To conclude, participants completed a demographic questionnaire that asked about their gender, political-party affiliation, and the degree to which they endorsed liberal and conservative beliefs, on a 7-point scale ranging from 1, strongly disagree, to 7, strongly agree. I can foresee finding a sample size to the original study (n = 88) that is similar to be challenging. Additionally, given that the complete materials were not provided in the article, the length of the entire procedure is unknown and may be a challenge as well.
Link to the repository: https://github.com/mmesinas/porter2016
Original link to paper: http://journals.sagepub.com.stanford.idm.oclc.org/doi/pdf/10.1177/0956797615612202
Methods
Power Analysis
An original effect size was not mentioned nor a power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size was discussed in the study, please see the next section for more details.
Planned Sample
From past research in this area, the authors conservatively estimated the sample size needed to find an effect. They concluded that they needed to sample 90 participants and stopped collecting data once that number was reached. However, two participants did not complete the dependent-variable measure and the final sample size was 88. Participants were recruited from Amazon Mechanical Turk (MTurk.com).
Materials
“Participants completed all tasks on a computer. They were asked to read a passage and then respond to questions. The beginning of the passage was the same for all participants: “Imagine that someone is communicating with you about a man named Peter. Peter is American, has an interest in politics, and voted for Barack Obama.” This information was intended to subtly imply that Peter (the target) was a Democrat. In the second part of the passage, participants were provided with the communicator’s description of Peter’s helpful and rude behaviors (for the complete text of these descriptions, see Table S1 in the Supplemental Material available online). In the favorable-LIB condition, Peter’s helping behavior was described abstractly (e.g., “[Peter] is someone who stands up for the interests of others”), and his rude behavior was described concretely (e.g., “Peter said something rude to another person recently”). In the unfavorable-LIB condition, Peter’s helping behavior was described concretely (e.g., “Peter helped another person, even when it did not benefit him”), and his rude behavior was described abstractly (e.g., “[Peter] is cold and unfriendly”).”
The Supplemental Materials can be found here: http://journals.sagepub.com.stanford.idm.oclc.org/doi/suppl/10.1177/0956797615612202/suppl_file/suppl-material.pdf
Procedure
“After reading the passage, participants were asked to assess the likelihood that the communicator was either a Democrat or a Republican. Ratings were made on a 7-point scale, anchored by 1, definitely a Republican, and 7, definitely a Democrat. As a check of the effectiveness of the LIB manipulation, we asked participants to estimate the percentage of future situations in which Peter was likely to be helpful and the percentage of future situations in which he was likely to be rude (Semin & de Poot, 1997). Finally, participants completed a demographic questionnaire that asked their gender, their political-party affiliation, and the degree to which they endorsed liberal and conservative beliefs (on 7-point scales ranging from 1, strongly disagree, to 7, strongly agree).”
The survey can be accesed via this link: https://gse.qualtrics.com/jfe/form/SV_01fOoRD10ONZoAR
Analysis Plan
The analysis plan will follow the original study’s plan. I will conduct the same manipulation check that was conducted in study 1a through the calculation of raw mean ratings of Peter’s (the target) future likelihood of being rude and helpful. Further, the mean rated likelihood that the communicator and target shared group membership for each condition (favorable- and unfavorable-linguistic intergroup bias).
I will calculate t-statistics, effect sizes, and p-values for these analyses. A t-test will be used because I want to compare the means of two independent groups.
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect. To be determined.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection. To be determined.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan To be determined.
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”. To be determined.
Results
Data preparation
I will import the data for study 1a to familiarize myself with the organization of the dataset. I will make sure to understand how each variable was labeled and inputted, especially for the LIB and ULIB conditions.
####Data Preparation
####Load Relevant Libraries and Functions
library(tidyverse)## ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──## ✔ ggplot2 3.1.0 ✔ purrr 0.2.5
## ✔ tibble 1.4.2 ✔ dplyr 0.7.7
## ✔ tidyr 0.8.2 ✔ stringr 1.3.1
## ✔ readr 1.1.1 ✔ forcats 0.3.0## ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()####Import data
setwd("/Users/melissamesinas/Desktop/PSYCH_251")
getwd()## [1] "/Users/melissamesinas/Desktop/PSYCH_251"raw_pilotB <- read_csv('data/Porteretal2016_pilotB.csv')## Parsed with column specification:
## cols(
## FLIB_DV = col_integer(),
## ULIB_DV = col_integer(),
## future_help = col_integer(),
## future_rude = col_integer(),
## gender = col_integer(),
## polid = col_integer(),
## conservative = col_integer(),
## liberal = col_integer(),
## social_issues = col_integer(),
## mTurkCode = col_integer(),
## Condition = col_integer(),
## DV = col_integer()
## )#### Data exclusion / filtering
#### Prepare data for analysis - create columns etc.
pilotB = raw_pilotB %>%
mutate(Condition=ifelse(is.na(FLIB_DV), "ULIB", "FLIB"),
DV=ifelse(is.na(FLIB_DV), ULIB_DV, FLIB_DV))
head(pilotB)## # A tibble: 6 x 12
## FLIB_DV ULIB_DV future_help future_rude gender polid conservative liberal
## <int> <int> <int> <int> <int> <int> <int> <int>
## 1 NA 5 30 60 2 3 2 6
## 2 NA 4 50 50 2 1 5 5
## 3 3 NA 60 40 2 3 2 3
## 4 NA 3 25 75 2 1 3 6
## 5 6 NA 95 5 2 1 2 5
## 6 NA 4 80 20 2 3 2 5
## # ... with 4 more variables: social_issues <int>, mTurkCode <int>,
## # Condition <chr>, DV <int>Confirmatory analysis
“The primary dependent measure was participants’ inferences regarding the communicator’s political affiliation. As predicted, participants in the favorable-LIB condition were significantly more likely to believe that the communicator was a Democrat, and thus shared a party affiliation with the target, than were participants in the unfavorable-LIB condition, t(86) = 2.89, p = .005, d = 0.62 (Fig. 1). This difference was not moderated by participants’ self-reported political- party affiliation or ideological endorsement (ps > .18). Our findings suggested initial support for our hypothesis that individuals can infer a communicator’s social identity from his or her language, regardless of their own social identity.”
The following code was used to test pilot B:
#### load libraries
library(tidyverse)
#### imprt data
setwd("/Users/melissamesinas/Desktop/PSYCH_251/data")
pilotB<-read_csv('Porteretal2016_pilotB.csv')## Parsed with column specification:
## cols(
## FLIB_DV = col_integer(),
## ULIB_DV = col_integer(),
## future_help = col_integer(),
## future_rude = col_integer(),
## gender = col_integer(),
## polid = col_integer(),
## conservative = col_integer(),
## liberal = col_integer(),
## social_issues = col_integer(),
## mTurkCode = col_integer(),
## Condition = col_integer(),
## DV = col_integer()
## )head(pilotB)## # A tibble: 6 x 12
## FLIB_DV ULIB_DV future_help future_rude gender polid conservative liberal
## <int> <int> <int> <int> <int> <int> <int> <int>
## 1 NA 5 30 60 2 3 2 6
## 2 NA 4 50 50 2 1 5 5
## 3 3 NA 60 40 2 3 2 3
## 4 NA 3 25 75 2 1 3 6
## 5 6 NA 95 5 2 1 2 5
## 6 NA 4 80 20 2 3 2 5
## # ... with 4 more variables: social_issues <int>, mTurkCode <int>,
## # Condition <int>, DV <int>#### analysis
t.test(DV ~ Condition, data = pilotB)##
## Welch Two Sample t-test
##
## data: DV by Condition
## t = 1.0729, df = 2.281, p-value = 0.3836
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -3.429553 6.096220
## sample estimates:
## mean in group 1 mean in group 2
## 5.333333 4.000000#code to calculate means grouped by condition
summarized_pilotB <- pilotB%>%
group_by(Condition) %>%
summarize(mean = mean(DV),
n = n(),
se = sd(DV) / sqrt(n)) %>%
mutate(lower.ci.DV = mean - qt(1 - (0.05 / 2), n - 1) * se,
upper.ci.DV = mean + qt(1 - (0.05 / 2), n - 1) * se)
#bar graph
g <- ggplot(summarized_pilotB, aes(x=Condition, y=mean))
g + geom_bar(stat="identity") +
geom_errorbar(aes(ymin=lower.ci.DV, ymax=upper.ci.DV)) +
scale_x_discrete(breaks=c("FLIB", "ULIB"), labels=c("Favorable", "Unfavorable"))
The results from the pilot B data are t(2.28) = 1.07, p = 0.38
Below is the original graph from Study 1A:
Graph
Exploratory analyses
#code for manipulation check - helpful
t.test(future_help ~ Condition, data = pilotB)##
## Welch Two Sample t-test
##
## data: future_help by Condition
## t = 1.9449, df = 5.4541, p-value = 0.1046
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -9.450131 74.783465
## sample estimates:
## mean in group 1 mean in group 2
## 71.66667 39.00000manipulation_check_help_pilotB <- pilotB%>%
group_by(Condition) %>%
summarize(mean = mean(future_help),
n = n(),
se = sd(future_help) / sqrt(n)) %>%
mutate(lower.ci.future_help = mean - qt(1 - (0.05 / 2), n - 1) * se,
upper.ci.future_help = mean + qt(1 - (0.05 / 2), n - 1) * se)#code for manipulation check - rude
t.test(future_rude ~ Condition, data = pilotB)##
## Welch Two Sample t-test
##
## data: future_rude by Condition
## t = -1.8126, df = 4.9996, p-value = 0.1296
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -69.32115 11.98781
## sample estimates:
## mean in group 1 mean in group 2
## 28.33333 57.00000manipulation_check_rude_pilotB <- pilotB%>%
group_by(Condition) %>%
summarize(mean = mean(future_rude),
n = n(),
se = sd(future_rude) / sqrt(n)) %>%
mutate(lower.ci.future_rude = mean - qt(1 - (0.05 / 2), n - 1) * se,
upper.ci.future_rude = mean + qt(1 - (0.05 / 2), n - 1) * se)As expected, participants in the favorable-LIB condition believed that Peter was more likely to be helpful in the future (M = 71.67%, SD =11.67%) than did participants in the unfavorable-LIB condition (M = 39%, SD = 12.08%), t(5.45) = 1.94, p = 0.10. Similarly, participants in the favorable-LIB condition indicated that Peter was less likely to be rude in the future (M =28.33%, SD = 11.67%) compared with the participants in the unfavorable-LIB condition(M = 57%, SD = 10.68%), t(5) = -1.81, p = 0.12.
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.