Introduction

One of my research interests entails the language use multilingual speaking communities use to communicate through their intergenerational teaching. I use ethnographic methods to collect my data, however, I find that the proposed study for replication brings a new perspective I can include in my research. The participants I conduct my research with are multilingual and speak a number of combinations that include Zapotec (an Indigenous language native to Mexico), Spanish, and English. One of my research interests includes the language preference my participants use among peers their age and adults who are experienced members in the culture-specific learning setting. I was intrigued by the findings of this article because the researchers examine how implicit messages vary in the degree to which the communicator’s language is concrete rather than abstract. Hence, research has demonstrated that subtle linguistic variations can have a substantial effect on what is conveyed about a target (Porter, Rheinschmidt-Same, & Richeson, 2016). As I design my dissertation, this study has helped me realize the importance of potential favorable or unfavorable LIB (defined in the next section) within the intragroup setting of my study.

The procedures of the study required participants to complete all tasks on a computer as they were asked to read a passage and then respond to questions. The beginning of the passage was the same across all conditions: “Imagine that someone is communicating with you about a man named Peter. Peter is American, has an interest in politics, and voted for Barack Obama.” This information was intended to subtly imply that Peter (the target) was a Democrat. The stimuli are introduced in the second part of the passage as participants were given the communicator’s description of Peter’s helpful and rude behaviors. In the favorable linguistic intergroup bias (favorable-LIB) condition, which is the use of abstract language to describe in-group members’ desirable behaviors and concrete language to describe their undesirable behaviors, Peter’s helping behavior was described abstractly (such as “Peter is someone who stands up for the interests of others”). In the unfavorable LIB condition, Peter’s helping behavior is described more concretely (such as “Peter helped another person, even if it did not benefit him”) and his rude behavior was described more abstractly (such as “Peter is cold.”).

Participants were asked to assess the likelihood that the communicator was a Republican or Democrat after reading the passage. Participants assessed their ratings on a 7-point scale, anchored by 1, definitely a Republican, and 7, definitely a Democrat. Next, in order to check the effectiveness of the LIB manipulation, the authors asked participants to estimate the percentage of future situation in which Peter was likely to be helpful, along with the percentage of future situations in which he was likely to be rude. To conclude, participants completed a demographic questionnaire that asked about their gender, political-party affiliation, and the degree to which they endorsed liberal and conservative beliefs, on a 7-point scale ranging from 1, strongly disagree, to 7, strongly agree. I can foresee finding a sample size to the original study (n = 88) that is similar to be challenging. Additionally, given that the complete materials were not provided in the article, the length of the entire procedure is unknown and may be a challenge as well.

Link to the repository: https://github.com/psych251/porter2016

Original link to paper: http://journals.sagepub.com.stanford.idm.oclc.org/doi/pdf/10.1177/0956797615612202

Preregistration link: https://osf.io/er735/

Methods

Power Analysis

Considerations of feasibility for selecting planned sample size was discussed in the study, please see the next section for more details.

Using the G*Power calculator, I used the statistics provided by the authors to calculate power (t(86) = 2.89, p = .005, d = 0.62). The calculator estimated that 66 participants would be needed to run a power analysis of 80%. However, given the short duration of the survey, this replication ran the same amount of participants as in the original study, 88.

Planned Sample

From past research in this area, the authors conservatively estimated the sample size needed to find an effect. They concluded that they needed to sample 90 participants and stopped collecting data once that number was reached. However, two participants did not complete the dependent-variable measure and the final sample size was 88. Participants were recruited from Amazon Mechanical Turk (MTurk.com). They did not note the specific power analysis in their estimation of 90 participants.

Materials

“Participants completed all tasks on a computer. They were asked to read a passage and then respond to questions. The beginning of the passage was the same for all participants: “Imagine that someone is communicating with you about a man named Peter. Peter is American, has an interest in politics, and voted for Barack Obama.” This information was intended to subtly imply that Peter (the target) was a Democrat. In the second part of the passage, participants were provided with the communicator’s description of Peter’s helpful and rude behaviors (for the complete text of these descriptions, see Table S1 in the Supplemental Material available online). In the favorable-LIB condition, Peter’s helping behavior was described abstractly (e.g., “[Peter] is someone who stands up for the interests of others”), and his rude behavior was described concretely (e.g., “Peter said something rude to another person recently”). In the unfavorable-LIB condition, Peter’s helping behavior was described concretely (e.g., “Peter helped another person, even when it did not benefit him”), and his rude behavior was described abstractly (e.g., “[Peter] is cold and unfriendly”).”

The Supplemental Materials can be found here: http://journals.sagepub.com.stanford.idm.oclc.org/doi/suppl/10.1177/0956797615612202/suppl_file/suppl-material.pdf

Procedure

“After reading the passage, participants were asked to assess the likelihood that the communicator was either a Democrat or a Republican. Ratings were made on a 7-point scale, anchored by 1, definitely a Republican, and 7, definitely a Democrat. As a check of the effectiveness of the LIB manipulation, we asked participants to estimate the percentage of future situations in which Peter was likely to be helpful and the percentage of future situations in which he was likely to be rude (Semin & de Poot, 1997). Finally, participants completed a demographic questionnaire that asked their gender, their political-party affiliation, and the degree to which they endorsed liberal and conservative beliefs (on 7-point scales ranging from 1, strongly disagree, to 7, strongly agree).”

The survey can be accesed via this link: https://gse.qualtrics.com/jfe/form/SV_01fOoRD10ONZoAR

Analysis Plan

The analysis plan will follow the original study’s plan. A t-test will be run on the rated likelihood that the communicator and target shared group membership for each condition (favorable- and unfavorable-linguistic intergroup bias). I will conduct the same manipulation check that was conducted in study 1a through the calculation of raw mean ratings of Peter’s (the target) future likelihood of being rude and helpful by running t-tests as well.

I will calculate t-statistics, effect sizes, and p-values for these analyses.

Differences from Original Study

The only difference in this study was the distribution of the replication sample size did not match the sample size of the original study. For instance, there were more women and less Democrats in the original study. These differences were not anticipated.

Methods Addendum (Post Data Collection)

Actual Sample

The sample size of the replication study was 87.There were 32 women and 57 men who participated in this study. Participants identified in the following political party affiliation: 48% Democrat, 24% Republican, 26% Independent, and 2% Other political party. I excluded two participants because they did not provide numeric answers to the manipulation checks.

Differences from pre-data collection methods plan

None.

Results

For the confirmatory analysis, participants in the favorable-LIB condition were not significantly more likely to believe that the communicator was a Democrat, and thus shared a party affiliation with the target, than were participants in the unfavorable-LIB condition, t(82.92) = 0.65, p = .52, d = 0.14. The mean for the favorable-LIB condition was 5.10 and the mean for the unfavorable-LIB condition was 4.89. These findings contradict our prediction.

Data preparation

I will import the data for study 1a to familiarize myself with the organization of the dataset. I will make sure to understand how each variable was labeled and inputted, especially for the LIB and ULIB conditions.

####Load Relevant Libraries and Functions
library(tidyverse)
## ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
## ✔ ggplot2 3.1.0     ✔ purrr   0.2.5
## ✔ tibble  1.4.2     ✔ dplyr   0.7.7
## ✔ tidyr   0.8.2     ✔ stringr 1.3.1
## ✔ readr   1.1.1     ✔ forcats 0.3.0
## ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
library(lsr)

####Data Preparation
setwd("/Users/melissamesinas/Desktop/PSYCH_251")
raw_final <- read_csv('Porteretal2016_final.csv')
## Parsed with column specification:
## cols(
##   FLIB_DV = col_integer(),
##   ULIB_DV = col_integer(),
##   future_help = col_integer(),
##   future_rude = col_integer(),
##   gender = col_integer(),
##   polid = col_integer(),
##   conservative = col_integer(),
##   liberal = col_integer(),
##   social_issues = col_integer(),
##   mTurkCode = col_integer()
## )
#### Prepare data for analysis - create columns etc.
final = raw_final %>%
  mutate(Condition=ifelse(is.na(FLIB_DV), "ULIB", "FLIB"),
         DV=ifelse(is.na(FLIB_DV), ULIB_DV, FLIB_DV))

head(final)
## # A tibble: 6 x 12
##   FLIB_DV ULIB_DV future_help future_rude gender polid conservative liberal
##     <int>   <int>       <int>       <int>  <int> <int>        <int>   <int>
## 1      NA       5          42          53      2     3            4       5
## 2      NA       7          50          50      2     1            6       6
## 3       5      NA          85           5      2     1            2       6
## 4       7      NA          50          50      1     2            4       3
## 5      NA       2          40          60      1     1            1       7
## 6       6      NA          85          15      1     1            4       5
## # ... with 4 more variables: social_issues <int>, mTurkCode <int>,
## #   Condition <chr>, DV <int>

Confirmatory analysis

From the original study: “The primary dependent measure was participants’ inferences regarding the communicator’s political affiliation. As predicted, participants in the favorable-LIB condition were significantly more likely to believe that the communicator was a Democrat, and thus shared a party affiliation with the target, than were participants in the unfavorable-LIB condition, t(86) = 2.89, p = .005, d = 0.62 (Fig. 1). This difference was not moderated by participants’ self-reported political- party affiliation or ideological endorsement (ps > .18). Our findings suggested initial support for our hypothesis that individuals can infer a communicator’s social identity from his or her language, regardless of their own social identity.”

A t-test will be used to assess if participants thought the communicator was more likely to be a Democrat in the favorable LIB condition than the unfavorable LIB condition.

#### analysis
t.test(DV ~ Condition, data = final)
## 
##  Welch Two Sample t-test
## 
## data:  DV by Condition
## t = 0.65172, df = 82.924, p-value = 0.5164
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.4234081  0.8361066
## sample estimates:
## mean in group FLIB mean in group ULIB 
##           5.095238           4.888889
# code to calculate means grouped by condition 
summarized_final <- final%>%
 group_by(Condition) %>%
  summarize(mean = mean(DV), 
            n = n(), 
            se = sd(DV) / sqrt(n)) %>%
  mutate(lower.ci.DV = mean - qt(1 - (0.05 / 2), n - 1) * se,
      upper.ci.DV = mean + qt(1 - (0.05 / 2), n - 1) * se)

#### effect size 
cohensD(DV ~ Condition, 
       data = final)
## [1] 0.1387267
#bar graph
g <- ggplot(summarized_final, aes(x=Condition, y=mean)) 
g + geom_bar(stat="identity") +
  geom_errorbar(aes(ymin=lower.ci.DV, ymax=upper.ci.DV)) +
  scale_x_discrete(breaks=c("FLIB", "ULIB"), labels=c("Favorable", "Unfavorable"))

Participants in the favorable-LIB condition were not significantly more likely to believe that the communicator was a Democrat, and thus shared a party affiliation with the target, than were participants in the unfavorable-LIB condition, t(82.92) = 0.65, p = .52, d = 0.14. These findings do not support our prediction.

Below is the original graph from Study 1A:

Graph

Graph

Exploratory analyses

From the original study: “As expected, participants in the favorable-LIB condition believed that Peter was more likely to be helpful in the future (M = 70.29%, SD = 23.58) than did participants in the unfavorable-LIB condition (M = 57.83%, SD = 24.08), t(86) = 2.45, p = .016, d = 0.53. Similarly, participants in the favorable-LIB condition indicated that Peter was less likely to be rude in the future (M = 33.67%, SD = 25.48) compared with participants in the unfavorable-LIB condition (M = 53.93%, SD = 25.22), t(86) = 3.73, p < .001, d = 0.80.”

A t-test will be used to assess if participants in the favorable LIB-condition thought Peter was more likely to be helpful than participants in the unfavorable LIB condition. A second t-test will be analyzed to see if participants in the favorable LIB-condition thought Peter was less likely to be rude than participants in the unfavorable LIB condition

#code for manipulation check - helpful
t.test(future_help ~ Condition, data = final)
## 
##  Welch Two Sample t-test
## 
## data:  future_help by Condition
## t = 3.497, df = 80.337, p-value = 0.0007695
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##   7.066434 25.727217
## sample estimates:
## mean in group FLIB mean in group ULIB 
##           62.28571           45.88889
#code to calculate means grouped by condition 
manipulation_check_help_final <- final%>%
 group_by(Condition) %>%
  summarize(mean = mean(future_help), 
            n = n(), 
            se = sd(future_help) / sqrt(n)) %>%
  mutate(lower.ci.future_help = mean - qt(1 - (0.05 / 2), n - 1) * se,
      upper.ci.future_help = mean + qt(1 - (0.05 / 2), n - 1) * se)

#### effect size 
cohensD(future_help ~ Condition, 
       data = final)
## [1] 0.7548096
#code for manipulation check - rude 
t.test(future_rude ~ Condition, data = final)
## 
##  Welch Two Sample t-test
## 
## data:  future_rude by Condition
## t = -4.5643, df = 82.249, p-value = 1.736e-05
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -28.56373 -11.22358
## sample estimates:
## mean in group FLIB mean in group ULIB 
##           32.59524           52.48889
#code to calculate means grouped by condition 
manipulation_check_rude_final <- final%>%
 group_by(Condition) %>%
  summarize(mean = mean(future_rude), 
            n = n(), 
            se = sd(future_rude) / sqrt(n)) %>%
  mutate(lower.ci.future_rude = mean - qt(1 - (0.05 / 2), n - 1) * se,
      upper.ci.future_rude = mean + qt(1 - (0.05 / 2), n - 1) * se)

#### effect size 
cohensD(future_rude ~ Condition, 
       data = final)
## [1] 0.9831637

As expected, participants in the favorable-LIB condition believed that Peter was more likely to be helpful in the future (M = 62.629%, SD = 2.96%) than did participants in the unfavorable-LIB condition (M = 45.89%, SD = 3.65%), t(80.34) = 3.50, p < .001, d = 0.75 Similarly, participants in the favorable-LIB condition indicated that Peter was less likely to be rude in the future (M = 32.60%, SD = 3.30%) compared with the participants in the unfavorable-LIB condition (M = 52.49%, SD = 2.85%), t(82.25) = -4.56, p = 1.736e-05, d = 0.98.

Discussion

Summary of Replication Attempt

For the confirmatory analysis, participants in the favorable-LIB condition were not significantly more likely to believe that the communicator was a Democrat, and thus shared a party affiliation with the target, than were participants in the unfavorable-LIB condition, t(82.92) = 0.65, p = .52, d = 0.14. The mean for the favorable-LIB condition was 5.10 and the mean for the unfavorable-LIB condition was 4.89. These findings contradict our prediction as the p-value was less than .05 and the effect size was minimal at 0.14. Hence, this analysis failed to replicate the original result.

Commentary

The exploratory analyses indicate that the manipulation checks did result as predicted, indicating that participants would view a target more favorably when favorable linguistic intergroup bias is used to communicat about them. From these results, I infer that there is reason to believe that linguistic intergroup bias can influence someone how they perceive others. However, the confirmatory analysis did not result in signifiance and failed to replicate the original findings. As mentioned earlier in the report, the sample sizes differed in its distribution of demographics, which could lead to a plausible moderator. Additionally, the change in political climate is worth considerating as a moderator as well.