Replication of Gollwitzer et al., 2017 by Mechanical Turk (2017, Nature Human Behavior)
Julie Cachia (jcachia@stanford.edu)
April 15, 2019
Introduction
Justification for choice of experiment:
This study looks at how aversion to non-social pattern deviancy predicts aversion towards stigmatized individuals. This paper is related to my interests because part of the cross-cultural work I will be doing in my lab will investigate how we react to people who don’t match our affective preferences (ideal affect).
Description of paper:
Across 8 studies, this paper (Gollwitzer, Marshall, Wang & Bargh, 2017) showed that aversion to non-social deviancy (in patterns and shapes) predicted aversion to social deviancy (stigmatized individuals, norm breakers, negative and positive deviants, and minority groups).
I aim to replicate Study 1 for this project. Specifically, I will conduct a linear regression to test whether aversion to pattern deviancy significantly predicts aversion to social deviancy.
Description of the stimuli and procedures that will be required to conduct this experiment:
Participants will be recruited via Amazon’s Mechanical Turk.
The measures:
-Political orientation (3 items)
-Need for closure (15 items)
-Neuroticism (2 items)
-Pattern deviancy (15 items)
-Social deviancy (15 items)
-Attention check (2 items)
-Demographics: income (1 item); education (1 item)
Anticipated challenges:
-Ensuring I have the intended sample size even after excluding participants who failed the attention check
Links:
Project Repository: https://github.com/JYAC/gollwitzer2017
Original Paper: https://github.com/JYAC/gollwitzer2017/tree/master/original_paper
Qualtrics Survey: https://stanforduniversity.qualtrics.com/jfe/form/SV_1SlT9mNwFnIOpEh
References:
Gollwitzer, A., Marshall, J., Wang, Y., & Bargh, J. A. (2017). Relating pattern deviancy aversion to stigma and prejudice. Nature Human Behaviour, 1(12), 920.
Methods
Power Analysis
The original effect size was 0.38 (standardized beta coefficient). Power analysis revealed that in order to detect this effect size, we need the following sample sizes for each level of power:
to achieve 80% power, we need N = 49;
to achieve 90% power, we need N = 65;
to achieve 95% power, we need N = 79.
The full survey takes about 5 minutes (4 minutes when I tried) to complete. Therefore, in order to be consistent with California minimum wage ($7.25/hour), each participant should get paid $0.60 per survey. In order to stay within the course budget of $50, we will choose 80% power.
The original paper reports a 9% exclusion rate (for failing the attention check), so assuming the same exclusion rate, we will need 54 participants:
[N - (0.09N)] = 49
N = 53.84 (54 participants required)
(54 participants x $0.60) + (54 participants * $0.25 (MTurk Fee)) = $45.68
Planned Sample
Planned sample size will be 54 participants (accounting for 9% exclusion rate), based on the power analysis conducted above.
Materials
Quoted from the original article:
“Need for closure. We included a validated short version of the need-for-closure scale26. The scale consisted of 15 items (for example, ‘I don’t like situations that are uncertain’).
Neuroticism. We included a validated two-item neuroticism scale: ‘I see myself as someone who… is relaxed, handles stress well’ (reverse-coded) and ‘I see myself as someone who… gets nervous easily’27,54.
Political orientation. Participants answered the following questions on a Likert scale ranging from 1 (extremely liberal) to 9 (extremely conservative): ‘In terms of economic issues, how liberal or conservative are you?’, ‘In terms of social and cultural issues, how liberal or conservative are you?’ and ‘Where on the following scale of political orientation would you place yourself?’
Pattern deviancy. We assessed participants’ aversion in response to five images depicting non-social pattern deviancy (see Supplementary Methods for images). These images were taken from popular Buzzfeed.com articles presenting images depicting non-social pattern deviancy. For example, one of these articles was named: ‘31 things that will make any neat freak’s eye twitch’ (https://www.buzzfeed.com/erinchack/things-that-will-make-any-neat-freaks-eye-twitch?utm_term= .gjMkaOGJR#.xbwoO1jQY). Underneath each image, a prompt read ‘The above image makes me…’ followed by three statements assessing participants’ discomfort, anxiousness, and annoyance in response to the image (‘feel uncomfortable’, ‘feel anxious’, and ‘feel annoyed’). Participants answered on a Likert scale from 1 (not at all agree) to 7 (strongly agree). The images were presented in random order.
Social deviancy. Social deviancy aversion was measured identically to pattern deviancy aversion except that the images depicted deviant individuals (see Supplementary Methods for images). These images were validated as depicting socially deviant individuals in Supplementary Study 2.
Attention check items. We included a direct (‘I was focused while filling out this survey’) and indirect attention check item (‘People vary in the amount they pay attention to these kinds of surveys… if you have read this question carefully, please write the word yes in the blank box below labeled other.’; see Supplementary Note for details). The same two attention check items were used as exclusion criteria in studies 2–6."
The exact scales described above were used in this replication.
Procedure
Quoted from the original article:
“Before beginning the study, participants gave informed consent (the same is true of all reported studies). Participants then completed the need for closure, neuroticism, and political orientation measures in random order. Thereafter, they completed the pattern and social deviancy measures in random order. Lastly, participants completed the attention check items and demographics (the same is true of studies 2–6). No consistent demographic effects were found across the reported studies (see Supplementary Note for details).”
The exact procedure outlined above was followed.
Analysis Plan
First, participants who failed the attention check were excluded, defined as follows in the supplementary materials:
“Our first attention check item assessed whether participants were focused (“I was focused while filling out this survey”). Likert-scale: 1 (strongly disagree) to 7 (strongly agree). Only participants who replied with 6 or 7 to this item were included in the analyses. The second item indirectly assessed participants’ attention: “People vary in the amount they pay attention to these kinds of surveys. Some take them seriously and read each question, whereas others go very quickly and barely read the questions at all. If you have read this question carefully, please write the word yes in the blank box below labeled other. There is no need for you to respond to the scale below.” Participants were then presented with a Likert scale (1 to 7) and a text-box labeled “other.” Only participants who wrote “yes,” “YES,” or “Yes” into the text-box were included in the analyses.”
Next, compute pattern deviancy aversion scores for each participant by averaging the three items used to assess 1) discomfort, 2) anxiousness, and 3) annoyance in response to the 5 pattern deviance images.
Similarly, compute social deviancy aversion scores for each participant by averaging the three items used to assess 1) discomfort, 2) anxiousness, and 3) annoyance in response to the 5 social deviance images.
Compute Need for Closure score for each participant by calculating the mean score across all 15 items.
Compute Neuroticism score for each participant by: 1) reverse coding first item, and 2) calculating the mean of the two items.
Key analysis of interest: Linear regression testing whether pattern deviancy aversion (IV) predicts social deviancy aversion (DV).
Additional analyses: Repeat the above linear regression, but this time controlling for Need for Closure, Neuroticism, and Political Orientation to see whether the association remains.
Differences from Original Study
Since the original study was also coducted on MTurk, I don’t expect big differences in our sample. However, it is possible that the MTurk population is slightly different now (in 2018) than when the original study was conducted (2017).
Methods Addendum (Post Data Collection)
Actual Sample
2 participants were excluded for failing the attention checks (excat same criteria were used as the original study, as described above). This resulted in an exclusion rate of 4% and a final sample size of 52. 37% were female, and 63% were male. The average age was 35.41 (SD = 10.42).
Differences from pre-data collection methods plan
None.
Results
Data preparation
Data preparation following the analysis plan.
###Data Preparation
setwd("/Users/juliecachia/Desktop/RProjects/Replication Data") #Set working directory
####Load Relevant Libraries and Functions
library(tidyverse)
library(ggplot2)
library(ggthemes)
#install.packages(("umx")) #Install this package for Cronbach's alpha calculation
library(umx) #For Cronbach Alpha calculation in exploratory analysis
#install.packages("QuantPsyc")
library(QuantPsyc)
####Import data
RawData <- read_csv("data.csv")#### Data exclusion / filtering
RawData <- RawData[3:nrow(RawData),] #Take out 2 first heading rows
colnames(RawData)[colnames(RawData) == "FL_28_DO"] <- "RandOrder" #Rename column representing block randomizer display order for Social Deviancy Aversion and Pattern Deviancy Aversion (was not able to specify this within Qualtrics)
RawData$RandOrder[RawData$RandOrder=="FL_10|FL_24"] <- "PatternDevFirst" #"FL_10|FL_24" means Pattern Deviancy Aversion was presented first
RawData$RandOrder[RawData$RandOrder=="FL_24|FL_10"] <- "SocialDevFirst" #"FL_24|FL_10" means Social Deviancy Aversion was presented first
#Using filter function in tidyverse, exclude participants who failed attention checks: 1) scoring below 6 on first item, and/or 2) not entering "Yes", "yes", or "Yes" into the text field of the second item
PassedACheck1 <- RawData$AttentionCheck1 > 5
PassedACheck2 <- grepl("yes", RawData$AttentionCheck2_8_TEXT, ignore.case=TRUE)
#Filter dataset to obtain only those rows that passed both attention checks
ExcludedData <- filter(RawData, PassedACheck1 & PassedACheck2)
#### Prepare data for analysis - create columns
#Include participant id column so that later I can group_by(id)
ExcludedData <- ExcludedData %>%
mutate(id = row_number())
###Column names for items of relevant variables:
#Pattern Deviancy Items: PatternDevA_1, PatternDevA_2, PatternDevA_3 ... PatternDevE_1, PatternDevE_2, PatternDevE_3
#Social Deviancy Items: SocialDevA_1, SocialDevA_2, SocialDevA_3 ... SocialDevE_1, SocialDevE_2, SocialDevE_3
#Need for Closure Items: NeedForClosure1 ... NeedForClosure15
#Neuroticism Items: Neuroticism_1, Neuroticism_2
#Political Orientation Item: PoliticalOrientation
#Make data tidy
TidyData <- ExcludedData %>%
gather(contains("Dev"), contains("NeedClosure"), contains("Neuroticism"), PoliticalOrientation, key = "Scale", value = "Rating") #Gather all scales into Scales column and all ratings into Rating column
#Turn whole Rating column into numeric
TidyData$Rating <- as.numeric(TidyData$Rating)
#Reverse code Neuroticism_1 by subtracting every value from 6:
TidyData$Rating[TidyData$Scale == "Neuroticism_1"] <- 6 - TidyData$Rating[TidyData$Scale == "Neuroticism_1"]
#Rename all items of a scale (e.g., Neuroticism_1, Neuroticism_2) into single label (e.g., Neuroticism)
TidyData$Scale[startsWith(TidyData$Scale, "Pattern")] <- "PatternDev"
TidyData$Scale[startsWith(TidyData$Scale, "Social")] <- "SocialDev"
TidyData$Scale[startsWith(TidyData$Scale, "NeedClosure")] <- "NeedForClosure"
TidyData$Scale[startsWith(TidyData$Scale, "Neuroticism")] <- "Neuroticism"
#Create final dataframe specifically for the key analysis
OnlyKeyVariables <- TidyData %>%
group_by(id, Scale, RandOrder) %>%
summarise(ScaleMeans = mean(Rating))
#Spread data back out so we have a row for each participant and a column for each variable mean
OnlyKeyColumns <- OnlyKeyVariables %>%
spread(key = Scale, value = ScaleMeans)
OnlyKeyColumns #Output final dataframe## # A tibble: 52 x 7
## # Groups: id [52]
## id RandOrder NeedForClosure Neuroticism PatternDev PoliticalOrient…
## <int> <chr> <dbl> <dbl> <dbl> <dbl>
## 1 1 SocialDe… 4.13 3 3.87 5
## 2 2 SocialDe… 5.2 2.5 1.47 9
## 3 3 PatternD… 4.73 1.5 1 9
## 4 4 SocialDe… 3.53 2 1.4 4
## 5 5 PatternD… 5.2 4.5 1.27 2
## 6 6 SocialDe… 3.6 1 1 3
## 7 7 SocialDe… 4.67 1 1 1
## 8 8 SocialDe… 4.47 2 1 2
## 9 9 SocialDe… 4.47 4 2.93 1
## 10 10 PatternD… 2.67 1 1.6 1
## # … with 42 more rows, and 1 more variable: SocialDev <dbl>Participant demographics
#Sample size after exclusion
length(ExcludedData$Age) ## [1] 52#Mean & SD for Age
mean(as.numeric(ExcludedData$Age), na.rm=TRUE) #Mean## [1] 35.41176sd(ExcludedData$Age, na.rm=TRUE) #SD## [1] 10.42147#Gender (1 = male, 2 = female)
pMale <- sum(ExcludedData$Gender==1)/length(ExcludedData$Age) #proportion male
pFemale <- 1-pMale #proportion female
pMale## [1] 0.6346154pFemale## [1] 0.3653846#Political orientation
mean(as.numeric(ExcludedData$PoliticalOrientation), na.rm=TRUE) #Mean## [1] 4.038462sd(ExcludedData$PoliticalOrientation, na.rm=TRUE) #SD## [1] 2.566462Barplot of education:
ggplot(ExcludedData, aes(x=as.numeric(Education)))+
geom_bar()+
theme_classic()+
scale_x_continuous(breaks=c(1,2,3,4,5,6,7), labels=c("HS No Diploma", "HS Diploma", "Some College", "Associate Degree", "Bachelor’s Degree", "Master’s Degree", "PhD"))+
xlab("Education")+
ylab("Frequency")
Barplot of income:
ggplot(ExcludedData, aes(x=as.numeric(Income)))+
geom_bar()+
theme_classic()+
scale_x_continuous(breaks=c(1,2,3,4,5), labels=c("Under $25K", "$25-50K", "$50-100K", "$100-250K", "More than $250K"))+
xlab("Income")+
ylab("Frequency")
Barplot of political orientation:
ggplot(ExcludedData, aes(x=as.numeric(PoliticalOrientation)))+
geom_bar()+
theme_classic()+
scale_x_continuous(breaks=c(1,2,3,4,5,6,7,8,9), labels=c("Extremely liberal", " ", " ", " ", " ", " ", " ", " ", "Extremely conservative"))+
xlab("Political Orientation")+
ylab("Frequency")
Confirmatory analysis
#Conduct linear regression to test whether Pattern Deviancy Aversion predicts Social Deviancy Aversion
KeyAnalysis <- lm(SocialDev ~ PatternDev, data=OnlyKeyColumns)
summary(KeyAnalysis)##
## Call:
## lm(formula = SocialDev ~ PatternDev, data = OnlyKeyColumns)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.8446 -0.6538 -0.1155 0.5420 3.6605
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.1789 0.2996 3.935 0.000257 ***
## PatternDev 0.7606 0.1203 6.322 6.9e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.136 on 50 degrees of freedom
## Multiple R-squared: 0.4442, Adjusted R-squared: 0.4331
## F-statistic: 39.97 on 1 and 50 DF, p-value: 6.896e-08Side-by-side graph with original graph is ideal here
The original paper doesn’t have a graph of this result, so instead I added the original slope (and assumed the same y intercept), shown in red (dotted line).
ggplot(OnlyKeyColumns, aes(x=PatternDev, y=SocialDev))+
geom_point()+
geom_smooth(method=lm)+
theme_classic(base_size = 12)+
theme(axis.line = element_line(color = 'black'))+
geom_abline(intercept = coef(KeyAnalysis)[1], slope = 0.38, color="red",
linetype="dashed", size=1)+
ggtitle("Dependence of Social Deviancy Aversion on Pattern Deviancy Aversion")+
xlab("Pattern Deviancy Aversion")+
ylab("Social Deviancy Aversion")+
ylim(1,7)
ggsave("PDA_SDA.pdf")## Saving 7 x 5 in imageComparing effect sizes (standardized beta coefficients):
Original article reported unstandardized beta coefficient = 0.38.
coef_lmbeta <- lm.beta(KeyAnalysis)
coef_lmbeta## PatternDev
## 0.6665191Our replication yielded an standardized beta coefficient = 0.67.
Exploratory analyses
Controlling for Need for Closure, Neuroticism, and Political Orientation:
AdditionalAnalysis <- lm(SocialDev ~ PatternDev + NeedForClosure + Neuroticism + PoliticalOrientation, data=OnlyKeyColumns) #include 3 covariates
summary(AdditionalAnalysis)##
## Call:
## lm(formula = SocialDev ~ PatternDev + NeedForClosure + Neuroticism +
## PoliticalOrientation, data = OnlyKeyColumns)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.4460 -0.7278 -0.1955 0.5313 3.4344
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.59317 0.84120 0.705 0.484
## PatternDev 0.79361 0.14019 5.661 8.73e-07 ***
## NeedForClosure 0.28447 0.23734 1.199 0.237
## Neuroticism -0.21230 0.13631 -1.557 0.126
## PoliticalOrientation -0.03361 0.07080 -0.475 0.637
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.135 on 47 degrees of freedom
## Multiple R-squared: 0.4786, Adjusted R-squared: 0.4342
## F-statistic: 10.78 on 4 and 47 DF, p-value: 2.768e-06Discussion
Summary of Replication Attempt
The primary result from the confirmatory analysis was that aversion towards non-social deviant patterns predicted aversion towards deviant individuals, F(1, 50) = 39.97, p < 0.001, R2 = .44. The standardized beta coefficient was 0.67. This fully replicated the original finding (standardized beta coefficient = 0.38, p < 001). In fact, the effect size was larger than the original.
Commentary
Follow-up exploratory analyses revealed that pattern deviancy aversion predicted social deviancy aversion even when controlling for Need for Closure, Neuroticism and Political Orientation, F(4, 47) = 10.78, p > 0.001, R2 = .48, thereby replicating the same analysis in the original paper.
The pattern deviancy aversion scale and social deviancy aversion scale seemed to be reliable (Cronbach’s alpha = 0.95 and 0.94, respectively).
Finally, there were no order effects based on whether Social Deviancy Aversion or Pattern Deviancy Aversion was measured first.
The successful replication provides additional evidence in support of the idea that there is a domain-general propensity to dislike pattern deviancy (both social and non-social).
No objections have been raised by the current or original authors about the replication attempt.