Replication of The moral roots of environmental attitudes by Feinberg and Willer (2012, Psychological Science)
Note to TAs: this replication will be done in collaboration with a friend of mine, using his sources of funding, as he is also interested in whether this study will replicate. We will be adding in two additional control condition, but the nature of the analyses used in the original study will mean that this work will both demonstrate whether the original study replicates and also show any results from the additional control conditions. We will still just be focusing on Study 3, and primary attention will be paid to the “Environmental attitudes and policy preferences” section of Study 3, but disgust and its potential mediating role will also be measured.
Introduction
This project aims to both determine whether the findings of Study 3 replicate. It additionally aims to assess whether the nature of the control condition used in the original study affected these findings, by adding in both a control condition with a political message, and a control condition that focuses on environmental topics without using moralized language. Note: as this will eventually be published, I’ll have a literature review and more in-depth intro in here as well, coming soon.
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
Sample size - 1000. 500 liberals, 500 conservatives. True independents will not be allowed to take the survey.
Materials
The experimental articles:
“The harm/care message described the harm and destruction humans are causing to the environment and emphasized how important it is for people to care about and protect the environment. Participants in the purity/sanctity condition read about how polluted and contaminated the environment has become and how important it is for people to clean and purify the environment. Additionally, three small pictures were presented with the persuasive messages. The purity/sanctity pictures showed a cloud of pollution looming over a city, a person drinking contaminated water, and a forest covered in garbage. The harm/care pictures showed a destroyed forest of tree stumps, a barren coral reef, and cracked land suffering from drought. Importantly, both messages ended positively, providing information regarding what people can do to improve the environment”
The control article:
“Participants in the neutral condition read an apolitical message on the history of neckties.”
Two additional control articles, created by my friend and myself, will be added. One will be an article that talks about the environment/climate change but uses morally neutral language, and the other will be a political but non-partisan article (talking about the branches of government or something along those lines).
Procedure
The original study was described as follows:
“Participants completed a demographic questionnaire…Participants in the harm/care and purity/sanctity conditions read a persuasive message modeled after a newspaper op-ed. These messages utilized words typi- cal of their respective moral domains…Participants in the neutral condition read an apolitical message on the history of neckties…After reading the message, participants indicated how much they felt five emotions, including disgust…Participants then completed a 3-item measure of proenvironmental attitudes, a 5-item mea- sure of support for proenvironmental legislation, and a 3-item measure of belief in global warming.”
My replication will differ from this in the following way: there will two more control conditions, one with a message about the environment and climate change that uses morally neutral language, and one control condition that describes something political but not partisan. This will allow for a better understanding of whether the large differences from the original control condition had anything to do with the apolitical message that it had, and whether it was the morally valenced language that created differences in proenvironmental
Additionally, we will be including some attention checks, but not filtering on them.
Analysis Plan
Analyses:
Multiple regression analysis that compares the interactions between each of the five conditions and political ideology. Simple slopes analyses comparing both conservatives across conditions, and comparing liberals across conditions, as well as comparing conservatives and liberals within each condition.
Additional analyses (TBD, over the course of this coming week) to measure the difference between the control conditions and the experimental conditions.
Differences from Original Study
The addition of the control conditions is an obvious difference, but the multiple regression analysis will still allow for the same results to be calculated. Over this coming week, I’ll be gaining access to as much of the original materials used as possible; I’ll fill in this section when I know what I won’t have access to.
Methods Addendum (Post Data Collection)
Actual Sample
Differences from pre-data collection methods plan
Results
Data preparation
For repeat survey-takers, include the data from the first time they took the survey but exclude all other data. If any independents manage to take the survey, remove their data as well.
Create an attention check score column, in which the number of missed attention checks are indicated.
Confirmatory analysis
Multiple regression analysis that compares the interactions between each of the five conditions and political ideology. Again, there are five conditions because we’ll be adding in an extra control condition.
Simple slopes analyses comparing both conservatives across conditions, and comparing liberals across conditions, as well as comparing conservatives and liberals within each condition.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Actual model TBD (dicussing with my friend this week) but will be comparing the differences between each of the controls and the experimental conditions (e.g. comparing effect sizes, etc.)
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.