Replication of Study 1A by Liane Young and Rebecca Saxe (2011, Cognition)
Author
Replication Author[s]: Seyi Lawal
Published
December 7, 2024
Introduction
The main thesis of the original paper is that while intent matters in making moral judgments, intent will play a smaller role in making moral judgments of purity-related violations vs. harm-related violations. In the first study, the authors test whether for harms and purity violations that are equally wrong, when committed intentionally, there’s a difference in the moral judgment when those actions are committed accidentally. The study is operationalized as a 2 (intent: intentional vs. accidental) x 3 (domain: harm vs. incest vs. ingestion) between subjects design, where participants are presented with a moral scenario and asked to evaluate the extent to which the action is morally wrong on a scale of 1 - 7. There are 2 scenarios representing each domain. The main finding we are planning to replicate are the three, 2 (intent) x 2 (domain) ANOVAs comparing the importance of intent for harm vs. incest, incest vs. ingestion, and harm vs. ingestion. Through the analysis, the authors found an intent by domain interaction when comparing the harm violations to the purity violations, but not the two purity violations to one another.
The partial eta squared of the main result in the paper (the intent by domain interaction between harm and purity violations) was 0.15 (cohen’s f = 0.42. Given for each ANOVA, there are 4 groups, 1 degree of freedom, and we’re targeting p < 0.05 (in effect p < 0.0167 given we are using the same data for 3 comparisons), We need a sample size of 95 to have 80% power, 120 to have 90% power, and 144 to have 95% power. Secondary analyses had partial eta squareds as low as 0.03 (cohen’s f = 0.18), which would require a sample size of 513 to detect. For this study we went with a sample size in the middle of these two of 351.
Planned Sample
We will screen out repeat participants through the same exclusion rules as the original paper - people who have taken a similar survey before and repeat prolific IDs. We will target 351 participants to have 80% power to detect the main interaction effect in the paper, understanding that some may drop given the exclusion criteria above.
Materials
Below is the text for the scenarios participants will be prompted with. This follows the original paper exactly.
“Harm
Allergy - Intentional: Imagine your cousin is over for dinner. You know she is allergic to peanuts. You decide to add peanuts to the dish because they really bring out the flavor for you. You grind up the peanuts, add them in, and serve your cousin.
Allergy - Accidental: Imagine your cousin is over for dinner. You have no idea she is allergic to peanuts. You decide to add peanuts to the dish because they really bring out the flavor for you. You grind up the peanuts, add them in, and serve your cousin.
Poison - Intentional: Imagine you and a co-worker are taking a tour of a chemical plant. During the coffee break, you go to pour some coffee. You like yours black, but your co-worker asks for sugar in hers. You spoon some powder into your co-worker’s coffee. You know full well that you are putting poison in her coffee.
Poison - Accidental: Imagine you and a co-worker are taking a tour of a chemical plant. During the coffee break, you go to pour some coffee. You like yours black, but your co-worker asks for sugar in hers. You spoon some powder into your co-worker’s coffee. You do not know that someone replaced the sugar with poison.
Incest
Sibling - Intentional: Imagine you are at a dorm party and really hit it off with someone. This person feels the same way about you. That night, you end up talking, and you discover that you are actually long lost siblings. At the end of the night, you decide to sleep together (using two forms of birth control to be safe).
Sibling - Accidental: Imagine you are at a dorm party and really hit it off with someone. This person feels the same way about you. At the end of the night, you decide to sleep together (using two forms of birth control to be safe). The next day, you end up talking, and you discover that you are actually long lost siblings.
Parent - Intentional: Imagine you were adopted at birth - you have never met your birth parents. The weekend of your college reunion, you go to your school’s football game and meet someone a bit older. You discover that this person is actually your biological parent. You two get along really well and, that night, end up sleeping together (using two forms of birth control to be safe).
Parent - Accidental: Imagine you were adopted at birth - you have never met your birth parents. The weekend of your college reunion, you go to your school’s football game and meet someone a bit older. You have no idea that this person is actually your biological parent. You two get along really well and, that night, end up sleeping together (using two forms of birth control to be safe).
Ingestion
Dog - Intentional: Imagine that a car just killed your beloved dog. Your significant other has heard that dog meat is delicious and freezes the meat of your dog before it goes bad. Later, you decide to make yourself dinner. You see a package in the freezer. It is correctly labeled “dog”. You end up eating your dog for dinner.
Dog - Accidental: Imagine that a car just killed your beloved dog. Your significant other has heard that dog meat is delicious and freezes the meat of your dog before it goes bad. Later, you decide to make yourself dinner. You see a package in the freezer. It is incorrectly labeled “beef” – but it is actually the meat from your dog. Without realizing it, you end up eating your dog for dinner.
Urine - Intentional: Imagine that you are waiting to brush your teeth while your friend is in the bathroom. As she leaves, you notice a cup and a pregnancy test. Your friend warns you that the cup contains her urine for her pregnancy test. You finish brushing your teeth and use that cup to rinse out your mouth.
Urine – Accidental: Imagine that you are waiting to brush your teeth while your friend is in the bathroom. As she leaves, you notice a cup and a pregnancy test. Your friend forgets to tell you that the cup contains her urine for her pregnancy test. You finish brushing your teeth and use that cup to rinse out your mouth. ”
Procedure
We will follow the procedure exactly as described in the original paper. “Participants were assigned randomly to one of six conditions in a 2 (intentional versus accidental) 3 (harm versus incest versus ingestion) between-subjects experimental design. Participants then judged the moral wrongness of the action [through the question “How morally wrong was the action?”], on a 7-point scale anchored at ‘‘not at all morally wrong’’ (1) to ‘‘very morally wrong’’ (7). Two scenarios represented each domain: harm (allergy, poison), incest (parent, sibling) or ingestion (dog meat, urine). There were two versions of each scenario (intentional, accidental); each participant saw only one version.”
Analysis Plan
The analysis will follow the original paper exactly. We will first screen out repeat participants through the same exclusion rules as the original paper - people who have taken a similar survey before and repeat prolific IDs. Repeat Prolific ID screening is handled by Prolific. We did not collect Prolific IDs. We will then run three, 2 (intent) x 2 (domain) ANOVAs comparing harm vs. incest, harm vs. ingestion, and incest vs. ingestion. The analysis will be collapsed across the 2 scenarios in each domain, as done in the original paper. For the 2 main effects and the interaction, we’ll report the F-stat, p value, and partial eta squared.
To replicate the original findings, we will be looking to confirm the intent by domain interaction is significant when comparing the harm violations vs. purity violations, and not significant when comparing the two purity violations.
Differences from Original Study
The only differences I expect are from the fact that we’re using a different set of participants (different platform - prolific vs. mturk, small differences in final participant count) and the date we’re running the study (late 2000s vs. 2024). In case general perspectives on any of these domains have shifted over time, we could see a change in the results; however, I do not anticipate this to be the case.
Methods Addendum (Post Data Collection)
We did not make any changes to the methods for this study.
Actual Sample
Our actual sample size was 341 participants. All were English speaking adults living in the United States, who were over 18 years old. 342 participants completed the study, out of the 351 recruited. One participant was screened out of the final sample because they had responded Yes to having taken the survey before.
Differences from pre-data collection methods plan
None
Design Overview
How many factors were manipulated?
2 - intent and domain
How many measures were taken?
1 - perceptions of moral wrongfulness of the action
Did it use a within-participants or between-participants design?
Between participants
Were measures repeated?
No
What would have been the consequence of changing one of these decisions? (e.g. moving from within to between participants). Could it have been either way?
They could have done a within participants study; however, it would have enlightened the participants to what the manipulation was and possibly have biased participants’ responses.
Were steps taken to reduce demand characteristics?
Only having 1 question per scenario reduces demand characteristics as it’s unclear what the researcher is trying to get at.
How would you critique the design?
Can you think of any potential confounds to the study?
It is possible that wtih the accidental purity violations, participants’ do not actually believe the action was an accident (e.g., how can you not realize that you’re eating dog instead of beef, or drinking urine instead of water?). So it would be interesting to test whether with accidental harm violations people actually believe it was an accident more than with accidental purity violations.
Do you see any limitations to generalizability?
The study chose very specific scenarios of harm vs. purity violations. It’s unclear whether these findings would generate to other purity violations. For example, adultery is considered to be a purity violation in psychology, but I believe people would judge accidental adultery differently from purposeful adultery.
Results
Data preparation
Data preparation following the analysis plan.
### Data Preparation#### Load Relevant Libraries and Functionsinstall.packages("pacman", repos ="http://cran.us.r-project.org")
package 'pacman' successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\seyil\AppData\Local\Temp\RtmpAxefCB\downloaded_packages
pacman::p_load(dplyr,tidyverse,ggplot2,lubridate,stringr,kableExtra, scales, effectsize, tools, png, grid)#### Import data; load data from csvcsv_files <-list.files(path ="~/Coursework/Fall 2024/PSYC 201A/young2011/data/FullSample", pattern ="\\.csv$", full.names =TRUE)raw_data <- csv_files %>%map_dfr(~read_csv(.x) %>%mutate(subject =file_path_sans_ext(basename(.x))))# #### Data exclusion / filtering## add scenario to response row; then, remove html response rowscleaned_data <- raw_data %>%mutate(stimulus =case_when( trial_type =="survey-likert"~lag(stimulus),TRUE~ stimulus # Keep the original value if the condition is not met ) )cleaned_data <- cleaned_data %>%filter(trial_type !="html-keyboard-response")#remove any participants who say they have take a similar test beforeparticipants_who_have_taken_survey <- cleaned_data %>%filter(response =="{\"TakenSurveyBefore\":\"Yes\"}") %>%distinct(subject) %>%select(subject) cleaned_data <- cleaned_data %>%anti_join(participants_who_have_taken_survey, by ="subject")#### Prepare data for analysis - create columns etc.#create columns for scenario (allergy, poison, sibling incest, parent incest, eating dog, drinking urine), intent (intentional versus accidental), domain (harm versus incest versus ingestion), and violation type (harm vs. purity)cleaned_data <- cleaned_data %>%mutate(scenario =case_when(str_detect(stimulus, "You <strong>have no idea</strong> she is allergic to peanuts") ~"allergy",str_detect(stimulus, "<strong>You do not know that someone replaced the sugar with poison.</strong>") ~"poison",str_detect(stimulus, "<strong>The next day, you end up talking, and you discover that you are actually long lost siblings.</strong>") ~"sibling_incest",str_detect(stimulus, "You <strong>have no idea</strong> that this person is actually your biological parent") ~"parent_incest",str_detect(stimulus, "It is <strong>incorrectly labeled “beef”") ~"eating_dog",str_detect(stimulus, "Your friend <strong>forgets to tell you</strong> that the cup contains her urine for her pregnancy test.") ~"drinking_urine",str_detect(stimulus, "You <strong>know</strong> she is allergic to peanuts.") ~"allergy",str_detect(stimulus, "<strong>You know full well that you are putting poison in her coffee.</strong>") ~"poison",str_detect(stimulus, "That night, you end up talking, and you discover that you are actually long lost siblings.") ~"sibling_incest",str_detect(stimulus, "You <strong>discover</strong> that this person is actually your biological parent") ~"parent_incest",str_detect(stimulus, "It is <strong>correctly labeled “dog”</strong>.") ~"eating_dog",str_detect(stimulus, "Your friend <strong>warns</strong> you that the cup contains her urine for her pregnancy test") ~"drinking_urine" ) )cleaned_data <- cleaned_data %>%mutate(domain =case_when( scenario =="allergy"~"harm", scenario =="poison"~"harm", scenario =="sibling_incest"~"incest", scenario =="parent_incest"~"incest", scenario =="eating_dog"~"ingestion", scenario =="drinking_urine"~"ingestion" ))cleaned_data <- cleaned_data %>%mutate(violation_type =case_when( domain =="harm"~"harm", domain =="incest"~"purity", domain =="ingestion"~"purity" )) cleaned_data <- cleaned_data %>%mutate(intent =case_when(str_detect(stimulus, "You <strong>have no idea</strong> she is allergic to peanuts") ~"accidental",str_detect(stimulus, "<strong>You do not know that someone replaced the sugar with poison.</strong>") ~"accidental",str_detect(stimulus, "<strong>The next day, you end up talking, and you discover that you are actually long lost siblings.</strong>") ~"accidental",str_detect(stimulus, "You <strong>have no idea</strong> that this person is actually your biological parent") ~"accidental",str_detect(stimulus, "It is <strong>incorrectly labeled “beef”") ~"accidental",str_detect(stimulus, "Your friend <strong>forgets to tell you</strong> that the cup contains her urine for her pregnancy test.") ~"accidental",str_detect(stimulus, "You <strong>know</strong> she is allergic to peanuts.") ~"intentional",str_detect(stimulus, "<strong>You know full well that you are putting poison in her coffee.</strong>") ~"intentional",str_detect(stimulus, "That night, you end up talking, and you discover that you are actually long lost siblings.") ~"intentional",str_detect(stimulus, "You <strong>discover</strong> that this person is actually your biological parent") ~"intentional",str_detect(stimulus, "It is <strong>correctly labeled “dog”</strong>.") ~"intentional",str_detect(stimulus, "Your friend <strong>warns</strong> you that the cup contains her urine for her pregnancy test") ~"intentional" ) )cleaned_data <- cleaned_data %>%filter (trial_type !="survey-multi-choice")###increment moral_wrongness assessment to account for the fact that reponses are mapped onto 0-6 scale, instead of 1 - 7 scalecleaned_data <- cleaned_data %>%mutate(moral_wrongness =as.numeric(str_sub(response, 7, 7)) +1)#create subset dataframe for comparing harm vs. incest (filter domain column)dataframe_harm_incest <- cleaned_data %>%filter(domain =="harm"| domain =="incest")#create subset dataframe for comparing harm vs. ingestion (filter domain column)dataframe_harm_ingestion <- cleaned_data %>%filter(domain =="harm"| domain =="ingestion")#create subset dataframe for comparing incest vs. ingestion (filter domain column)dataframe_incest_ingestion <- cleaned_data %>%filter(domain =="incest"| domain =="ingestion")
Confirmatory analysis
To confirm the results from the paper, we will run three 2 (intent) x 2(domain) ANOVAs which compare the role of intent for harm vs. incest, harm vs. ingestion, and incest vs. ingestion.
On the next pages are the results of the ANOVAs, followed by the original figure from the paper, and then a similar figure from our replication.
#calculate aov comparing harm vs. incestcat("\n AOV comparing harm vs. incest")
AOV comparing harm vs. incest
anova_result_harm_incest <-aov(moral_wrongness ~ intent * domain, data = dataframe_harm_incest)print(summary(anova_result_harm_incest))
## create bar graph in fig 2#calculate mean and standard error for moral_wrongness by scenario and intentdf_summary <- cleaned_data %>%group_by(scenario, intent) %>%summarize(mean_moral_wrongness =mean(moral_wrongness),se_moral_wrongness =sd(moral_wrongness) /sqrt(n()) )
`summarise()` has grouped output by 'scenario'. You can override using the
`.groups` argument.
#add violation type to summary data framedf_summary <- df_summary %>%mutate(violation_type =case_when( scenario =="allergy"~"harm", scenario =="poison"~"harm", scenario =="sibling_incest"~"purity", scenario =="parent_incest"~"purity", scenario =="eating_dog"~"purity", scenario =="drinking_urine"~"purity" ))#create plotdf_summary$intent <-fct_rev(df_summary$intent)cat("\n Original Plot")
ggplot(df_summary, aes(x = scenario, y = mean_moral_wrongness,label = scenario, fill = violation_type)) +geom_bar(stat ="identity", position =position_dodge(width =0.8)) +geom_errorbar(aes(ymin = mean_moral_wrongness - se_moral_wrongness, ymax = mean_moral_wrongness + se_moral_wrongness),width =0.2, position =position_dodge(width =0.8)) +facet_wrap(~intent, scales ="free_x") +#split intentional vs. accidentascale_fill_manual(values =c("harm"="gray", "purity"="black")) +# Color mappinglabs(x ="Scenario", y ="Morality Rating (1 (Not at all morally wrong) - 7 (very morally wrong))", title ="Experiment 1: Influence of intent on moral judgments - harm vs. purity") +theme_minimal() +theme(axis.text.x =element_text(size =12),axis.text.y =element_text(size =12) )
Exploratory analyses
For exploratory purposes, I also looked at the t-tests noted in the paper. The authors found that incest was judged morally worse than harm when both committed accidentally, though harm was judged worse than incest when both were committed intentionally. Similarly, ingestion was morally worse than harm when both were committed accidentally, but not significantly different when committed intentionally.
My results shown below confirm the same:
Accidental incest: 3.16 out of 7; Accidental harm: 2.31 out of 7; p = 0.04)
Intentional incest: 6.07 out of 7; Intentional harm: 6.64 out of 7; p = 0.03)
Accidental ingestion : 4.83 out of 7; Accidental harm: 2.31 out of 7; p < 0.001)
Intentional ingestion : 5.42 out of 7; Intentional harm: 6.64 out of 7;p < 0.001)
I also looked at Accidental Incest vs. Accidental Ingestion and Intentional Incest vs. Intentional Ingestion and found a similar reversal where accidental ingestion was judged worse than accidental incest, but intentional incest was judged worse than intentional ingestion:
Accidental incest: 3.16 out of 7; Accidental ingestion: 4.83 out of 7; p = p<0.001)
Intentional incest: 6.07 out of 7; Intentional ingestion: 5.42 out of 7; p = 0.045)
Further, there was no statistically significant difference in how participants viewed intentional ingestion and accidental ingestion
Accidental ingestion: 4.83 out of 7; Intentional ingestion: 5.42 out of 7; p = 0.099)
###create column that covers both domain and intentcleaned_data <- cleaned_data %>%mutate(combined_scenario=paste0(domain,"_", intent) )###create sub-data frames to set up t-testsdataframe_accident_harm_incest <- cleaned_data %>%filter(combined_scenario =="harm_accidental"| combined_scenario =="incest_accidental")cat("\n T-test comparing accidental harm vs. accidental incest")
T-test comparing accidental harm vs. accidental incest
t.test(moral_wrongness ~ combined_scenario, data = dataframe_accident_harm_incest)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = -2.0796, df = 96.917, p-value = 0.04021
alternative hypothesis: true difference in means between group harm_accidental and group incest_accidental is not equal to 0
95 percent confidence interval:
-1.66882416 -0.03893094
sample estimates:
mean in group harm_accidental mean in group incest_accidental
2.306122 3.160000
T-test comparing intentional harm vs. intentional incest
t.test(moral_wrongness ~ combined_scenario, data = dataframe_intentional_harm_incest)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = 2.2797, df = 66.081, p-value = 0.02586
alternative hypothesis: true difference in means between group harm_intentional and group incest_intentional is not equal to 0
95 percent confidence interval:
0.07041106 1.06324748
sample estimates:
mean in group harm_intentional mean in group incest_intentional
6.640000 6.073171
T-test comparing accidental harm vs. accidental ingestion
t.test(moral_wrongness ~ combined_scenario, data = dataframe_accident_harm_ingestion)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = -6.4674, df = 107.38, p-value = 3.03e-09
alternative hypothesis: true difference in means between group harm_accidental and group ingestion_accidental is not equal to 0
95 percent confidence interval:
-3.298465 -1.750828
sample estimates:
mean in group harm_accidental mean in group ingestion_accidental
2.306122 4.830769
T-test comparing intentional harm vs. intentional ingestion
t.test(moral_wrongness ~ combined_scenario, data = dataframe_intentional_harm_ingestion)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = 4.5634, df = 98.316, p-value = 1.456e-05
alternative hypothesis: true difference in means between group harm_intentional and group ingestion_intentional is not equal to 0
95 percent confidence interval:
0.6870841 1.7444311
sample estimates:
mean in group harm_intentional mean in group ingestion_intentional
6.640000 5.424242
T-test comparing accidental incest vs. accidental ingestion
t.test(moral_wrongness ~ combined_scenario, data = dataframe_accident_incest_ingestion)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = -4.1891, df = 107, p-value = 5.774e-05
alternative hypothesis: true difference in means between group incest_accidental and group ingestion_accidental is not equal to 0
95 percent confidence interval:
-2.4614104 -0.8801281
sample estimates:
mean in group incest_accidental mean in group ingestion_accidental
3.160000 4.830769
T-test comparing intentional incest vs. intentional ingestion
t.test(moral_wrongness ~ combined_scenario, data = dataframe_intentional_incest_ingestion)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = 2.0241, df = 102.58, p-value = 0.04556
alternative hypothesis: true difference in means between group incest_intentional and group ingestion_intentional is not equal to 0
95 percent confidence interval:
0.01304677 1.28480985
sample estimates:
mean in group incest_intentional mean in group ingestion_intentional
6.073171 5.424242
T-test comparing accidental ingestion vs. intentional ingestion
t.test(moral_wrongness ~ combined_scenario, data = dataframe_intentional_accidental_ingestion)
Welch Two Sample t-test
data: moral_wrongness by combined_scenario
t = -1.6627, df = 126.89, p-value = 0.09884
alternative hypothesis: true difference in means between group ingestion_accidental and group ingestion_intentional is not equal to 0
95 percent confidence interval:
-1.2997816 0.1128352
sample estimates:
mean in group ingestion_accidental mean in group ingestion_intentional
4.830769 5.424242
Discussion
Summary of Replication Attempt
The original paper found that when comparing harm to incest and harm to ingestion there was a significant intent by domain interaction; however, when comparing incest to ingestion, there was no intent by domain interaction. These results were the basis for their conclusion that there’s a significant difference in the role of intent for harm violations compared to both kinds of purity violations, but not between the purity violations.
We partially replicated these findings. The high level results are in the table below; bold values indicate values where the significance is inconsistent with the results of the original paper. In the original paper, there was no Intent by Domain interaction for Incest vs. Ingestion. In our replication, we found a significant interaction. In the original paper there were main effects of domain in the Harm vs. Incest and Harm vs. Ingestion ANOVAs; we did not replicate that effect.
ANOVA
Main Effect Intent (p-value)
Main Effect Domain (p-value)
Intent by Domain Interaction (p-value)
Harm vs. Incest
p < 0.001
p = 0.603
p = 0.002
Harm vs. Ingestion
p < 0.001
p = 0.052
p < 0.001
Incest vs. Ingestion
p < 0.001
p = 0.032
p < 0.001
Commentary
I see this as a partial replication of the original study, particularly when combined with the exploratory analysis. Participants definitely see a difference in the impact of intent depending on whether the action is a harm violation or a purity violation.
However, there appears to be a difference in how participants viewed the two types of purity violations. The exploratory analyses showed a similar reversal, where accidental ingestion was judged worse than accidental incest, but intentional incest was judged worse than intentional ingestion. It would be worth testing other scenarios to see if we can tease apart what further differentiates incest and ingestion; there may be a subliminal thought that there is no way one could accidentally ingest these taboo substances and that might be driving the high moral wrongness judgments in the accidental ingestion scenario.
Author Credit Statement
Seyi Lawal: Software, Data Curation, Formal analysis, Investigation, Validation, Writing - Original Draft, Writing - Review & Editing, Visualization. Kexin Jiang: Data Curation, Formal analysis, Investigation, Validation, Writing - Original Draft, Writing - Review & Editing, Visualization. Cassie Wang: Data Curation, Formal analysis, Investigation, Validation, Writing - Original Draft, Writing - Review & Editing, Visualization. Belynda Herrera: Data Curation, Formal analysis, Investigation,Validation, Writing - Original Draft, Writing - Review & Editing, Visualization
