Replication of Experiment 1A by Young & Saxe (2011, Cognition)
Author
Kexin Jiang (k3jiang@ucsd.edu)
Published
November 22, 2024
Introduction
The study investigated the causal effect of agents’ intent on moral judgment. It hypothesized that there are distinct moral domains—harm and purity—in which the agent’s innocent intention matters more for the former and less for the latter. Experiment 1A directly examines this hypothesis. The results supported it, showing that purity violations were judged morally worse than harm when both were committed accidentally, while harm was judged worse than purity violations when both were committed intentionally. I chose this study because I have a strong interest in social cognition. Theoretically, the study emphasizes how intent influences our understanding of others’ actions, which I find both fascinating and significant for the field. Practically, it serves as a great example of how to uncover distinct cognitive signatures across different domains by manipulating key factors.
This is a between-subjects experiment with six conditions consisting of two factors: intent (intentional vs. accidental) and moral domain (harm vs. incest vs. ingestion). To replicate this experiment, we will collect data from Prolific. During the experiment, participants will first read moral scenarios in the second person tense (e.g., “Your cousin comes over for dinner”, provided in the supplementary materials). Then, they will rate the moral wrongness of the action on a 7-point scale, anchored at “not at all morally wrong” (1) to “very morally wrong” (7). For each moral domain, there are two scenarios: harm (allergy, poison), incest (parent, sibling), and ingestion (dog meat, urine). Each scenario has two versions (intentional and accidental), and each participant will see only one version.
The challenge in replicating this experiment might be a lack of experience in collecting online data from Prolific. I hope I can manage the materials counterbalancing right. I will learn a lot.
There are two groups and one degree of freedom. With a significance level of p ≤ 0.05 and an effect size of 0.8, we used G*Power to estimate that a sample size of 351 is needed to achieve 80% power.
Planned Sample
We can replicate the sample size from the paper, targeting approximately 262 participants.
We will screen out repeat participants using the same exclusion criteria as in the original paper: excluding individuals who have previously taken a similar survey and eliminating data from participants with identical prolific IDs.
Materials
Each participant made a moral judgment for a single scenario. Participants were assigned randomly to one of six conditions in a 2 (intentional versus accidental) × 3 (harm versus incest versus ingestion) between-subjects experimental design. Here are the materials to present to participants.
Harm
Allergy - Intentional: Imagine your cousin is over for dinner. You know she is allergic to peanuts. You decide to add peanuts to the dish because they really bring out the flavor for you. You grind up the peanuts, add them in, and serve your cousin.
Allergy - Accidental: Imagine your cousin is over for dinner. You have no idea she is allergic to peanuts. You decide to add peanuts to the dish because they really bring out the flavor for you. You grind up the peanuts, add them in, and serve your cousin.
Poison - Intentional: Imagine you and a co-worker are taking a tour of a chemical plant. During the coffee break, you go to pour some coffee. You like yours black, but your co-worker asks for sugar in hers. You spoon some powder into your co-worker’s coffee. You know full well that you are putting poison in her coffee.
Poison - Accidental: Imagine you and a co-worker are taking a tour of a chemical plant. During the coffee break, you go to pour some coffee. You like yours black, but your co-worker asks for sugar in hers. You spoon some powder into your co-worker’s coffee. You do not know that someone replaced the sugar with poison.
Incest
Sibling - Intentional: Imagine you are at a dorm party and really hit it off with someone. This person feels the same way about you. That night, you end up talking, and you discover that you are actually long lost siblings. At the end of the night, you decide to sleep together (using two forms of birth control to be safe).
Sibling - Accidental: Imagine you are at a dorm party and really hit it off with someone. This person feels the same way about you. At the end of the night, you decide to sleep together (using two forms of birth control to be safe). The next day, you end up talking, and you discover that you are actually long lost siblings.
Parent - Intentional: Imagine you were adopted at birth - you have never met your birth parents. The weekend of your college reunion, you go to your school’s football game and meet someone a bit older. You discover that this person is actually your biological parent. You two get along really well and, that night, end up sleeping together (using two forms of birth control to be safe).
Parent - Accidental: Imagine you were adopted at birth - you have never met your birth parents. The weekend of your college reunion, you go to your school’s football game and meet someone a bit older. You have no idea that this person is actually your biological parent. You two get along really well and, that night, end up sleeping together (using two forms of birth control to be safe).
Ingestion
Dog - Intentional: Imagine that a car just killed your beloved dog. Your significant other has heard that dog meat is delicious and freezes the meat of your dog before it goes bad. Later, you decide to make yourself dinner. You see a package in the freezer. It is correctly labeled “dog”. You end up eating your dog for dinner.
Dog - Accidental: Imagine that a car just killed your beloved dog. Your significant other has heard that dog meat is delicious and freezes the meat of your dog before it goes bad. Later, you decide to make yourself dinner. You see a package in the freezer. It is incorrectly labeled “beef” – but it is actually the meat from your dog. Without realizing it, you end up eating your dog for dinner.
Urine - Intentional: Imagine that you are waiting to brush your teeth while your friend is in the bathroom. As she leaves, you notice a cup and a pregnancy test. Your friend warns you that the cup contains her urine for her pregnancy test. You finish brushing your teeth and use that cup to rinse out your mouth.
Urine – Accidental: Imagine that you are waiting to brush your teeth while your friend is in the bathroom. As she leaves, you notice a cup and a pregnancy test. Your friend forgets to tell you that the cup contains her urine for her pregnancy test. You finish brushing your teeth and use that cup to rinse out your mouth.
Procedure
Moral scenarios were presented to participants. Participants judged the moral wrongness of the action, on a 7-point scale anchored at “not at all morally wrong” (1) to “very morally wrong” (7). Two scenarios represented each domain: harm (allergy, poison), incest (parent, sibling) or ingestion (dog meat, urine). There were two versions of each scenario (intentional, accidental); each participant saw only one version.
Analysis Plan
We will replicate Experiment 1A from the original study. Before analysis, we will screen out repeat participants.
Our dependent variable will be the moral judgments made by participants on a 7-point scale.
First, to verify that judgments of the two stories representing each violation type did not differ, we will conduct three 2 (story) × 2 (intent) ANOVAs: for harm, incest, and ingestion.
If we replicate the findings of no main effect of story and no story-by-intent interaction, we will proceed with three 2 (intent) × 2 (domain) ANOVAs. This will be the key analysis of this study, allowing us to compare the role of intent in the following pairings: (1) harm versus incest, (2) harm versus ingestion, and (3) incest versus ingestion.
If any significant interaction effects are found, we will further conduct independent-samples t-tests to identify specific difference patterns, following the approach of the original paper. .
Differences from Original Study
The only difference may arise from the different participant samples. Additionally, we collect data from two platforms (Prolific vs. MTurk), and participants on these platforms may have different characteristics.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Design Overview
How many factors were manipulated?
Two factors were manipulated
How many measures were taken?
Just one, the 7-point scale for moral judgement.
Did it use a within-participants or between-participants design?
Between-subject design.
Were measures repeated?
No.
What would have been the consequence of changing one of these decisions? (e.g. moving from within to between participants). Could it have been either way?
If we switch from a between-subject design to a within-subject design, participants might anticipate that the experimenter expects them to behave differently in response to the different domain stories.
Were steps taken to reduce demand characteristics?
We can conduct a double-blind study to help reduce demand characteristics.
Can you think of any potential confounds to the study?
Education level could be a potential confound influencing moral judgment across different moral domains.
Results
Data preparation
Data preparation following the analysis plan.
### Data Preparation#### Load Relevant Libraries and Functionslibrary(tidyr)library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (5): stimulus, response, trial_type, plugin_version, question_order
dbl (4): rt, trial_index, time_elapsed, condition
lgl (3): subject_id, study_id, session_id
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 15 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (5): stimulus, response, trial_type, plugin_version, question_order
dbl (4): rt, trial_index, time_elapsed, condition
lgl (3): subject_id, study_id, session_id
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 15 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (5): stimulus, response, trial_type, plugin_version, question_order
dbl (4): rt, trial_index, time_elapsed, condition
lgl (3): subject_id, study_id, session_id
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 15 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (5): stimulus, response, trial_type, plugin_version, question_order
dbl (4): rt, trial_index, time_elapsed, condition
lgl (3): subject_id, study_id, session_id
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 15 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (5): stimulus, response, trial_type, plugin_version, question_order
dbl (4): rt, trial_index, time_elapsed, condition
lgl (3): subject_id, study_id, session_id
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#### Data exclusion / filteringcleaned_data <- raw_data %>%mutate(stimulus =case_when( trial_type =="survey-likert"~lag(stimulus),TRUE~ stimulus ) )cleaned_data <- cleaned_data %>%filter(trial_type !="html-keyboard-response")# exclude subs that took similar survey beforenum_sub =5cleaned_data <- cleaned_data %>%mutate(subID =rep(1:num_sub, each =7))participants_who_have_taken_survey <- cleaned_data %>%filter(response =="{\"TakenSurveyBefore\":\"Yes\"}") %>%distinct(subID) %>%select(subID) cleaned_data <- cleaned_data %>%anti_join(participants_who_have_taken_survey, by ="subID")#### Prepare data for analysis - create columns etc.# create columns for all variables in a dataframe from cleaned_datacleaned_data <- cleaned_data %>%mutate(scenario =case_when(str_detect(stimulus, "You <strong>have no idea</strong> she is allergic to peanuts") ~"allergy",str_detect(stimulus, "<strong>You do not know that someone replaced the sugar with poison.</strong>") ~"poison",str_detect(stimulus, "<strong>The next day, you end up talking, and you discover that you are actually long lost siblings.</strong>") ~"sibling_incest",str_detect(stimulus, "You <strong>have no idea</strong> that this person is actually your biological parent") ~"parent_incest",str_detect(stimulus, "It is <strong>incorrectly labeled “beef”") ~"eating_dog",str_detect(stimulus, "Your friend <strong>forgets to tell you</strong> that the cup contains her urine for her pregnancy test.") ~"drinking_urine",str_detect(stimulus, "You <strong>know</strong> she is allergic to peanuts.") ~"allergy",str_detect(stimulus, "<strong>You know full well that you are putting poison in her coffee.</strong>") ~"poison",str_detect(stimulus, "That night, you end up talking, and you discover that you are actually long lost siblings.") ~"sibling_incest",str_detect(stimulus, "You <strong>discover</strong> that this person is actually your biological parent") ~"parent_incest",str_detect(stimulus, "It is <strong>correctly labeled “dog”</strong>.") ~"eating_dog",str_detect(stimulus, "Your friend <strong>warns</strong> you that the cup contains her urine for her pregnancy test") ~"drinking_urine" ) )cleaned_data <- cleaned_data %>%mutate(domain =case_when( scenario %in%c("allergy", "poison") ~"harm", scenario %in%c("sibling_incest", "parent_incest") ~"incest", scenario %in%c("eating_dog", "drinking_urine") ~"ingestion",TRUE~"unknown" ))cleaned_data <- cleaned_data %>%mutate(violation_type =case_when( domain =="harm"~"harm", domain =="incest"~"purity", domain =="ingestion"~"purity",TRUE~"unknown" ))cleaned_data <- cleaned_data %>%mutate(intent =case_when( condition =="1"~"accidental", condition =="2"~"intentional",TRUE~"unknown" ))cleaned_data <- cleaned_data %>%filter(trial_type !="survey-multi-choice")cleaned_data <- cleaned_data %>%mutate(moral_wrongness =as.numeric(str_sub(response, 7, 7)))#create subset dataframe for comparing harm vs. incest (filter domain column)dataframe_harm_incest <- cleaned_data %>%filter(domain =="harm"| domain =="incest")#create subset dataframe for comparing harm vs. ingestion (filter domain column)dataframe_harm_ingestion <- cleaned_data %>%filter(domain =="harm"| domain =="ingestion")#create subset dataframe for comparing incest vs. ingestion (filter domain column)dataframe_incest_ingestion <- cleaned_data %>%filter(domain =="incest"| domain =="ingestion")
Confirmatory analysis
We will replicate Experiment 1A from the original study. Before analysis, we will screen out repeat participants.
Our dependent variable will be the moral judgments made by participants on a 7-point scale.
First, to verify that judgments of the two stories representing each violation type did not differ, we will conduct three 2 (story) × 2 (intent) ANOVAs: for harm, incest, and ingestion.
If we replicate the findings of no main effect of story and no story-by-intent interaction, we will proceed with three 2 (intent) × 2 (domain) ANOVAs, allowing us to compare the role of intent in the following pairings: (1) harm versus incest, (2) harm versus ingestion, and (3) incest versus ingestion.
If any significant interaction effects are found, we will further conduct independent-samples t-tests to identify specific difference patterns, following the approach of the original paper.
This is the key analysis of this study.
If we replicate the original findings, we expect to see a significant main effect for both intent and domain in all three ANOVAs, with only a significant intent-by-domain interaction in the first two ANOVAs, but not in the last.
Ultimately, we aim to confirm that innocent intent plays a greater role in moral judgments of harm versus purity violations. Specifically, we anticipate that accidental harms will be judged as less morally wrong than accidental purity violations,though harms were not judged less morally wrong than purity violations in general.
cat("\n AOV comparing incest vs. ingestion")
AOV comparing incest vs. ingestion
anova_result_incest_ingestion <-aov(moral_wrongness ~ intent * domain, data = dataframe_incest_ingestion)print(summary(anova_result_incest_ingestion))
Df Sum Sq Mean Sq F value Pr(>F)
intent 1 13.33 13.333 2.530 0.131
domain 1 5.00 5.000 0.949 0.345
intent:domain 1 3.33 3.333 0.632 0.438
Residuals 16 84.33 5.271
## create bar graph in fig 2#calculate mean and standard error for moral_wrongness by scenario and intentdf_summary <- cleaned_data %>%group_by(scenario, intent) %>%summarize(mean_moral_wrongness =mean(moral_wrongness, na.rm =TRUE), # Avoid NAs in calculationse_moral_wrongness =sd(moral_wrongness, na.rm =TRUE) /sqrt(n()) # Standard error )
`summarise()` has grouped output by 'scenario'. You can override using the
`.groups` argument.
# Map violation_type based on scenarioviolation_map <-c("allergy"="harm","poison"="harm","sibling_incest"="purity","parent_incest"="purity","eating_dog"="purity","drinking_urine"="purity")df_summary <- df_summary %>%mutate(violation_type = violation_map[scenario]) # Use violation_map to assign values#create plotggplot(df_summary, aes(x = scenario, y = mean_moral_wrongness, fill = violation_type, label = scenario)) +geom_bar(stat ="identity", position =position_dodge(width =0.8)) +geom_errorbar(aes(ymin = mean_moral_wrongness - se_moral_wrongness, ymax = mean_moral_wrongness + se_moral_wrongness),width =0.2, position =position_dodge(width =0.8)) +facet_wrap(~intent, scales ="free_x") +# Split by intentional vs. accidentalscale_fill_manual(values =c("harm"="gray", "purity"="black")) +# Custom colors for violation typelabs(x ="Scenario", y ="Morality Rating (1 = Not at all morally wrong, 7 = Very morally wrong)",title ="Experiment 1: Intentional vs. Accidental") +theme_minimal() +theme(axis.text.x =element_text(size =12),axis.text.y =element_text(size =12) )
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.
