Vasilyeva, Blanchard, & Lombrozo (2018, Cognitive Science) investigated people’s evaluations of causal relationships based on stability - that is, whether a causal relationship holds across possible moderating variables. Across three experiments, Vasilyeva et al. demonstrated that adults were more likely to endorse stable causal claims over unstable causal claims (i.e. that don’t hold across a potential moderator). Particularly in Experiment 3 (the target of this replication), adults not only endorsed stable causal claims over unstable causal claims, but also changed whether or not they would consider a medical intervention for themselves based on the presence of a moderating variable.
One key interest of mine is childrens’ developing causal reasoning - how do children reason about causal systems and explanations? A key part of this research program will also be characterizing the adult state and asking how adults reason about the same phenomena and questions. Replicating an experiment from this paper would help me better understand the design and analyses associated with asking adults about their judgments centered around important aspects of a causal relationship. This work could extend into further replications (i.e. of the other experiments in this paper) and/or further questions (developmental or otherwise) that would be very relevant to my research interests.
I plan on replicating the result from Experiment 3 - that people were more likely to endorse a causal relationship when the relationship was non-moderated vs. moderated, and that they were more likely to consider an intervention for themselves when that same relationship was non-moderated vs. moderated. I chose this result for two key reasons: first, the design for Experiment 3 was simpler than Experiment 2 (my original desired replication) - its analysis was a simple independent-samples t-test while the latter tested for three way interactions. I (guided by the TAs) was concerned about power (keeping in mind class budget), and chose Experiment 3 over Experiment 2. Second, it tested for a key practical aspect of people’s judgments about causal relationships - whether or not they would endorse a relationship to benefit themselves. This extended the previous findings beyond trivial endorsements of causal claims (i.e. about aliens) into participants’ (presumably) real-life judgments.
The study will be run on Prolific with adults in the United States. The task is simple - participants will read blocks of text describing some researchers’ investigation of a medical intervention, and then answer a few questions about what they have read in the text. There are no other materials other than the text from the questions and scenario. I will obtain the exact stimuli (scenario and question wording) from the researchers, build a Qualtrics survey for the task, and analyze the data using R. For this, I will need funds to run participants on Prolific. One key concern about this experiment in particular is the exclusion rate - the authors of the original paper excluded over 1/3 of their tested participants (they note that it may be due to the lack of visual aids and long scenario text in this task). I will keep an eye on this exclusion rate and discuss with the TAs and Mike to see if it constitutes a bigger issue for carrying out the project.
There appears to an OSF link on the paper, but it is broken. A further search shows that the data file for this experiment on OSF is empty - there are no open materials from this experiment online. I will have to contact the researchers to request more information about administration of the task, clarification on the exact study design, the stimuli and question wording and coding (beyond what is written in the paper, which spells out most of the wording already), and clarification on their analyses. Obtaining a near exact replication may be hard if obtaining these materials is difficult.
I hope to also conduct additional exploratory analyses (TBD) to push beyond the very simple analyses used for this experiment in the paper.
Project repository (on Github): https://github.com/psych251/vasilyeva2018.git
Original paper (as hosted in your repo): https://github.com/psych251/vasilyeva2018/blob/main/original_paper/Vasilyeva_Blanchard_Lombrozo_2018.pdf or https://onlinelibrary.wiley.com/doi/10.1111/cogs.12605
Please describe all the steps necessary to reproduce the key result(s) of this study.
Explicitly describe known differences in the analysis pipeline between the original paper and yours (e.g., computing environment). The goal, of course, is to minimize those differences, but differences may occur. Also, note whether such differences are anticipated to influence your ability to reproduce the original results.
Please describe the outcome measure for the success or failure of your reproduction and how this outcome will be computed.
Earlier in this report, you described the steps necessary to reproduce the key result(s) of this study. Please describe your progress on each of these steps (e.g., data preprocessing, model fitting, model evaluation).
Data preparation following the analysis plan.
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
###Exploratory analyses
Any follow-up analyses desired (not required).
Open the discussion section with a paragraph summarizing the primary result from the key analysis and assess whether you successfully reproduced it, partially reproduced it, or failed to reproduce it.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis of the dataset, (b) assessment of the meaning of the successful or unsuccessful reproducibility attempt - e.g., for a failure to reproduce the original findings, are the differences between original and present analyses ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the reproducibility attempt (if you contacted them). None of these need to be long.