Replication of Study 1 by Hopkins, Weisberg, and Taylor (2016, Cognition)
Introduction
Research Interests
I am currently a first-year PhD Student in the Graduate School of Education in the Developmental and Psychological Sciences program, and I am interested in using transdiagnostic longitudinal neuroimaging methods to understand the developmental trajectories of individuals within and across functional areas, so as to better understand how neural development relates to educational and other well-being outcomes.
This paper, and the seductive allure effect in general, is relevant to my interest in communicating scientific findings precisely and concisely. I might also be susceptible to this very effect in my own desire for explanations as I pivot from working as a school psychologist to neuroscience research as a PhD student.
Description of stimuli and procedures
The original study aimed to investigate whether participants would prefer reductive explanations and whether this would differ by the science of the phenomenon. Horizontal explanations were defined as referring “only to the science from which the phenomenon itself was drawn (e.g., using biological language to explain a biological phenomenon)”, while reductive explanations were defined as explanations which “included information from the next lower level in the hierarchy (e.g., using chemical language to explain a biological phenomenon)”.
Participants were presented with horizontal and reductive explanations of phenomenon in 6 sciences: physics, chemistry, biology, neuroscience, psychology, and social sciences. The stimuli were carefully constructed so that the reductive information would not provide any explanatory power, allowing the authors to separate out a potential preference for explanations with a reductive form even in the absence of any helpful reductive content.
The original authors observed that there was a main effect of explanation level, where participants rated reductive explanations as of higher quality than horizontal explanations. They also observed that there was a relatively positive view of neuroscience combined with a relatively negative view of psychology, with information in these domains positively and negatively influencing the perceived quality of the reductive explanations respectively.
Challenges
Participants may approach this task, and the explanations provided, with vastly different background knowledge, as was observed in the original study. As the replication sample will be crowd-sourced online via Prolific, participants may be motivated to complete the task as fast as possible rather than attempt to maximise their performance on the task, even with the instructional check at the start of the experiment and the attentional check in the middle of the Rating Explanations task.
Link to GitHub repository and original paper
The repository for this rescue project is located at https://github.com/psych251/hopkins2016_rescue, and the original paper
Hopkins, E. J., Weisberg, D. S., & Taylor, J. C. V. (2016). The seductive allure is a reductive allure: People prefer scientific explanations that contain logically irrelevant reductive information. Cognition, 155, 67–76. https://doi.org/10.1016/j.cognition.2016.06.011
is hosted here.
Summary of prior replication attempt
The replication utilised the methodology exactly as reported in the original study, and found that …
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
How much power does your planned sample have for original effect? For an attenuated effect that is half the size of the original?
(If power analysis is not possible or precise, discuss more fully how you determined a sample size that would be sufficient for rescue.)
Planned Sample
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
Materials
All materials can be found in Appendix A of the original article and will be used without modifications.
Procedure
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Controls
What attention checks, positive or negative controls, or other quality control measures are you adding so that a (positive or negative) result will be more interpretable?
Analysis Plan
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study and 1st replication
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
Results of control measures
How did people perform on any quality control checks or positive and negative controls?
Confirmatory analysis
The analyses as specified in the analysis plan.
Three-panel graph with original, 1st replication, and your replication is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Mini meta analysis
Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.