Replication of Study 1 by Corps & Rabagliati (2020, Journal of Memory and Language)
Introduction
Justification
In what form predictions occur remains an unsolved problem in the field of speech comprehension. Corps and Rabagliati (2020) revealed that predictions about semantic content, instead of word forms, facilitated the perception of distorted speech in Experiment 1. This closely aligns with my research interest in how top-down information facilitates speech comprehension in challenging scenarios. Another reason for choosing this study is that the context manipulation used in Experiment 1 is relatively new and was proposed to better separate word-form prediction from semantic content prediction. Thus, it is worthwhile to replicate these findings. In addition to these theoretical motivations, there are two practical considerations. First, the original study was conducted on Prolific Academic, which makes the project more feasible. Second, the procedure was relatively well-documented in the paper, with stimulus list and analysis scripts shared on OSF, which makes it easier to adhere to the original procedure and analysis as closely as possible.
Stimuli and Procedures
The stimuli consist of 30 auditorially presented questions and 30 distorted answers. For the procedure, I will strictly follow that described in the original paper (Experiment1/Method/Procedure).
I anticipate two main challenges. The first concerns the stimuli. The authors have not made the stimuli publicly available. Moreover, the original study was conducted in the United Kingdom, where the low-level acoustic features are different. That might lead to perceptual differences after applying the noise-vocoding. This means I will need to record new auditory stimuli and apply the noise-vocoding. However, the script for that procedure was not shared. One solution is to reach out to the original authors. The second challenge concerns the programming of the experiment, since I have no experience with jsPsych.
Repo Link
(If you open it from RPub, please right click the hyperlink. That should work!) Repository
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
Materials
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Procedure
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Analysis Plan
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
Confirmatory analysis
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.