Replication of Edmiston & Lupyan (2015) by Sample & Sample (20xx, Psychological Science)
Introduction
For the final project, I propose to replicate Edmiston & Lupyan (2015). This study examines the difference in which verbal labels and environmental sounds activate categories in the mind. I have a background in linguistics and cognitive science, and I am particularly interested in language and auditory processing. I believe this study has interesting insights into how human language is unique compared to other forms of informative inputs and theorizes how broad, verbal labels contributes to the efficiency in which humans process language. These findings may have potential implications in how natural language processing technologies, like large language models and speech recognition software, can be built to model more naturalistic language learning.
The stimuli presented in experiment 1A of this study involve six categories, such as bird, dog, and phone. For each category, two environmental sound cues were obtained, each corresponding to a more specific exemplar of that category. For instance, the guitar category includes the sound of an acoustic guitar strum and the sound of an electric guitar strum. Two images were obtained for each exemplar respectively for the task (e.g. two images of an acoustic guitar and two images of an electric guitar). Finally, verbal labels, spoken from both female and male speakers, were obtained for each category. The procedure is an image matching verification task. Participants are randomly presented with an auditory cue (either the verbal label or an environmental sound) and are tasked to respond “Yes” on a gaming controller if the image matches the auditory cue, or “No” if it does not. In total, participants completed 6 practice trials and 384 test trials. Verification speed was recorded.
The challenges I anticipate from replicating this procedure primarily arise from the lack of control of the experimental setting. In the original study, the experiment was performed in individual rooms and participants were equipped with headphones to minimize external noise. It would be difficult to enforce these conditions in an online setting. Furthermore, the actions involved in the matching task is fundamentally different, such that participants will likely be using their computer mouse to perform trials as opposed to the gaming controller from the original study. This may influence overall reaction time.
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
Materials
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Procedure
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Analysis Plan
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
Confirmatory analysis
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.