Replication of Study X by Sample & Sample (20xx, Psychological Science)
Introduction
[No abstract is needed.] Each replication project will have a straightforward, no frills report of the study and results. These reports will be publicly available as supplementary material for the aggregate report(s) of the project as a whole. Also, to maximize project integrity, the intro and methods will be written and critiqued in advance of data collection. Introductions can be just 1-2 paragraphs clarifying the main idea of the original study, the target finding for replication, and any other essential information. You will likely contact the authors of the original study, so this should be professional and forward facing. It will NOT have a literature review – that is in the original publication. You can write both the introduction and the methods in past tense.
This paper was one of the inspirations for my first year project. It includes a social implicit learning paradigm, which is what I am doing in my first year project. As the title of the paper suggests, it looks at social intuition, which is my central research interest. The study is also done in neurotypical participants but administers the autism quotient to look at correlations between autistic traits and other variables of interest. This makes the study more practical for an in class replication on a short timeline. I plan to get the face morphs used as stimuli for the replication in order to make this project much easier. I have some concerns about whether I will be able to get the stimuli since Leslie thinks someone else in our lab may have asked them for their stimuli and never got a response. If I cannot get the face stimuli, I will probably have to pivot, but I’ve sent out the email request and we’ll see what happens. Another benefit of this study is that it doesn’t use any skills that are too far out of my reach, but still presents some opportunities for growth.
In the acquisition phase, face morphs of dynamic facial expressions (happy or angry) in which their gaze shifts towards or away from the participant will be shown to participants across multiple trials. All faces are either identity A or B. Identity A and B are showing either a positive or negative disposition towards the participant. In the test phase, implicit learning was tested via morphs of the facial expressions of both identities put together, progressively shifting towards more of one identity than the other, while participants indicated whether the identity was closer to A or B. The smiling and the frowning faces were tested separately. The idea was that if the participant had learned the disposition of the two identities, they would be able to indicate whether the face was more A or B.
I will give 56 trials split into two blocks of 28 trials in the acquisition phase. The response in the test phase will ask participants to press “f” or “j” to indicate that they think the face is more identity A or B after viewing the video clip on each trial. In the nonsocial task, participants will do the same thing except identities A and B are actually circle vs. square.
The test phase will be four blocks of ten trials. It starts with 60% maximally smiling agent A and 40% maximally smiling agent B and progresses in steps of 5% towards 60% maximally smiling agent B, which only takes 5 trials. I’d like to give participants two rounds of the smiling trials. Then the same thing is repeated for the frowning faces, and I would give two rounds of that as well. Their accuracy scores in the test phase will serve as the measure of their implicit learning. At the end, participants will be asked questions probing whether they detected the contingencies in the experiment and if they did their data will be excluded.
In the original experiment, there is also a nonsocial task, but that will not be replicated due to feasibility on a very short timeline.
I attempted to get the face stimuli from the authors but was not successful. Thus the remaining tasks are to create the sitmuli and code the experiment to run the face stimuli correctly. I would likely use jspsych for that, which I anticipate will pose slight challenges but nothing I can’t overcome, especially with the support offered in class.
Do we need to add to the introduction?
Methods
60 adult participants were recruited for this study. The study was run on Prolific and participants were compensated for their time. The experiment took approximately 10-15 minutes to complete. Google Gemini was used to generate the face morph stimuli. The experiment was coded in JsPsych.
There were two blocks of 28 trials in the acquisition phase. In the acquisition phase, the happy face condition started with gaze forward, smiling and gradually shifted to gaze away from the participant, frowning. The angry condition started with gaze forward, frowning and gradually shifted to gaze away from the participant, smiling. The clips were 2s long.
There were four blocks of 10 trials in the test phase. The test phase began with a combined face of 60% maximally smiling agent A and 40% maximally smiling agent B and progressed in steps of 5% towards 60% maximally smiling agent B. Participants saw this four times. Then the same thing was repeated for the frowning faces, and the participants saw this four times as well. At the end, participants were asked questions probing whether they detected the contingencies in the experiment and if they did their data was excluded.
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
A power analysis run using a t-test in RStudio and the smallest original main effect of ηp^2 = 0.12 (disposition) found that I would need 547 participants to achieve a power of 80%. This was not feasible for my timeline so I instead ran the same test using the main effect of proportion (ηp^2 = 0.35), which returned n = 66. Therefore, 75 participants were recruited to leave room for unusable data. I recognize this may mean I will not have enough power to detect smaller effects, but recruiting 547 participants is not an option on my timeline.
Planned Sample
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
Materials
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Procedure
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Analysis Plan
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
To prepare the data, I used RStudio. In RStudio, I first loaded the relevant libraries, such as tidyverse. Then, I imported the data from Profilic. Participants who detected the contingencies from the experiment were identified from the questionnaire and their data was excluded from further analysis. In the dataframe (tibble), columns for participants, responses, and correct responses were created. Additionally, each row was a different participant.
Confirmatory analysis
The analyses as specified in the analysis plan.
A two-way repeated measures ANOVA with DISPOSITION (2 levels) and PROPORTION (5 levels) as factors was performed on participant responses to the social task. This test was used because I needed to compare the means of multiple groups to see if there was a significant difference between them. A correlation was tested between the questionnaire on autistic traits and responses to the social task. This correlation test was completed because I was looking at the potential connection between autistic traits and performance on the task.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.