Replication of ‘Latent motives guide structure learning during adaptive social choice’ by van Baar et al. (2022, Nature Human Behavior)
Introduction
For this project, I propose to replicate a portion of Experiment 1 from van Baar et al. (2022). Its findings demonstrate that people make predictions about the behavior of others by detecting and leveraging their latent motives. This paper is of import to my honors thesis, which explores how humans learn information that allows them to successfully generalize about others. In my thesis, I will run a behavioral experiment which may utilize these same economic games and a similar prediction task. I would run this experiment and, due to the constraints of the class timeline and my experience, conduct its analyses up until the computational modeling component.
In this experiment, in order to determine their decision-making strategy, participants first indicate how they would play each of four economic game types (the stimuli in this experiment). Then, they play four blocks of the “Social Prediction Game,” wherein in each of sixteen trials they predict what a (experimenter-generated) player would choose in these economic games and rate their confidence in their prediction. At the end, they self-report what they think the player’s strategy was in a free response format. Using t-tests, the prediction accuracy of participants is compared to what it would be under a few potential learning strategies humans may use to investigate how plausible they are. I plan to test some of the more basic hypotheses that experimenters found evidence against here. More specifically, they found that participants don’t 1) simply expect players to repeat their past behavior, 2) refrain from generalizing across trials, or 3) engage in a form of “naive statistical learning.”
The main challenge in replicating this experiment will be coding up the back-end. I already have experience with creating the front-end of a behavioral experiment through the progress I’ve made toward my honors thesis so far, but I’m currently in the process of learning this other component.
Relevant Links
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
Materials
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Procedure
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Analysis Plan
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
Confirmatory analysis
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.