Replication of Critical Mazes Recall Experiment from Mark Ho et al. (2022). People construct simplified mental representations to plan. Nature.
Introduction
I will be replicating the results shown in figure 3 of the paper People construct simplified mental representations to plan by Mark Ho et al. (2022).
In this paper, the authors propose task construal – a computational framework of planning that allows for the construction of simplified task representations prior to planning, claiming that this capacity to control our mental representations better mirrors human behavior. I am interested in understanding how we might construct and use flexible representations for mental simulation, so this work in the planning domain is highly relevant. In particular, their use of memory traces as a tool to probe whether an object was represented during planning is is methodologically relevant to my work as I look for ways to probe the content of our mental representations in my research.
For this replication, I will recreate the critical mazes recall experiment where the participants are first asked to navigate a maze composed of tetromimo-shaped obstacles, and then probed on their awareness of each obstacle after completing the navigation. Specifically, I will be reproducing the finding on figure 3a, where we plot a histogram of the participants’ mean awareness of each obstacle, splitting on whether or not the value-guided construal model assigned less than or greater than 50% probability of including the obstacle in the constral.
Rather than using base psiTurk and jsPsych (v.6.0.1), I will be using Prolific and migrating the code to the most recent version of jsPsych (v.7.3.3) for ease of future use. Rather than regenerating the scores from the computational models, I will use the same stimuli in the original paper. To generate the figures, I will directly compare the results with the model outputs from the original paper (i.e., I will not be recreating the model from scratch). The primary challenge will lie in recreating the behavioral experiment, as the plots comparing model and human performance are easier to reproduce using existing model outputs.
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
Materials
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Procedure
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Analysis Plan
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation following the analysis plan.
Confirmatory analysis
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.