My current PhD research focuses on the question of how humans should assign probabilities in various contexts, all with the hope to inform practical interventions to improve human reasoning. Studies show that the so-called “Monty Hall problem is one context where humans frequently assign incorrect probabilities. This is an impotant context since it tests one’s capacity to reason properly about likelihoods, a capacity which is central to reasoning about many topics in everyday life. For that reason, I hope to repliclate a paper which describes a promising procedure for inculcating correct reasoning about the Monty Hall problem.
The procedure utilises components of a digital learning environment, and it involves comparing pretest and posttest differences on variants of the Monty Hall problem. In the experiment, subjects are randomly assigned to four conditions: the “playing only” condition, the “explanation only” condition, the “playing and explanation” condition, and the control condition which exludes both playing and explanation. In the “playing only” condition, subjects repeatedly play the Monty hall problem, receiving feedback on their performance without any explanation as to why one response–the counter-intuitive one–is the correct response to the problem. In the “explanation only” condition, subjects are walked through an explanation of the Monty Hall problem and why the correct response is indeed the correct response. The “playing and explanation” condition includes both playing and explanation components, and the control condition features neither. A pretest questionnaire is administered to rule out subjects who have prior familiarity with the problem, while a post-test questionnaire tests participants on a similar but modified variant of the Monty Hall problem to measure potential improvements in their performance.
The project may face various challenges, and which challenges it faces depends on whether the digital environment can be accessed online. If the environment can be accessed online, then the procedure may be relatively straightforward: a sample of Mechanical Turkers can be recruited and provided with links to the pre- and posttests (on Qualtrix). After pre-testing, Turkers can be sent a link to a randomly assigned condition for the digital environment (which can be programmed to shut down after a period of time); perhaps even qualtrix has the functionality to randomly generate one of four links or to provide a link to another place where these links can be randomly generated (provided that subjects are not eliminated by the pre-test, of course). However, if the environment cannot be accessed online, then it may in principle be possible to replicate the experiment with students (and this methodology more closely resembles the original protocol which involved high school students). In this case, the main challenges would be to recruit students and provide them with a computer room in which to conduct experiments. The article explicitly specifies that the researchers can be contacted to use their software, and I have emailed the lead author to inquire about more details of their software. If she responds, this should clarify which challenges this project faces, and I will email the other authors if she does not respond.
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
You can comment this section out prior to final report with data collection.
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Any differences from what was described as the original plan, or ânoneâ.
Data preparation following the analysis plan.
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Any follow-up analyses desired (not required).
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.