Replication of Study 4 by Rattan, Good & Dweck (2012, Journal of Experimental Social Psychology)

Introduction

I chose to replicate Study 4 from Rattan et al. (2012) because my research interests center on formative assessment in mathematics education, particularly the mechanisms through which feedback influences students’ motivation and learning. Much of the existing research on formative assessment comes from educational interventions in authentic classroom settings, which are ecologically valid but less controlled. Experimental studies that isolate the effects of different types of feedback are rare, making this work especially relevant. This study is, to the best of my knowledge, the only one that experimentally examines how variations in feedback (comfort-oriented, strategy-oriented, or neutral) shape students’ motivation, beliefs about ability, and expectations. Replicating this design allows me to understand more about the gap between intervention-based research and more controlled experimentation, providing insight into the psychological mechanisms underlying formative assessment practices.

The stimuli in this experiment consist of written scenarios and feedback scripts. Participants are asked to imagine themselves as students in a calculus course who have just received a disappointing test score (65%). After reading an initial statement of support, they are randomly assigned to receive one of three types of feedback: (a) comfort-oriented feedback emphasizing general strengths and minimizing demands, (b) strategy-oriented feedback suggesting concrete steps to improve, or (c) control feedback expressing care without specific guidance. Following this manipulation, participants complete scales assessing their perceptions of the instructor’s implicit theory of math ability, the professor’s expectations and investment, their own motivation, and anticipated performance.

I anticipate three main challenges in conducting this replication. First, I will need to adapt the materials into an online format and ensure random assignment is implemented correctly, which requires technical accuracy in programming the survey tool, specially considering this will be my first time conducting an experiment of this kind. Second, I will need to secure a participant sample large enough to provide adequate statistical power, which will be challenging since I will not be using a Prolific sample and will instead need to recruit students in order to replicate the study with fidelity. Third, interpreting the results in relation to my broader research program will require careful consideration, since I will have to connect findings from a controlled and somewhat artificial setting to the complex realities of formative assessment in classrooms.

Repository and Paper Links: - https://github.com/psych251/rattan2012 - https://github.com/psych251/rattan2012/blob/main/original_paper/rattan2012.pdf

Methods

Power Analysis

Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.

Planned Sample

Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.

Materials

All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.

Procedure

Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.

Analysis Plan

Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.

Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.

Differences from Original Study

Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

Data preparation following the analysis plan.

Confirmatory analysis

The analyses as specified in the analysis plan.

Side-by-side graph with original graph is ideal here

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.