Replication of Study 1 by Mani, Mullainathan, Shafir, & Zhao (2013, Science)
Author
Vivian Huynh (vivhuynh@stanford.edu)
Published
December 14, 2023
Introduction
In the original study “Poverty Impedes Cognitive Function,” by Mani, Mullainathan, Shafir, and Zhao, researchers hypothesized that poverty has a direct effect on cognitive functions and conducted two studies to test their hypothesis. In these studies, it was evident that the cognitive capacities of low-income participants as opposed to high-income participants varied significantly when affected by financial situations. Researchers also examined farmers during harvesting season to test their cognitive function before they were paid and were temporarily “poor,” and after they were paid. Mani, Mullainathan, Shafir, and Zhao claim that poverty itself “reduces cognitive capacity,” as a result of the financial stress that consumes cognitive resources, which further perpetuates poverty in the poor–financially, mentally, and behaviorally.
As an undergraduate in Symbolic Systems with a concentration in Cognitive Science, the idea of limited cognition as a result of poverty interests me because of the intersectionality of being a first generation, low-income student that studies the brain. Upon reading this article, skepticism definitely arose because being low-income or growing up in poverty does not necessarily imply that people in these circumstances will be less successful, however, there is truth and evidence that demonstrates how poverty structurally–and even generationally–affects the cognition in those living in poverty. There are many factors that impact low-income populations aside from finances, and this article provides evidence of the behaviors and thought processes as a result of limited cognition from poverty itself.
To replicate this experiment, we will conduct two studies: the first study will randomly assign low-income and high-income participants to an easy condition or hard condition, with the conditions varying from a low cost or high cost hypothetical financial scenario. We will be using Raven’s Progressive Matrices Test to measure participants’ fluid intelligence through non-verbal, analytical skills and a spatial compatibility test in each condition, as the scenarios are meant to evoke thoughts of financial concern. The second study will observe the cognitive capacities of Qualtrics participants before and after they are paid, to mimic the study of farmers. The target population will be people who are paid at the end or beginning of each month, as finances can impede cognitive function when unforeseen financial situations occur, rent is due, etc. We will also use Raven’s matrices to test and a spatial compatibility test to measure the participants’ cognitive function before and after they receive their natural income.
Challenges that may arise include ineffectiveness of hypothetical scenarios as they may not trigger financial concerns for low-income participants because the scenarios are not real. As for the second study, we might not be able to measure the cognition of farmers before and after harvesting season, but instead measure that of those who are paid in a shorter period of time (monthly). Cognition measures may be inaccurate due to external factors such as mental health, familial issues, and more, but may be excluded due to random assignment.
Links
Original paper: https://github.com/psych251/mani2013_/blob/main/original_paper/science.1238041.pdf
Methods
First, we will be using Prolific to recruit participants from different financial backgrounds, ideally ranging from $20,000 to $70,000. Although not specified in the original Mani et. al., 2013, study, we hope that our sample will have varying salaries in order to compare those in the lower end and the higher end. After recruiting participants, we will direct them to Qualtrics to retrieve survey responses on their decisions upon given hypothetical scenarios that target financial problems. To ensure that participants understand the given scenarios, we will administer comprehension-check questions after each financial scenario before proceeding to their responses. Participants will be randomly assigned either the easy condition or the hard condition of the financial scenario. Upon completing the scenarios, participants will be assigned two tasks: Raven’s Progressive Matrices and a spatial incompatibility test. These tasks will be used to measure participants’ cognitive capacities after some hypothetical hardships.
Power Analysis
The original study had an effect size of d = 0.88 with 95% power and a 0.05 aloha level. With that being said, the minimum sample size should contain at least 20 total participants in each condition (hard, easy).
Planned Sample
We hope to draw in participants of all gender identities, namely, male/female/non-binary, so long as they report their annual income. The original study does not provide an age range for participants, but we hope to recruit participants between 25-40 years old. With this age range, perhaps participants will have more financial security or a steady income, to rule our outliers that may be hopping between jobs.
Materials
As noted in our Methods, participants will be recruited using Prolific, where they will be redirected to Cognition.Run where they will undergo a pre-screening to ensure that they are not actively students. Once directed to Cognition.Run, participants will be given a consent agreement. Participants will then be tasked to contemplate and answer hypothetical financial scenarios, then assigned a non-verbal cognitive task (spatial compatibility).
Procedure
“In experiment 1, participants (n = 101) were presented with four hypothetical scenarios a few minutes apart. Each scenario described a financial problem the participants might experience. For example: “Your car is having some trouble and requires $X to be fixed. You can pay in full, take a loan, or take a chance and forego the service at the moment… How would you go about making this decision?” These scenarios, by touching on monetary issues, are meant to trigger thoughts of the participant’s own finances. They are intended to bring to the forefront any nascent, easy to activate, financial concerns. After viewing each scenario, and while thinking about how they might go about solving the problem, participants performed two computerbased tasks used to measure cognitive function: Raven’s Progressive Matrices and a spatial compatibility task. The Raven’s test involves a sequence of shapes with one shape missing (27).
Participants must choose which of several alternatives best fits in the missing space. Raven’s test is a common component in IQ tests and is used to measure “fluid intelligence,” the capacity to think logically and solve problems in novel situations, independent of acquired knowledge (28, 29). The spatial incompatibility task requires participants to respond quickly and often contrary to their initial impulse. Presented with figures on the screen, they must press the same side in response to some stimuli but press the opposite side in response to others. The speed and accuracy of response measures cognitive control (30), the ability to guide thought and action in accordance with internal goals (31). Both are nonverbal tasks, intended to minimize the potential impact of literacy skills. Upon completion of these tasks, participants responded to the original scenario by typing their answers on the computer or speaking to a tape recorder and then moved on to the next scenario (an analysis of participants’ responses to the scenarios is available in table S1). We also collected participants’ income information at the end of the experiment. Participants were randomly assigned either to a “hard” condition, in which the scenarios involved costs that were relatively high (for example, the car would require $1500 to fix); or to an “easy” condition, where costs were lower (for example, the car would require $150 to fix). Because the sums in the easy condition are small, we expected this condition to evoke few of one’s own monetary concerns, for either poor or rich participants. In contrast, the large sums in the hard condition, we hypothesized, would evoke monetary concerns in the poor but not in the rich participants.”
Our procedure is similar to that of the original study, as it will implement similar, if not same, hypothetical financial scenarios followed by one cognitive function task, the one being spatial compatibility. Depending on the program we use to conduct the cognitive function tasks, we may or may not be able to record the duration spent on each test.
Analysis Plan
We will plot the Standard Mean Error for the measured performance of each cognitive task, comparing the results of low-income and high-income participants and their accuracy of each task (Raven’s Matrices, Spatial Compatibility Task).
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Differences from Original Study
In this study, we understand that it may not be plausible to record information on the farmer population in India, so we will not be able to test the impact of financial burdens on the same participants before and after harvest, or in other words, before and after farmers get paid. However, we might be able to test the cognitive capacities of the same participants before and after they are paid, depending on the dispursement of their paycheck (weekly, bi-weekly, monthly). Perhaps there will be a difference in cognitive capacity before and after participants are paid, even if there is a shorter duration between their checks. It is notable that we will be using Qualtrics to gather survey responses, which may or may not encourage falsity in terms of income. Using an online platform may also affect the results as a whole since not everyone who is low-income or high-income may have the time to participate in this study, let alone be exposed to it.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
We want to read the dataframe from our Qualtric responses, as well as the scores received from the cognition tasks. First, we have to tidy the data and organize participants’ demographics including: age, gender identity, income, and so on. It would be valuable to do this as it will allow us to potentially make inferences about each social aspect. We will not include participants that do not answer all of the required questions on Qualtrics (like annual income), as this will affect the cognitive capacity scores due to the ambiguity of their socio-economic status.
### Data Preparation#### Load Relevant Libraries and Functionslibrary(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.3 ✔ readr 2.1.4
✔ forcats 1.0.0 ✔ stringr 1.5.0
✔ ggplot2 3.4.3 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.0
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Rows: 187 Columns: 39
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (30): trial_type, internal_node_id, source_code_version, ip, user_agent...
dbl (6): trial_index, time_elapsed, run_id, condition, accuracy, block
lgl (2): success, timeout
dttm (1): recorded_at
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Warning: Expected 4 pieces. Missing pieces filled with `NA` in 4 rows [1, 2, 3,
4].
`summarise()` has grouped output by 'PROLIFIC_PID', 'group', 'ses_group',
'block', 'income', 'Age'. You can override using the `.groups` argument.
`summarise()` has grouped output by 'PROLIFIC_PID', 'group', 'ses_group',
'income', 'Age'. You can override using the `.groups` argument.
[1] 52.25
[1] 87500
`summarise()` has grouped output by 'group'. You can override using the
`.groups` argument.
`summarise()` has grouped output by 'group', 'ses_group'. You can override
using the `.groups` argument.
`summarise()` has grouped output by 'group'. You can override using the
`.groups` argument.
Confirmatory analysis
`geom_smooth()` using formula = 'y ~ x'
Warning in qt((1 - level)/2, df): NaNs produced
Warning in max(ids, na.rm = TRUE): no non-missing arguments to max; returning
-Inf
STATISTICAL ANALYSIS
ANALYSIS OF OVERALL DATA
TWO-WAY ANOVA
accuracy_aov <- aov(mean_acc ~ group * ses_group, cleaner_data) summary(accuracy_aov)
Regression Analysis
accuracy_model <- lm(mean_acc ~ group * ses_group, cleaner_data) summary(accuracy_model)
To confirm that there are differences between low-income and high-income groups, we will be using a t-test to measure the differences of cognitive capacity scores, hoping to see variation among scores depending on the participants’ income. The t-test will allow us to confirm whether or not there is significance in our result against the null hypothesis. Mani et. al., 2013, claims that low-income populations are cognitively impacted by financial stress, as opposed to high-income populations. To test this hypothesis, researchers assigned cognitive capacity tests and well as a spatial incompatibility task to compare the differences in scores.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
TWO-WAY ANOVA
rt_aov <- aov(mean_rt ~ group * ses_group, cleaner_data) summary(rt_aov)
`geom_smooth()` using formula = 'y ~ x'
Warning in qt((1 - level)/2, df): NaNs produced
Warning in max(ids, na.rm = TRUE): no non-missing arguments to max; returning
-Inf
rt_model <- lm(mean_rt ~ group * ses_group, cleaner_data) summary(rt_model)
accuracy_original_model <- aov(accuracy ~ group * ses_group, original_data) summary(accuracy_original_model)
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.
