Introduction

Justification for choice of study

For this project, I will be replicating Mani et al.’s 2013 Science paper that looked at how economic scarcity affected cognitive function in adults. I chose to replicate this paper because I am interested in researching how poverty-related stressors affect brain development and learning. The results of this paper provided a causal description of an environmental impact on cognition as a result of financial stress, which, to the best of my knowledge, has not yet been examined in school-aged youth. However, before examining whether or not economic scarcity can also affect cognitive systems integral to learning in children and adolescents, I want to first examine the validity scarcity in an adult population. If my replication study is able to successfully reproduce similar results as the original paper, then I hope that this can be further investigated in younger populations.

Description of stimuli and procedures

In this study, participants will first be randomly assigned to review and respond to a series of financially difficult or easy scenarios. Afterward a brief break, they will participate in two cognitive tasks: the first will compose of a series of Raven’s matrices puzzles, often used as a proxy for fluid reasoning; the second task will gauge cognitive control using the spatial incompatibility task. In the spatial incompatibility task, participants will view objects on one side of the screen and will either have to press a button if it is on the same or opposite side depending on the trail’s instructions. Participant’s cognitive control will be measured based on both speed and accuracy.

Anticipated challenges

There are a few challenges when replicating this particular study using an online format. First, in the original study, participants were shoppers at a New Jersey mall, and their data will be compared to participants on Prolific. This could potentially affect replicating the results of the study given the convenience sampling of the original paper. Additionally, the materials—both the original paper and the supplemental materials—do not explicitly mention in sufficient detail how they ensured participants comprehended and carefully examined each scenario. There is also no mention as to how long they had to review them. As a result, it may be difficult to compose similar comprehension checks and induce a scarcity effect via a remote format. Furthermore, with regards to the analysis, it is not quite clear how they grouped participants into “rich” or “poor” categories in the 2013 paper. Prior to running an analysis, I would like to pre-register this decision as to not effect the final results, but how to carefully and properly label someone into these two groups could affect the final results.

Methods

Power Analysis

With an original effect size of approximately d = .88 with 95% power and a .05 aloha level, the planned minimum sample size is 20 total participants.

Planned Sample

Participants from low and high SES backgrounds will be randomly assigned to either a hard or easy condition. In the original study, 101 adults (mean age = 35.3 years; 65 F) participated in their first experiment. While it is not explicitly clear how many participants were in each group, it is highly likely that approximately 50 participants were divided between each condition, and then further divided between high or low SES for a total of 25 participants per group.

Therefore, at the very least, I anticipate to recruit 25 participants per group (50 low and 50 high SES participants each randomly assigned to hard or easy conditions). Additionally, there will be a preference for adults around the age of 35.

Materials

All participants will be recruited using Prolific, a platform to recruit participants for online research. After providing informed consent, scenarios and subsequent comprehension check will be assessed on Qualtrics. After submitting the survey, participants will be automatically sent to Cognition.Run where they will take a behavioral task based of Diamond et al. (2003) which was programmed using jsPsych (de Leeuw, 2015).

Procedure

“In experiment 1, participants (n = 101) were presented with four hypothetical scenarios a few minutes apart. Each scenario described a financial problem the participants might experience. For example:”Your car is having some trouble and requires $X to be fixed. You can pay in full, take a loan, or take a chance and forego the service at the moment… How would you go about making this decision?” These scenarios, by touching on monetary issues, are meant to trigger thoughts of the participant’s own finances. They are intended to bring to the forefront any nascent, easy to activate, financial concerns. After viewing each scenario, and while thinking about how they might go about solving the problem, participants performed two computer-based tasks used to measure cognitive function:Raven’s Progressive Matrices and a spatial compatibility task. The Raven’s test involves a sequence of shapes with one shape missing (27) Participants must choose which of several alternatives best fits in the missing space. Raven’s test is a common component in IQ tests and is used to measure “fluid intelligence,” the capacity to think logically and solve problems in novel situations, independent of acquired knowledge (28, 29). The spatial incompatibility task requires participants to respond quickly and often contrary to their initial impulse. Presented with figures on the screen, they must press the same side in response to some stimuli but press the opposite side in response to others. The speed and accuracy of response measures cognitive control (30), the ability to guide thought and action in accordance with internal goals (31). Both are nonverbal tasks, intended to minimize the potential impact of literacy skills. Upon completion of these tasks, participants responded to the original scenario by typing their answers on the computer or speaking to a tape recorder and then moved on to the next scenario (an analysis of participants’ responses to the scenarios is available in table S1). We also collected participants’ income information at the end of the experiment.”

Procedure will follow similar steps as the original study with a few differences given the online nature of this project. First, after reviewing each scenario, participants will complete a comprehension check to ensure that each scenario was carefully read. Since there are no time restrictions on reviewing the survey given the nature of how the task is being completed, it can be likely that I will not replicate a the results not because of a lack of a scarcity effect but rather because participants did not adequately or seriously review each of the scenarios. Therefore, to ensure each participant carefully examined them, a 4-item questionnaire will be administered asked individuals to report back values that they read. Second, I will only be comparing results from the cognitive control task (i.e., the spatial incompatibility task).

Additionally, a major change in my project is that participants will do a significant amount of trials compared to the original study. According to the supplemental materials from Mani et al. (2013), participants only completed 10 practice trials of the heart and flowers task and after answering each one trial correctly could they complete the experimental task. However, the authors report that they only completed three trials. For my task, participants will complete 3 blocks of XX trials.

Analysis Plan

Ultimately, I aim to produce two figures: a bar chart comparing overall accuracy on the cognitive control task between each group; and a scatter plot fitting a regression model for participants in each condition. Ultimately, I aim to produce two figures: a bar chart comparing overall accuracy on the cognitive control task between each group; and a scatter plot fitting a regression model for participants in each condition.

Step 1: There must be a balanced number of participants within each of my 4 groups AND there must be a balanced number of participants from various income backgrounds (i.e., not too many high SES in the easy condition).

Step 2: Will need to use reported household income with household size in order to determine income-to-needs ration (INR) and use U.S. Federal Poverty guidelines to determine if an individual participant is high or low SES. Will probably recruit in at least two waves: first recruit high SES and then recruit low SES. This will help ensure that I accomplish Step 1.

Step 3: To ensure that participants carefully read and internalized each scenario, I will ask participants to report results on a simple comprehension check. I will also use Qualtrics information to identify the length of completion to submit the survey prior. This will be paramount to ensure that participants are in a state of mind where they are thinking about their finances prior to engaging in the cognitive control task.

Step 4: For the cognitive control task, I will need to know two things: their accuracy and reaction time on each trial.

Additional analysis of interest I would like to examine is differences in reaction time between each group. Even if accuracy is intact across conditions, could scarcity also be causing people to react more slowly to the task? If so, would this manifest in longer reaction times for low-income participants in the hard condition compared to all other groups?

Differences from Original Study

The first major difference is the nature of the study: in the original experiment, participants reviewed scenarios and engaged in two cognitive tasks at a mall. For my study, all participants will engage entirely online and will only do one of the cognitive tasks – the cognitive control task by Diamond et al. (2004). Given the online nature, participants will also complete a comprehension check after reviewing scenarios.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

Data preparation following the analysis plan.

############################################
#####   PREPARE DATA FOR ANALYSIS     ######
############################################ 
library(tidyverse) 
## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
## ✓ ggplot2 3.3.3     ✓ purrr   0.3.4
## ✓ tibble  3.1.4     ✓ dplyr   1.0.7
## ✓ tidyr   1.1.4     ✓ stringr 1.4.0
## ✓ readr   1.4.0     ✓ forcats 0.5.1
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
#### Import data from Cognition.Run

setwd("~/Desktop/mani2013/data")
easy_data <- read_csv("easy_data.csv") # raw data files for easy condition
## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   .default = col_character(),
##   run_id = col_double(),
##   condition = col_double(),
##   trial_index = col_double(),
##   time_elapsed = col_double(),
##   recorded_at = col_datetime(format = "")
## )
## ℹ Use `spec()` for the full column specifications.
## Warning: 438 parsing failures.
## row col   expected     actual            file
##   1  -- 38 columns 37 columns 'easy_data.csv'
##   2  -- 38 columns 37 columns 'easy_data.csv'
##   3  -- 38 columns 37 columns 'easy_data.csv'
##   4  -- 38 columns 37 columns 'easy_data.csv'
##   5  -- 38 columns 37 columns 'easy_data.csv'
## ... ... .......... .......... ...............
## See problems(...) for more details.
hard_data <- read_csv("hard_data.csv") # raw data files for hard condition
## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   .default = col_character(),
##   run_id = col_double(),
##   condition = col_double(),
##   trial_index = col_double(),
##   time_elapsed = col_double(),
##   recorded_at = col_datetime(format = ""),
##   platform_version = col_double()
## )
## ℹ Use `spec()` for the full column specifications.
# filter columns to ensure DF are the same size
easy_data_filtered <- easy_data %>%
  select(c(run_id,
           trial_index,
           subject_id,
           group,
           rt,
           response,
           task,
           correct_response,
           accuracy,
           block))

hard_data_filtered <- hard_data %>%
  select(c(run_id,
           trial_index,
           subject_id,
           group,
           rt,
           response,
           task,
           correct_response,
           accuracy,
           block))

data <- rbind(easy_data_filtered, hard_data_filtered) # merge into one file

#### DATA CLEANING AND FILTERING

# remove JSON text from survey responses
data$response <- gsub('\\{"Q0":"','', data$response)
data$response <- gsub('"\\}', '', data$response)

data_filtered <- data %>% 
  filter(group == "EASY" |
           group == "HARD")


#### INCOME AND SES ANALYSIS  

# determine SES groups using federal poverty guidelines
income_data <- data_filtered %>%
  select(c(run_id,
           task,
           response)) %>%
  filter(task == "household_size" |
           task == "income") %>%
  pivot_wider(names_from = "task", 
              values_from = "response") %>%
  mutate(ses_group = case_when(as.numeric(income) < 12880 & as.numeric(household_size) == 1 ~ "POOR",
                               as.numeric(income) < 17420 & as.numeric(household_size) == 2 ~ "POOR",
                               as.numeric(income) < 21960 & as.numeric(household_size) == 3 ~ "POOR",
                               as.numeric(income) < 26500 & as.numeric(household_size) == 4 ~ "POOR",
                               as.numeric(income) < 31040 & as.numeric(household_size) == 5 ~ "POOR",
                               as.numeric(income) < 35580 & as.numeric(household_size) == 6 ~ "POOR",
                               as.numeric(income) < 40120 & as.numeric(household_size) == 7 ~ "POOR",
                               as.numeric(income) < 44660 & as.numeric(household_size) == 8 ~ "POOR",
                               as.numeric(income) < 49200 & as.numeric(household_size) == 9 ~ "POOR",
                               as.numeric(income) < 53740 & as.numeric(household_size) == 10 ~ "POOR",
                               as.numeric(income) > 12880 & as.numeric(household_size) == 1 ~ "RICH",
                               as.numeric(income) > 17420 & as.numeric(household_size) == 2 ~ "RICH",
                               as.numeric(income) > 21960 & as.numeric(household_size) == 3 ~ "RICH",
                               as.numeric(income) > 26500 & as.numeric(household_size) == 4 ~ "RICH",
                               as.numeric(income) > 31040 & as.numeric(household_size) == 5 ~ "RICH",
                               as.numeric(income) > 35580 & as.numeric(household_size) == 6 ~ "RICH",
                               as.numeric(income) > 40120 & as.numeric(household_size) == 7 ~ "RICH",
                               as.numeric(income) > 44660 & as.numeric(household_size) == 8 ~ "RICH",
                               as.numeric(income) > 49200 & as.numeric(household_size) == 9 ~ "RICH",
                               as.numeric(income) > 53740 & as.numeric(household_size) == 10 ~ "RICH"))

income_data$ses_group <- factor(income_data$ses_group, levels = c("POOR",
                                                                  "RICH"))

data_filtered <- merge(data_filtered, income_data, by = c('run_id')) # add to main dataframe

############################################
###########   TASK ANALYSIS    #############
############################################ 
task_perf <- data_filtered %>%
  filter(task == "response")

# RT and accuracy need to be numeric
task_perf$rt <- as.numeric(task_perf$rt)
task_perf$accuracy <- as.numeric(task_perf$accuracy)

# create summary statistics
task_perf <- task_perf %>%
  group_by(group, ses_group) %>%
  summarize(mean_acc = mean(accuracy),
            mean_rt = mean(rt),
            acc_sd = sd(accuracy),
            acc_n_obs = length(accuracy),
            acc_sem = acc_sd / sqrt(acc_n_obs),
            acc_ci = acc_sem * 1.96,
            rt_sd = sd(rt),
            rt_n_obs = length(rt),
            rt_sem = rt_sd / sqrt(rt_n_obs),
            rt_ci = rt_sem * 1.96)
## `summarise()` has grouped output by 'group'. You can override using the `.groups` argument.
############################################
######   ACCURACY AND RT FIGURES     #######
############################################  
# Create Figure 1 - Accuracy Bar Chart
figure1 <- ggplot(task_perf, aes(x = ses_group, y = mean_acc, 
                                 fill = group)) +
  geom_bar(position="dodge", stat="identity") + # ADD COLOR-BLIND PALETTE
  geom_errorbar(aes(ymin = mean_acc - acc_ci, 
                    ymax = mean_acc + acc_ci), width=.2,
                position=position_dodge(.9)) +
  ggtitle("Figure 1: Accuracy by Group")
figure1

The following code below will run once there is a full dataset but since Figure 1 can be produced that means that dataframe we need to run the analysis works!

# Create Figure 2 - Accuracy Regression Chart
figure2 <- ggscatter(task_perf, x = income, y = mean_acc,
          color = "condition", palette = "", 
          add = "reg.line", conf.int = TRUE) + # add reg line
  stat_cor(aes(color = condition)) + # add correlation coefficient
  ggtitle("Figure 2: Accuracy Scatterplot") +
  labs(y = "Mean Accuracy", x = "Income (in thousands)")
figure2

# Create Figure 3 - RT Bar Chart
ggplot(task_perf, aes(x = ses_group, y = mean_rt, 
                         fill = group)) +
  geom_bar(position="dodge", stat="identity") +
  scale_fill_brewer(palette="") + # ADD COLOR-BLIND PALETTE
  geom_errorbar(aes(ymin = mean_rt - acc_ci, 
                    ymax = mean_rt + acc_ci), width=.2,
                position=position_dodge(.9)) +
  ggtitle("Figure 3: RT by Group") 

# Create Figure 4 - Accuracy Regression Chart
ggscatter(task_perf, x = income, y = mean_rt,
          color = "group", palette = "", 
          add = "reg.line", conf.int = TRUE) + # add reg line
  stat_cor(aes(color = group)) + # add correlation coefficient
  ggtitle("Figure 4: Reaction Time Scatterplot") +
  labs(y = "Mean RT", x = "Income (in thousands)")

############################################
#######    STATISTICAL ANALYSIS     ########
############################################ 

### t.tests 

t.test(task_perf$group, task_perf$mean_acc)

### TWO-WAY ANOVA
accuracy_aov <- aov(mean_acc ~ group * ses_group,
                    task_perf) #same thing as reg?

rt_aov <- aov(mean_rt ~ group * ses_group,
              task_perf) #same thing as reg?

### Regression Analysis
accuracy_model <- lm(mean_acc ~ group * ses_group, 
                     task_perf)

reactiontime_model <- lm(mean_rt ~ group * ses_group, 
                         task_perf)

Confirmatory analysis

My analysis will be successful if I am able to reproduce the following:

  • No significant differences emerge between participants in the easy condition based on a t-test
  • A t-test reveals that low SES participants in the hard condition will have lower accuracy
  • A two-way analysis of variance (ANOVA) will highlight a robust interaction between income and condition
  • Regression models will confirm that as income will be predictive of accuracy for participants in the hard condition only (and not predicitve for participants in the easy condition)

Update on 11/24/2021

The study is (now!!) ready for Prolific! In the end, we decided to forgo using Qualtrics entirely and are simply running the entire experiment on Cognition.Run (i.e., jsPsych). This platform will be able to extract URL parameters to capture participant IDs — but I also will not need that, since each participant will have their own run ID provided through Cognition.Run. As a result, I will be able to have all the data that I’ll need to run my analysis

Based on TA feedback from Check 2, I decided to make the review of the scenarios shorter — from one minute to 30 seconds. Also, a TA caught that there were delays with images loading, so we made that fix to avoid having any glitches with images loading throughout the task.

Side-by-side graph with original graph is ideal here

Exploratory analyses

After careful consideration, I have decided to make alterations to the task design slightly to examine a few ideas based on a more careful review of the task design. First, I have decided to have participants do 40 trials instead of 3 trials split into 4 blocks (each with 10 trials). I plan to analyze data based on blocks to see how each participant performs throughout the task. However, I will also compare participant’s overall performance with their average performance on the first three trials to replicate the original analysis. This addition to the statistical analysis will be the second change from the original study.

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.