### Data Preparation
#### Load Relevant Libraries and Functions
library(qualtRics)
library(tidyverse)
library(effsize)
library(ggplot2)
library(lsr)
library(psych)
library(metafor)
#### Import data
file <- "data/final.csv"
root_dir <- dirname(getwd())
filepath <- paste(root_dir, "/", file, sep="")
raw_df <- read_survey(filepath)
#### Data exclusion / filtering
# Include only participants that have agreed to participate
df <- raw_df |> filter(Consent == "I agree to participate")
# Include only participants that have passed attention check
df <- df |> filter(`6_Q14_1` == 50)
#### Prepare data for analysis - create columns etc.
# Create column indicating condition
# Participants with the intervention have seen control prompt
df <- df |> mutate(condition = ifelse(is.na( `Control Prompt`), "Low", "High"))
likert_column <- c("Strongly disagree", "Somewhat disagree", "Neither agree nor disagree", "Somewhat agree", "Strongly agree")
df$Agree_1 <- factor(df$`1_Agree Statements`, levels=likert_column, labels=1:5) |> as.numeric()
df$Agree_2 <- factor(df$`2_Agree Statements`, levels=likert_column, labels=1:5) |> as.numeric()
df$Agree_3 <- factor(df$`3_Agree Statements`, levels=likert_column, labels=1:5) |> as.numeric()
df$Agree_4 <- factor(df$`4_Agree Statements`, levels=likert_column, labels=1:5) |> as.numeric()
df$Agree_5 <- factor(df$`5_Agree Statements`, levels=likert_column, labels=1:5) |> as.numeric()
# Average pain levels used within Zion's study
df <- df |>
rowwise() |>
mutate(control_pain = mean(c(Agree_1, Agree_2, Agree_3, Agree_4, Agree_5)))
# Drop entries that contain -99: -99 means seen, but unread
df_99 <- df |> filter_all(any_vars(. == -99))
df <- df |> anti_join((df_99))Replication of Economic Insecurity Increases Physical Pain by Chou et al. (2016, Psychological Science)
Introduction
This project replicates part of a behavioral psychology study which inquires about if socioeconomic status has physical effects on the body. The initial publication from Chou et al. (2016) conducted a total of six studies, and a meta-analysis on various causal pathways to explain the phenomena. This project focuses on replicating the fifth study, which asks if the degree of control one feels they have affects how much physical pain they feel. Participants are placed into conditions where they have to recall experiences when they had either a high or low sense of control. They then report the amount of physical pain they feel after the recollection period. Differences between the two conditions are then compared.
Materials and Data
Materials, data, and experimental protocols can be reached at the following links:
- Pre-Registration: https://osf.io/vmfue
- Survey: https://stanforduniversity.qualtrics.com/jfe/form/SV_b0ZGaA73CKWMnzw
- Code and Data: https://github.com/psych251/chou2016_rescue
First Replication Attempt
This project is a replication inspired by a replication conducted by Zion, 2017. In this replication, the author follows the protocol closely to the initial study described in the article; the survey material is mostly similar, and participants are recruited from the same platform (Amazon Mechanical Turk).
The author deviates by adding manipulation checks to determine if participants have a clear understanding of what it means to have a high or low sense of control. This is mainly done by asking participants five questions relating to sense of control that are included in the other studies from the initial publication. After a successful manipulation check, the author was unable to replicate the results of the initial publication; there was no significant difference between the average pain levels in each group. Zion performs further analysis by collecting from participants painkiller use and if this is a potential control for pain levels that participants report.
Summary of prior replication attempt
The first replication aimed to keep their iteration as close to the initial study as possible, given the information in the article. At a high level, the replication project recieves samples through Amazon Mechanical Turk (MTurk), then assigns them to either the high or low control conditions. They then measure average reported pain levels within each group, and conducted a t test and determined effect size using cohen’s d.
The replication study mainly deviates from the initial study within their procedures and, subsequently, their analysis. At the end of their study, the replication asks five additional questions which “captures respondents’ psychological sense of lacking control.” The average of those questionsis used as an additional control within the analysis. Additionally, the replication asks participants if they have used painkillers and the type/frequency of them.
Within the analysis, the replication observes the group averages of the psychological sense of lacking control to ensure the treatment condition actually intervenes on a sense of control, rather than intervening on the specific instances participants are thinking of, which generally could be more painful independent of the amount of control they have.
Another factor to note is the replication increases the sample size, since the initial study was under-powered.
Both the replication and the initial study do not seem to take into consideration the painkiller use within the random treatment of groups. If it is believed that painkillers can alter a participant’s repsonse to the questions of pain, it would be important to know how many people per group are taking painkillers. This would introduce painkillers as something that could affect (mediate) the captured measure based upon group association.
Methods
Materials
The materials used in this study closely align with both the initial study’s description and replication. The following excerpts describe the materials:
“Participants were asked to write about an autobiographical experience (Whitson & Galinsky, 2008). Half recalled a time when they lacked control. The other half recalled a time when they had complete control.
After the recall task, participants were asked to “choose the overall pain level that best describes how much physical pain you are experiencing RIGHT NOW.” They responded using a visual slider scale (Portenoy & Kanner, 1996), on which 0 indicated no pain and 100 indicated the worst pain ever experienced.
Participants also reported their age, gender, and current employment status, and whether they were using painkillers.”
“Subjects then completed a short, five-item measure of control, that was an additional scale added to the author’s original procedure. See Differences from Original Study section for more details.” (Zion)
Exact survey materials can be accessed at the provided links above.
Procedure
The procedure in this replication attempt closely aligns with the initial study and replication. Participants are first asked to a set of pre-survey questions regarding how much pain they would feel experiencing a set of stimuli (e.g. getting hit by a pillow, stepping on a nail, getting hand burned, etc.) They are then are randomly assigned to Group A or Group B. In Group A (high-control), participants are asked to recount a time when they were in complete control of a situation. In Group B (low-control), participants are asked to do the opposite – namely recount a time they felt “a complete lack of control.” Both groups then spend a few minutes reflecting on these incidents, and then are asked to indicate the level of physical pain (on a scale of 0 - 100) they are experiencing. Participants are then asked to report how often they take painkillers during a month, the types of painkillers, along with demographic factors (sex, age, employment type, and race).
Controls
Within the pre-survey questions, one attention check is included which states “Please answer 50. On a scale of 0 - 100, where would you put getting hit by a ball?” Overall, the pre-survey section is a calibration and a method of understanding how a participants internalize the pain scale.
Asking each participant to then state their current level of pain pre-intervention provides a stronger baseline to the effect of the intervention.
Analysis Plan
Summary statistics for overall and per-condition are reported, as in the original study. The original study and replication study make no note of any data cleaning or exclusion rules. If participants do not pass the attention check, their input will not be included.
Average pain levels (post-intervention) will be reported, as well as regression models adding controls for age, gender, employment status, and painkiller use. The key analysis of interest is the two-sampled t-test between conditions (high sense of control vs low sense of control). Cohen’s d score is also reported.
Planned Sample
The plan is to use Prolific to recruit participants. To control for participation quality, Prolific participants are picked with high (>95%) approval ratings.
Power Analysis
The results from the initial paper compares the average pain levels between the conditions (control vs. lack of control). A two-tailed two-sample t-test is used to determine if the sample means between the two groups are statistically different from each other.
The intial paper reports a \(t=2.94\) Cohen’s d effect size of 0.41. Additionally, the first replication reports a \(t=0.55\) effect size of \(d=0.191\) with 156 samples in total (78 per group).
Post-hoc, the initial study was indeed under powered with a power of 0.53. The replication has a power of 0.93.
Running a power analysis on the initial report, the following sample sizes would have been required to gain the calculated effect size (for groups of equal sizes):
- Power = 80%: n = 95 per group
- Power = 90%: n = 126 per group
- Power = 95%: n = 156 per group
Though the replication did have a well-powered experiment, the sample size is lower than the size needed for 80% power; it is a bit conservative with respect to conducting a replication. As a result, this study aims to have \(n=125\) participants per group, which is 2.5x the original sample size. According to the power-analysis, a-priori, this is well-powered and has a substantially larger sample than the initial, which is a safe method to test the reproducibility of an experiment.
Differences from Original Study and First Replication
This replication mainly varies from the original study and first replication by asking participants pre-survey questions about pain levels. A challenge of this study seem mostly related to assessing the level of severity of their specific memory; some individuals in Group B may recall very painful memories, while some may not. It is also a challenge to determine if the pain they experience is directly attributed to the intervention, or if circumstances like environment and initial pain levels before impact the final measurement. Capturing their understanding of pain before the intervention can help to stabilize responses, and also be used for data exploration. Additionally, the first two studies recruit participants on Mechanical Turk, while this study recruits participants from Prolific. Participants within Prolific have to be within the U.S., be fluent in English, and have an approval rating of at least 95% to be in the study (matching the criteria for the original and first replication).
Methods Addendum (Post Data Collection)
Actual Sample
A total of 266 participants were recruited for the study. In total, 252 participants passed attention checks, and were used in the analysis. More participants were recruited in order to get the desired amount of participants per group from the power analysis.
Results
Data preparation
Survey data is read, parsed, and prepared for analysis. Participants who do not pass the attention check are removed here. Experiment condition is also indicated here.
Demographics
A total of 252 participants are included in this study. Of the participants, 144 are female, 99 are male, and 9 are other.
head(df |> count(Gender))# A tibble: 3 × 2
# Rowwise:
Gender n
<chr> <int>
1 Female 144
2 Male 99
3 Other 9Most participants are employed, but have varying occupation types.
df |> count(Employment)# A tibble: 8 × 2
# Rowwise:
Employment n
<chr> <int>
1 Not Working (looking for work) 30
2 Not working (disabled) 8
3 Not working (other) 24
4 Not working (retired) 13
5 Not working (temporary layoff from a job) 1
6 Prefer not to answer 3
7 Working (paid employee) 148
8 Working (self-employed) 25Most participants report not taking painkillers.
head(df |> count(Painkiller))# A tibble: 2 × 2
# Rowwise:
Painkiller n
<chr> <int>
1 No 205
2 Yes 47The average age of the participants are 30 years old, with a standard deviation of 12.8 years of age.
print(paste("The average age is ", mean(df$Age), " years old."))[1] "The average age is 38.5476190476191 years old." print(paste("The standard deviation of age is", sqrt(var(df$Age)), " years old."))[1] "The standard deviation of age is 12.8597132699053 years old."The racial breakdown within the study is a fairly mixed demographic.
df |> count(Race)# A tibble: 6 × 2
# Rowwise:
Race n
<chr> <int>
1 American Indian or Alaska Native 2
2 Asian 19
3 Black or African American 23
4 Hispanic 29
5 Some Other Race 6
6 White 173Key Analysis: Pain Levels Amongst Sense of Control
The average pain levels are reported after being prompted to think about a time of either high or low control. 131 participants were asked to think about a time when they had a high sense of control, while 121 participants were asked to think about a time when they had a low sense of control.
df |> group_by(condition) |> summarize(avg = mean(`Pain Level_1`), size = n())# A tibble: 2 × 3
condition avg size
<chr> <dbl> <int>
1 High 10.4 131
2 Low 11.5 121 boxplot(`Pain Level_1` ~ condition, data = df)
The low control condition has an average pain rating of 11.52. The high control condition has an average pain rating of 10.40. The t-test fails to reject the null hypothesis; there is not sufficient evidence to support that the average ratings between the two groups are different. The effect size is reported to be -0.064 with a 95% confidence interval of (-0.31, 0.18).
control <- df |> filter(condition == "Low") |> select(`Pain Level_1`)
intervention <- df |> filter(condition == "High") |> select(`Pain Level_1`)
t.test(control, intervention)
Welch Two Sample t-test
data: control and intervention
t = 0.50322, df = 240.14, p-value = 0.6153
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-3.275147 5.522576
sample estimates:
mean of x mean of y
11.52066 10.39695 cohen_d_rescue <- effsize::cohen.d(`Pain Level_1` ~ condition, data=df)
cohen_d_rescue
Cohen's d
d estimate: -0.06376285 (negligible)
95 percent confidence interval:
lower upper
-0.3121549 0.1846292 Results of control measures
Attention Check
99% of participants (including those who were not used in analysis) passed the attention check; they responded with the value 50.
raw_df |> count(`6_Q14_1`)# A tibble: 2 × 2
`6_Q14_1` n
<dbl> <int>
1 50 265
2 62 1Calibration Check
This check aims to determine if participants have a general understanding of how to rate pain levels. The following plot shows the distribution of pain levels conditioned by stimuli.
# Plot manipulation check by grouping pain levels based on stimuli
compare_df <- df |>
rename(Pillow=`1_Q14_1`, Nail=`2_Q14_1` , Bee=`3_Q14_1`, Scratch=`4_Q14_1`, Burn=`5_Q14_1`) |>
pivot_longer(c(Pillow, Nail, Bee, Scratch, Burn), names_to = "Group", values_to = "Pain") |>
select(Group, Pain, condition)
ggplot(compare_df, aes(x=Group, y=Pain)) +
geom_boxplot() +
geom_point(alpha=0.1) +
ggtitle("Baseline Pain Levels") +
xlab("Stimulus") +
theme_classic()
It can be observed that stimuli that are more painful (e.g. getting stung by a bee burning a hand, stepping on a nail) have higher levels of pain reported compared to less painful stimuli (getting hit by a pillow or getting scratched). A pairwise t-test shows each stimuli is statistically significant from one another (excluding pillow x scratch). This calibration check indicates that given conditions to think about, participants are able to discern the amount of pain they would feel on the given scale.
# Compare pairwise if the difference in pairwise means are significant
pairwise.t.test(compare_df$Pain, compare_df$Group, p.adjust.method = "bonferroni")
Pairwise comparisons using t tests with pooled SD
data: compare_df$Pain and compare_df$Group
Bee Burn Nail Pillow
Burn <2e-16 - - -
Nail <2e-16 4e-08 - -
Pillow <2e-16 <2e-16 <2e-16 -
Scratch <2e-16 <2e-16 <2e-16 0.0023
P value adjustment method: bonferroni Confirmatory analysis
The three tests are compared against each other. The averages per condition show that the original study does find a significant difference. The first replication and this replication (rescue) fail to find significant results. This iteration does have comparable sample statistics to the initial publication, though they are not sigificant.
results <- df |> group_by(condition) |> summarize(
Mean=mean(`Pain Level_1`, na.rm = TRUE),
Sd = sd(`Pain Level_1`, na.rm = TRUE),
N = n()
)
control_results <- results[which(results$condition == "Low"),]
intervention_results <- results[which(results$condition == "High"),]
comparison_df<-data.frame(
Mean=c(13.75,6.57,23.1,18.51, control_results$Mean, intervention_results$Mean),
sd=c(21.49,11.62,23.88,23.88,control_results$Sd, intervention_results$Sd),
Study=as.factor(c("Original","Original","Replication","Replication","Rescue", "Rescue")),
Condition=c("Low Control","High Control","Low Control","High Control","Low Control", "High Control"),
Insert= c(0.0, 0.1, 0.3, 0.5, 0.7, 1.0),
N=c(50,50,72,86,control_results$N, intervention_results$N)
)
ggplot(comparison_df, aes(x=Study, y=Mean, fill=Condition)) +
geom_bar(position=position_dodge(), stat="identity",
colour='black') +
geom_errorbar(aes(ymin=Mean-sd/sqrt(N), ymax=Mean+sd/sqrt(N)), position=position_dodge(.9), width=.2) +
ggtitle("Pain Based on Sense of Control") +
ylab("Mean Pain Level") +
theme_classic()
Exploratory analyses
Because the calibration check provides various averages along the pain scale, observing the difference between the lowest rated stimuli (pre-intervention) versus the pain level (post-intervention). The histogram shows the distribution of the two conditions are essentially identical and right-skewed. Per participant, the amount of physical pain participants experienced tended to be around the same pain level of being hit by a pillow.
# Exploratory analysis, calibrate against lowest pre-test stimuli
pre_post_df <- df |> select(c(`Pain Level_1`, `1_Q14_1`, condition)) |> mutate(diff=`Pain Level_1`-`1_Q14_1`)
ggplot(pre_post_df, aes(x=diff, color=condition)) + geom_histogram(fill="white") + theme_classic()`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
Some analyses from the first replication are conducted.
Including the controls (gender, age, and employment), the condition is not statistically significant with the outcome. The indication of painkillers is statistically signfiicant in predicting the outcome value.
model = lm(`Pain Level_1` ~ condition + Gender + Age + Employment + Painkiller, df)
summary(model)
Call:
lm(formula = `Pain Level_1` ~ condition + Gender + Age + Employment +
Painkiller, data = df)
Residuals:
Min 1Q Median 3Q Max
-24.518 -8.774 -4.067 1.576 67.158
Coefficients:
Estimate Std. Error t value
(Intercept) 13.0254 7.7960 1.671
conditionLow 0.3513 2.1430 0.164
GenderMale -2.6158 2.2506 -1.162
GenderOther 4.0399 5.7649 0.701
Age 0.2882 0.0983 2.932
EmploymentNot Working (looking for work) -10.1975 6.9159 -1.474
EmploymentNot working (other) -17.3112 6.9774 -2.481
EmploymentNot working (retired) -13.3985 7.6941 -1.741
EmploymentNot working (temporary layoff from a job) -20.4023 17.7196 -1.151
EmploymentPrefer not to answer -8.4265 11.4768 -0.734
EmploymentWorking (paid employee) -14.9251 6.1465 -2.428
EmploymentWorking (self-employed) -19.8730 6.8085 -2.919
PainkillerYes 10.6526 2.7656 3.852
Pr(>|t|)
(Intercept) 0.096071 .
conditionLow 0.869934
GenderMale 0.246294
GenderOther 0.484127
Age 0.003691 **
EmploymentNot Working (looking for work) 0.141664
EmploymentNot working (other) 0.013789 *
EmploymentNot working (retired) 0.082901 .
EmploymentNot working (temporary layoff from a job) 0.250720
EmploymentPrefer not to answer 0.463537
EmploymentWorking (paid employee) 0.015912 *
EmploymentWorking (self-employed) 0.003848 **
PainkillerYes 0.000151 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 16.52 on 239 degrees of freedom
Multiple R-squared: 0.1613, Adjusted R-squared: 0.1192
F-statistic: 3.831 on 12 and 239 DF, p-value: 2.342e-05Discussion
Mini meta analysis
A meta analysis can be conducted across the three studies. There seem to be no other replications of this paper in publication. Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?
cohen_d_replication <- psych::d.ci(0.1906083, n1=78, n2=78)
cohen_d_initial <- psych::d.ci(0.41, n1=50, n2=50)
cohen_d_initial[,c("effect")]effect
0.41 meta_df <- data.frame(
name=c("Original", "Replication", "Rescue"),
d=c(cohen_d_initial[,c("effect")], cohen_d_replication[,c("effect")], cohen_d_rescue$estimate),
n=c(100, 156, nrow(df)),
upper=c(cohen_d_initial[,c("upper")], cohen_d_replication[,c("upper")], cohen_d_rescue$conf.int[c("upper")]),
lower=c(cohen_d_initial[,c("lower")], cohen_d_replication[,c("lower")], cohen_d_rescue$conf.int[c("lower")])
)
meta_df <- meta_df |> mutate(sd=sqrt(n) * (upper-lower) / 3.92) |> mutate(var=sd^2)
re_model <- rma(yi = d, vi = var, data = meta_df, method = "REML", knha = TRUE)
re_model
Random-Effects Model (k = 3; tau^2 estimator: REML)
tau^2 (estimated amount of total heterogeneity): 0 (SE = 4.0502)
tau (square root of estimated tau^2 value): 0
I^2 (total heterogeneity / total variability): 0.00%
H^2 (total variability / sampling variability): 1.00
Test for Heterogeneity:
Q(df = 2) = 0.0277, p-val = 0.9863
Model Results:
estimate se tval df pval ci.lb ci.ub
0.1783 0.1366 1.3047 2 0.3219 -0.4096 0.7661
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1forest(re_model, slab=meta_df$name)
The three studies generate an overall effect size that is slightly positive, but with a lower confidence bound being negative; there is not enough confidence to demonstrate that the effect is larger than zero. ### Summary of Replication Attempt
The primary finding from this replication attempt is that there is not evidence to support the rejection of the difference in group means. Simply put, this replication was unable to produce results similar to the initial study. A possible reason why this study did not replicate is due to the pre-survey questions. It can be argued that the pre-survey is priming the participant before the intervention. But when looking at how participants responded (pre- vs post-intervention), the two groups had consistent outcomes; many people did not shift pain levels (relative to the baseline of pillow taps). Though the replication study and this study generate the same outcome, it is possible that the initial study performed a survey that was different from the simple description described in the paper.
Commentary
The exploratory analysis showed that participants responded pretty similarly among groups. This could be a confound of the pre-survey, but is likely not because the pre-survey is in a sense more of a practice. The study did not replicate from the original, but it did produce the same results as the first replication. This could be an indication that the first study was lucky, or there were many iterations of the study and the final version with really promising results is shown. There are definitely differences between the studies, and is likely within the survey design; because all studies use comparable platforms, and the two replication studies almost perfectly mirror each other, the biggest gray area is in the initial study material. There is still confounding regarding the experiences participants thought about. If the study picked consistent stimuli, there could possibly be a better control of the outcome.
On a scale of 0 - 1, this project has a replication of 0.25. A positive aspect of all the studies is that participants in the low-control condition did, on average have a higher pain score, even if not statistically significant. Both the original replication and rescue replication are Very Close to the original study, even though the initial survey material is not publicly available.