## Warning: package 'kableExtra' was built under R version 3.4.2Introduction
Prevalence of economic insecurity and physical pain have both increased dramatically in recent years. A series of studies by Chou, Parmar, and Galinsky (2016) report a causal link between the two, providing further evidence for a relationship between broad social trends and physical health. This finding has important public health implications: pain has an economic cost of more than $635 billion and increased over 1000% in the past two decades. Connecting these two major social issues may critically shape public policy and should be investigated further.
Chou and colleagues expand upon their finding by showing the causal link between economic insecurity and physical pain is mediated by feelings of lack of control. The psychological construct of lack of control has been implicated in numerous psychological and physiological states. The goal of this replication is to replicate the results of study 5, which reports a causal link between their proposed mediating variable, lack of control, and their dependent variable, self-reported physical pain. While this does not replicate the main finding of the paper, the relationship between lack of control and physical pain is an important component of Chou’s proposed model.
Methods
Power Analysis
One hundred participants were recruited by the author for Study 5. A simple t-test was used in their analysis of the data (t(98)=2.04; p=0.04)), which yielded an effect size of d=0.41. A post-hoc power analysis revealed that the study was powered at 53%. To achieve 80% power, an a priori power analysis conducted using G Power suggested a sample of 156 subjects across two groups.
Planned Sample
156 adult subjects will be recruited via Amazon’s Mechanical Turk and will be randomly assigned to either the high control or low control condition. mTurk workers will be recruited from the United States with a hit rate >90%. Any participants who does not provide consent (responds “disagree” to the question ‘consent’) will be excluded from the survey. All other participants will be included.
Materials
Participants were presented with the same questions as in Study 5 of Chou et al., 2016.
Procedure
Participants began by reading the IRB header and were asked to agree / disagree to continue. They were then randomized to either a high control or low control condition. In the high control condition, participants were asked to write about a time in their life when they felt in total control of a situation. They are allowed to choose the amount of time spent on this response. In the low-control condition, participants are asked to similarly describe a time in their life when they had no control over a situation. They are required to spend 1.5 minutes on this response before the page automatically advances. The purpose of this automatic advancement after 1.5 minutes is to reinforce the effect of the lack of control manipulation.
All participants are then asked to record the amount of physical pain they are currently experiencing, on a 0-100 slide scale.
Subjects then completed a short, five-item measure of control, that was an additional scale added to the author’s original procedure. See Differences from Original Study section for more details.
Finally, participants responded to a short demographics questionnaire in which they were asked their age, gender, employment status, and use of painkillers.
Excerpt from paper:
“Participants were asked to write about an autobiographical experience (Whitson & Galinsky, 2008). Half recalled a time when they lacked control. The other half recalled a time when they had complete control.
After the recall task, participants were asked to “choose the overall pain level that best describes how much physical pain you are experiencing RIGHT NOW.” They responded using a visual slider scale (Portenoy & Kanner, 1996), on which 0 indicated no pain and 100 indicated the worst pain ever experienced.
Participants also reported their age, gender, and current employment status, and whether they were using painkillers."
Replication Link
https://stanforduniversity.qualtrics.com/jfe/form/SV_cOKVcCFKgzvOxxj
Analysis Plan
Analysis Steps
Data Cleaning: Summarize the data so that each participant has a separate column for the manipulation condition (low control / high control), response to pain questionnaire, and control variables / demographics. The mean score for the five-item lack of control questionnaire will be calculated for each participant as the average of the five individual responses.
- Analysis:
Calculate descriptive statistics for the two groups. Mean and SD will be calculated for pain rating (overall and by group), five-item control questionnaire (overall and by group), and control variables / demographics.
Conduct a t-test for mean difference in pain ratings between the two manipulation groups.
Conduct a regression model in which in pain ratings between the two manipulation groups controlling for age, gender, and painkiller use.
Plot the relationship between condition and pain ratings with a scatterplot, including a linear regression line.
Differences from Original Study
Sample differences
None: the original paper used a sample of 100 mTurk workers in 2015. It can be assumed that a similar sample would be obtained with the replication.
Setting differences
Slight: We will be paying participants 60 cents for their time. The original study paid 41 cents. Given the minor adjustment in cost, we do not believe this will have any effect on the results.
Procedure differences
The replication was conducted exactly as outlined in Study 5 of Chou et al., with the addition of a short, five-item scale that captures respondents’ psychological sense of lacking control (Lachman & Weaver, 1998). This scale was used in study 4 of the paper, and serves as an additional measure of control to be used in additional analyses of the overall hypothesis presented by Chou and colleagues.
Two additional control variable questions were added to the end of the survey. Following the question regarding use of pain killers, participants who responded yes to this were routed to two additional questions about painkiller type and use frequency. These were added so that additional analyses could be conducted on the subpopulation of participants who were taking painkilling drugs to see if this a self-identified clinical sample would experience the same pattern of results.
The study was made available to participants at 12:30pm EST on a weekday and took approximately two hours to collect. We do not have data on the time of day data collection was conducted with the original sample or how this may or may not impact the demographics of the population.
Analysis plan differences
As identified above, we are collecting an additional measure of control - the Lachman & Weaver five-item control scale. First, I hope to use this scale as a manipulation check to see if the manipulation result in a significant mean difference in this scale. The goal of the high/low control manipulation is to induce a sense of control (high control condition) or a sense of lack of control (low control condition). The authors do not indicate any check to ensure this manipulation was effective in the original paper. The manipulation may - rather than altering sense of control - cause participants to reflect on experiences with different affective valences. Instances of low control may simply be more upsetting, and it is this affective component that drives difference in pain ratings.
Methods Addendum
No changes to the methodology were made. Data collection proceeded as described above.
Actual Sample
Data was collected from 158 subjects. While attempts were made to assign subjects equally to the two manipulation groups, the two groups were not identical in size. Data was collected from 86 subjects assigned to the high control condition and 72 subjects assigned to the low control condition.
There was no plan made for the exclusion of data, therefore data from all 158 subjects were collected.
Differences from pre-data collection methods plan
No changes were made to the data collection methods.
Results
Data preparation
Descriptive Statistics - Control Variables and Demographics
Age
## [1] 33.31765The mean age of the sample was 33.95 years (SD=9.67 years).
Gender
## # A tibble: 4 x 2
## gender percent
## <fctr> <dbl>
## 1 0.6329114
## 2 1 57.5949367
## 3 2 41.1392405
## 4 3 0.6329114The sample was 57.59% male, 41.14% female, and 0.63% other. One subject did not respond to this question.
Employment
## # A tibble: 8 x 2
## employment percent
## <fctr> <dbl>
## 1 0.6329114
## 2 1 63.2911392
## 3 2 24.6835443
## 4 4 3.7974684
## 5 5 0.6329114
## 6 6 0.6329114
## 7 7 5.0632911
## 8 8 1.265822887.97% of the sample was currently working. 63.29% of the sample were paid employees employment and 24.7% were self employed. 11.39% of the sample was not working.
Current Painkiller Use
## # A tibble: 3 x 2
## painkiller percent
## <fctr> <dbl>
## 1 0.6329114
## 2 1 20.2531646
## 3 2 79.113924120.3% of the sample endorsed current use of a painkiller.
Descriptive Statistics - Pain Ratings
Across All Subjects
## [1] 21.01266## vars n mean sd median trimmed mad min max range skew kurtosis
## X1 1 158 21.01 24.12 10 17.66 14.83 0 92 92 0.95 -0.37
## se
## X1 1.92The mean rating of current pain across all 158 subjects was 21.01 (SD=24.12) on a 100 point slide scale.
Descriptive Statistics for Subjects in the High Control Condition
## [1] 23.10465## vars n mean sd median trimmed mad min max range skew kurtosis
## X1 1 86 23.1 23.88 12.5 20.73 18.53 0 75 75 0.64 -1.09
## se
## X1 2.57The mean pain rating of current pain for the 86 subjects in the high control condition was 23.10 (SD=23.88).
Descriptive Statistics for Subjects in the Low Control Condition
# Pain Ratings for Subjects in Low Control Condition:
avg_pain_low <- mean(subset(d2$pain_rate_1,d2$condition=='low_control'))
print(avg_pain_low)## [1] 18.51389describe(subset(d2$pain_rate_1,d2$condition=='low_control'))## vars n mean sd median trimmed mad min max range skew kurtosis
## X1 1 72 18.51 24.33 6.5 13.95 9.64 0 92 92 1.32 0.63
## se
## X1 2.87The mean pain rating of current pain for the 72 subjects in the low control condition was 18.51 (SD=24.33).
Descriptive Statistics - Five Item Control Scale Ratings
Across All Subjects
## [1] 3.713924## vars n mean sd median trimmed mad min max range skew kurtosis
## X1 1 158 3.71 0.89 3.9 3.76 1.04 1.4 5 3.6 -0.46 -0.57
## se
## X1 0.07The mean score on the five-item lack of control scale for all 158 subjects was 3.71 (SD=0.89).
Subjects in High Control Condition
## [1] 3.704651## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 86 3.7 0.88 3.8 3.74 0.89 1.4 5 3.6 -0.37 -0.8 0.1The mean score on the five-item lack of control scale for the 86 subjects in the high control condition was 3.70 (SD=0.88).
Subjects in Low Control Condition
## [1] 3.725## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 72 3.73 0.9 4 3.78 0.89 1.4 5 3.6 -0.56 -0.36 0.11The mean score on the five-item lack of control scale for the 72 subjects in the high control condition was 3.73 (SD=0.90).
Confirmatory analysis
Primary Analysis
##
## Two Sample t-test
##
## data: subset(d2$pain_rate_1, d2$condition == "high_control") and subset(d2$pain_rate_1, d2$condition == "low_control")
## t = 1.1932, df = 156, p-value = 0.2346
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -3.00876 12.19028
## sample estimates:
## mean of x mean of y
## 23.10465 18.51389## [1] 0.1906083A Two Sample t-test (with the assumption of equal variance) of the mean pain rating between the two manipulation conditions returned a t=0.55; df=141.97; p=0.58; 95% CI of the mean difference between conditions was -3.87 - 6.90. The effect size, measured by Cohen’s d, was 0.191.
Linear Model controlling for age and gender
To determine the influence of potentially mediating variables, a linear model has been constructed. In addition to the condition to which each subject has been assigned, this model adds age, gender, employment status, and painkiller use to the set of variables predicting pain rating.
library(stargazer, quietly = TRUE)
lin_mod_a = lm(pain_rate_1 ~ condition, data = d2)
lin_mod_b = lm(pain_rate_1 ~ condition + age + gender + employment_yn + painkiller, data = d2)
stargazer(lin_mod_a, lin_mod_b, type="html",
dep.var.labels=c("Pain Rating"),
covariate.labels=c("Condition", "Age", "Gender", "Employment Status", "Painkiller Use"), out="models.htm")| Dependent variable: | ||
| Pain Rating | ||
| (1) | (2) | |
| Condition | -4.591 | -2.761 |
| (3.847) | (3.680) | |
| Age | -0.179 | |
| (0.190) | ||
| Gender | -1.993 | |
| (3.810) | ||
| Employment Status | -5.210 | |
| (5.844) | ||
| Painkiller Use | -21.014*** | |
| (4.546) | ||
| Constant | 23.105*** | 51.703*** |
| (2.597) | (9.402) | |
| Observations | 158 | 156 |
| R2 | 0.009 | 0.144 |
| Adjusted R2 | 0.003 | 0.115 |
| Residual Std. Error | 24.085 (df = 156) | 22.579 (df = 150) |
| F Statistic | 1.424 (df = 1; 156) | 5.038*** (df = 5; 150) |
| Note: | p<0.1; p<0.05; p<0.01 | |
A linear model controlling for age, gender, employment status, and painkiller use was constructed. Painkiller use was the only factor that was significantly positively related to pain rating (b=10.76; SE=3.38; t-value=3.18; p<0.01). As with the t-test above, the condition to which the subject was assigned (high control / low control) was not a significant variable in the model.
Next, to visualize the data, we have constructed a boxplot of pain ratings across the two manipulation conditions (low control & high control).
ggplot(d2, aes(x = condition, y = pain_rate_1)) +
geom_boxplot(fill = "#4271AE", colour = "#1F3552", alpha = 0.7, outlier.colour = "#1F3552", outlier.shape = 20) +
scale_y_continuous(name = "Mean Pain Rating", breaks = seq(0, 80, 10), limits=c(0, 80)) +
scale_x_discrete(name = "Control Condition") +
ggtitle("Mean Pain Rating Across Control Conditions") +
stat_summary(fun.y=mean, colour='red', geom='point') +
theme_bw() + theme(plot.title = element_text(hjust = 0.5))
A boxplot of pain ratings across the two control conditions is presented. Pain ratings were assessed on a 0-100 scale, however, for purposes of visualization, a 0-80 point scale is shown. The mean for each group is marked in red on the boxes.
Exploratory analyses
We included an measure of lack of control - the five-item lack of control scale (Lachman & Weaver, 1998) in addition to the measures used by Chou and colleagues. It is of interest to see whether or not assignment to the high control or low control manipulation condition affected subsequent scores on this lack of control scale. As reported in the descriptive statistics above, The mean score on the five-item lack of control scale for all 158 subjects was 3.71 (SD=0.89). The mean score for the 86 subjects in the high control condition was 3.70 (SD=0.88). The mean score for the 72 subjects in the high control condition was 3.73 (SD=0.90).
t.test(subset(d2$five_item_control, d2$condition=='high_control')
, subset(d2$five_item_control, d2$condition=='low_control')
, alternative = "two.sided", var.equal = TRUE)##
## Two Sample t-test
##
## data: subset(d2$five_item_control, d2$condition == "high_control") and subset(d2$five_item_control, d2$condition == "low_control")
## t = -0.14301, df = 156, p-value = 0.8865
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -0.3014058 0.2607081
## sample estimates:
## mean of x mean of y
## 3.704651 3.725000A Two Sample t-test (assumed equal variance) for difference in means between groups found no significant differences and yeilded a t-value of -0.143 and a p-value of 0.886 on 150.29 degrees of freedom. The 95% confidence interval around the mean difference is -0.31 - 0.26.
Next, we want to understand the relationship between the outcome on this measure of control and pain ratings. We construct a linear model to see if the score on the five item control scale is predicitive of pain ratings.
##
## Call:
## lm(formula = pain_rate_1 ~ five_item_control, data = d2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -39.937 -16.944 -9.987 18.079 70.056
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 51.387 7.917 6.491 1.08e-09 ***
## five_item_control -8.179 2.074 -3.944 0.000121 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 23.07 on 156 degrees of freedom
## Multiple R-squared: 0.09068, Adjusted R-squared: 0.08485
## F-statistic: 15.56 on 1 and 156 DF, p-value: 0.0001207library(stargazer, quietly = TRUE)
lin_mod2 = lm(pain_rate_1 ~ five_item_control, data = d2)
stargazer(lin_mod2, type="html",
dep.var.labels=c("Pain Rating"),
covariate.labels=c("Condition", "Five-Item Control Scale", out="models.htm"))| Dependent variable: | |
| Pain Rating | |
| Condition | -8.179*** |
| (2.074) | |
| Five-Item Control Scale | 51.387*** |
| (7.917) | |
| Observations | 158 |
| R2 | 0.091 |
| Adjusted R2 | 0.085 |
| Residual Std. Error | 23.071 (df = 156) |
| F Statistic | 15.556*** (df = 1; 156) |
| Note: | p<0.1; p<0.05; p<0.01 |
Mean score on the five item control scale is a significant negative predictor of pain ratings. Less perceived control is significantly related to higher pain scores. We can represent this relationship visually below, in a scatter plot depicting the relationship between pain and perceived control. To illustrate the null effect of condition, the condition to which each subject was assigned is depicted.
ggplot(d2, aes(x=five_item_control, y=pain_rate_1, color=(condition))) +
geom_point() +
geom_smooth(method=lm, se=FALSE, fullrange=TRUE) +
labs(title="Pain Rating as a Function of Five-Item Control Scale",
x="Mean Five-Item Control", y = "Pain Rating") +
theme_bw() + theme(plot.title = element_text(hjust = 0.5)) + scale_colour_manual(name='Condition', values=c('high_control'='steelblue', 'low_control'='firebrick'))
Discussion
As described above, the original publication of Study 5 showed a causal link between lack of control and self-reported physical pain. This was included in the larger model presented in the 2016 Psych Science paper, which demonstrated a causal link between economic insecurity and physical pain that was mediated by perceived lack of control.
A faithful attempt to replicate Study 5 from Chou et al., 2016 is presented. Methodologically, we used the exact same protocol as Chou and colleagues with two minor additions. First, we included a short, five-item scale that captures respondents’ psychological sense of lacking control (Lachman & Weaver, 1998). Second, we added two addition control variables for subjects who indicated they currently use painkillers: painkiller type and frequency.
A post-hoc power analysis suggested a sample size of n=156 to achieve 80% power to identify a significant mean difference between the two manipulation groups. Complete data were collected from 158 subjects. As with Chou et al., 2016, we recruited our sample from Amazon’s mechanical turk without exclusion criteria.
Both age and gender were similarly distributed across groups and were similar to the demographics presented in the original paper. In the original paper, mean (SD) age was 31.05 (9.80), which was similar to our mean (SD) age of 33.95 (9.67). The original paper had a slightly higher percentage of female subjects (48.52%) compared with our replication (41.14%). While not reported in the original paper, we report an employment rate (employee or self-employed) of 87.97%. 20.3% of our sample endorsed using a painkiller. While not reported in the original study, we began data collection at 12:30pm EST on a weekday, in the hope of collecting data during a time when Americans would be active and alert on mTurk.
Summary of Replication Attempt
The original study reported a significant mean difference in pain ratings between subjects who were assigned to the high control condition and those assigned to the low control condition. Subjects assigned to the low control condition reported a mean pain rating of more than twice the mean pain rating reported by subjects in the high control condition. Our replication failed to find this difference. In fact, the mean pain rating was actually slightly - but not significantly - higher in the high control condition compared to the low control condition. It is of note that the mean pain ratings for both groups were significantly higher than those reported in the original paper. Across both groups, the mean pain rating in the original paper was 10.16 on a 100 point scale. In the attempted replication, the mean pain rating across both groups was 21.01. We have no explanation for this substantially higher mean pain rating.
As mentioned above, we included an additional measure of control - the Lachman and Weaver 5-item control scale. We found no difference in the mean score on this scale across the two conditions, suggesting the same degree of perceived control was present in subjects regardless of the manipulation. This implies the manipulation did not succeed in changing the amount of psychological control the subjects experienced. The score on this measure of psychological control did, however, significantly predict physical pain. This suggests that lack of control is indeed related to physical pain, however, we cannot infer causality from this study, only that a relationship exists between the two constructs.
We failed to replicate the main finding from Study 5 of Chou et al., 2016. Our results do not support the notion that lack of control is causally related to physical pain ratings. Our working hypothesis is that the manipulation was ineffective and did not induce a state of high or low control in the subjects. Given that mean scores on the Lachman & Weaver control scale were significantly related to pain ratings, a relationship between psychological control and physical pain does exist. In total, the results of these two studies provide evidence that 1) lack of control is related to physical pain; 2) this relationship may be causal, though we cannot confirm this.
Commentary
There are several working theories that could explain the discrepancy between the original findings and our attempted replication. Perhaps there have been significant changes in the social climate since the original study was conducted such that lack of control is more difficult to induce experimentally. Given the recent shift in the political climate in the past year, it is plausible that the threshold for lack of control is much higher now than it was when Chou and colleagues collected their data. Another hypothesis is that the authors collected their data closer to the housing market collapse and subsequent recession that occurred in 2008. Subjects may have been more likely to recall instances of high / low control related to financial stressors when data for the original study was collected. While their study was published in 2016, we do not know when this data was actually collected. Finally, a third possibility is that workers on Amazon’s Mechanical Turk have become less engaged with recall / short response writing tasks and did not reflect on a time of high or low control as much as they did when Chou and colleagues collected their data.
The failed replication has implications for all studies that are conducted using online convenience samples like those available on mTurk. It also highlights the need for frequent replication of research conducted using these samples to check the reliability of findings. As individuals continue to engage in the new gig-based economy, like mechanical turk, they may simply become more adept at pursuing their underlying goals of making money rather than contributing to scientific research. It is the responsibility of the research community that we understand the realities of these convenience samples, and not assume their intentions are in line with our own.