Reproducibility Report: Group B Choice 3
For this exercise, please try to reproduce the results from Experiment 2 of the associated paper (de la Fuente, Santiago, Roman, Dumitrache, & Casasanto, 2014). The PDF of the paper is included in the same folder as this Rmd file.
Methods summary:
Researchers tested the question of whether temporal focus differs between Moroccan and Spanish cultures, hypothesizing that Moroccans are more past-focused, whereas Spaniards are more future-focused. Two groups of participants (\(N = 40\) Moroccan and \(N=40\) Spanish) completed a temporal-focus questionnaire that contained questions about past-focused (“PAST”) and future-focused (“FUTURE”) topics. In response to each question, participants provided a rating on a 5-point Likert scale on which lower scores indicated less agreement and higher scores indicated greater agreement. The authors then performed a mixed-design ANOVA with agreement score as the dependent variable, group (Moroccan or Spanish, between-subjects) as the fixed-effects factor, and temporal focus (past or future, within-subjects) as the random effects factor. In addition, the authors performed unpaired two-sample t-tests to determine whether there was a significant difference between the two groups in agreement scores for PAST questions, and whether there was a significant difference in scores for FUTURE questions.
Target outcomes:
Below is the specific result you will attempt to reproduce (quoted directly from the results section of Experiment 2):
According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjectS factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2). Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001, and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001. (de la Fuente et al., 2014, p. 1685).
Step 1: Load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(rstatix)
# #optional packages/functions:
library(afex) # anova functions
library(ez) # anova functions 2
# library(scales) # for plotting
# std.err <- function(x) sd(x)/sqrt(length(x)) # standard errorStep 2: Load data
# Just Experiment 2
data_path <- 'data/DeLaFuenteEtAl_2014_RawData.xls'
d <- read_excel(data_path, sheet=3) %>%
rename(agreement = "Agreement (0=complete disagreement; 5=complete agreement)") %>%
group_by(group, participant) %>% # group by group & participant
mutate(participant_ID = cur_group_id()) %>% # assign unique ID
ungroup()Step 3: Tidy data
participant_level <- d %>%
# group_by(group, participant) %>% # group by group & participant
# mutate(participant_ID = cur_group_id()) %>% # assign unique ID
# ungroup() %>%
group_by(participant_ID, group, subscale) %>%
summarize(mean_agreement = mean(agreement)) %>%
mutate(participant_ID=as.factor(participant_ID),
group=as.factor(group),
subscale=as.factor(subscale)) Step 4: Run analysis
Pre-processing
aggregate_level <- d %>%
group_by(group, subscale) %>%
get_summary_stats(agreement, type = "mean_se")Descriptive statistics
Try to recreate Figure 2 (fig2.png, also included in the same folder as this Rmd file):
fig2 <- aggregate_level %>%
mutate(group=case_when(group=="young Spaniard"~"Spaniards",
group=="Moroccan"~"Moroccans")) %>%
mutate(group=fct_relevel(group, c("Spaniards", "Moroccans")),
subscale=fct_relevel(subscale, c("PAST", "FUTURE"))) %>%
ggplot(aes(x=group, y=mean, fill=subscale)) +
geom_col(position = "dodge") +
geom_errorbar(aes(ymin=mean-se, ymax=mean+se),
position=position_dodge2(width=.5, padding=.5)) +
labs(y="Rating",
x="") +
coord_cartesian(ylim=c(2,4)) +
theme(legend.title = element_blank())
fig2
Inferential statistics
According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjects factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2).
# reproduce the above results here
# Run mixed ANOVA
aov_result <- aov_ez(
id = "participant_ID", # subject identifier column
dv = "mean_agreement", # dependent variable
between = "group", # between-subjects factor
within = "subscale", # within-subjects factor
data = participant_level
)
summary(aov_result)##
## Univariate Type III Repeated-Measures ANOVA Assuming Sphericity
##
## Sum Sq num Df Error SS den Df F value Pr(>F)
## (Intercept) 1543.46 1 15.250 76 7692.0904 < 2.2e-16 ***
## group 0.44 1 15.250 76 2.1920 0.14287
## subscale 3.97 1 37.777 76 7.9793 0.00604 **
## group:subscale 9.12 1 37.777 76 18.3456 5.328e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1# These were the missing participants. I tried to see how the results change when they are excluded
removed_missing <- participant_level %>%
filter(!participant_ID %in% c("25", "65"))Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001,
# reproduce the above results here
# unpaired t-test comparing agreement scores for subscale = PAST
past <- participant_level %>%
filter(subscale=="PAST")
t.test(mean_agreement~group, var.equal=TRUE, data=past)##
## Two Sample t-test
##
## data: mean_agreement by group
## t = 3.8562, df = 76, p-value = 0.0002394
## alternative hypothesis: true difference in means between group Moroccan and group young Spaniard is not equal to 0
## 95 percent confidence interval:
## 0.2851528 0.8943343
## sample estimates:
## mean in group Moroccan mean in group young Spaniard
## 3.280886 2.691142and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001.(de la Fuente et al., 2014, p. 1685)
# reproduce the above results here
future <- participant_level %>%
filter(subscale=="FUTURE")
t.test(mean_agreement~group, var.equal=TRUE, data=future)##
## Two Sample t-test
##
## data: mean_agreement by group
## t = -3.2098, df = 78, p-value = 0.001929
## alternative hypothesis: true difference in means between group Moroccan and group young Spaniard is not equal to 0
## 95 percent confidence interval:
## -0.5758588 -0.1349746
## sample estimates:
## mean in group Moroccan mean in group young Spaniard
## 3.138333 3.493750Step 5: Reflection
Were you able to reproduce the results you attempted to reproduce? If not, what part(s) were you unable to reproduce?
(Note: I stopped after reaching the three-hour time limit.) I wasn’t able to reproduce the inferential statistics because of some missing data that was not accounted for in the paper. For example, for a couple participants they did not have data for BOTH future and past subscales, so I couldn’t run the mixed ANOVA with the same denominator degrees of freedom. The paper reports denominator df = 78, but because two participants were missing data my denominator df = 76. Even for my second t-test, in which I compared future-focused statements and had the same df as the authors, I was not able to get the same test statistic.
What aspects made it difficult? What aspects made it easy?
Overall, the data was in a straightforward format with reasonable column names. I was able to reproduce the figure. However, the inferential statistics portion was difficult. There was not a unique participant ID that applied across the whole dataset so I had to create one for the analysis. Critically, data was missing for some participants and I didn’t know how to account for that. The authors didn’t report descriptive statistics within their reports of the main results, so it is hard to tell from where the discrepancy arose.