Reproducibility Report: Group B Choice 3
Ramya Kumar
For this exercise, please try to reproduce the results from Experiment 2 of the associated paper (de la Fuente, Santiago, Roman, Dumitrache, & Casasanto, 2014). The PDF of the paper is included in the same folder as this Rmd file.
Methods summary:
Researchers tested the question of whether temporal focus differs between Moroccan and Spanish cultures, hypothesizing that Moroccans are more past-focused, whereas Spaniards are more future-focused. Two groups of participants (\(N = 40\) Moroccan and \(N=40\) Spanish) completed a temporal-focus questionnaire that contained questions about past-focused (“PAST”) and future-focused (“FUTURE”) topics. In response to each question, participants provided a rating on a 5-point Likert scale on which lower scores indicated less agreement and higher scores indicated greater agreement. The authors then performed a mixed-design ANOVA with agreement score as the dependent variable, group (Moroccan or Spanish, between-subjects) as the fixed-effects factor, and temporal focus (past or future, within-subjects) as the random effects factor. In addition, the authors performed unpaired two-sample t-tests to determine whether there was a significant difference between the two groups in agreement scores for PAST questions, and whether there was a significant difference in scores for FUTURE questions.
Target outcomes:
Below is the specific result you will attempt to reproduce (quoted directly from the results section of Experiment 2):
According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjectS factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2). Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001, and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001. (de la Fuente et al., 2014, p. 1685).
Step 1: Load packages
#install.packages("afex")
#install.packages("ez")
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(afex)
library(ez)
# #optional packages/functions:
#library(afex) # anova functions
#library(ez) # anova functions 2
library(scales) # for plotting
std.err <- function(x) sd(x)/sqrt(length(x)) # standard errorStep 2: Load data
# Just Experiment 2
data_path <- 'data/DeLaFuenteEtAl_2014_RawData.xls'
d <- read_excel(data_path, sheet=3)Step 3: Tidy data
df.exp2 <- d %>%
rename(agreement = `Agreement (0=complete disagreement; 5=complete agreement)`) %>%
mutate(agreement = as.numeric(agreement)) %>%
group_by(participant, group, subscale, item) %>%
mutate(duplicate = row_number()) %>%
ungroup() %>%
pivot_wider(
id_cols = c(participant, duplicate, group),
names_from = c(subscale, item),
values_from = agreement,
names_glue = "{subscale}_{item}") %>%
filter(duplicate != 2) #not sure which duplicate to removeStep 4: Run analysis
Pre-processing
df.exp2 <- df.exp2 %>%
mutate(
FUTURE_agreement_mean = rowMeans(select(., starts_with("FUTURE_")), na.rm = TRUE),
PAST_agreement_mean = rowMeans(select(., starts_with("PAST_")), na.rm = TRUE),
) %>%
pivot_longer(
cols = c(PAST_agreement_mean, FUTURE_agreement_mean),
names_to = "statement_type",
values_to = "agreement_rating") %>%
mutate(
statement_type = case_when(
statement_type == "PAST_agreement_mean" ~ "Past-Focused Statements",
statement_type == "FUTURE_agreement_mean" ~ "Future-Focused Statements")) %>%
mutate(
group = case_when(
group == "young Spaniard" ~ "Spaniards",
group == "Moroccan" ~ "Moroccans"))
df.exp2.summary <- df.exp2 %>%
select(participant,group,statement_type,agreement_rating) %>%
group_by(group, statement_type) %>%
summarise(
mean_agreement = mean(agreement_rating, na.rm = TRUE),
se_agreement = sd(agreement_rating, na.rm = TRUE) / sqrt(n())
) %>%
ungroup()Descriptive statistics
Try to recreate Figure 2 (fig2.png, also included in the same folder as this Rmd file):
exp2.plot <- ggplot(df.exp2.summary, aes(x = group, y = mean_agreement, fill = statement_type)) +
geom_bar(stat = "identity", position = position_dodge(0.9), width = 0.8) +
geom_errorbar(aes(ymin = mean_agreement - se_agreement,
ymax = mean_agreement + se_agreement),
position = position_dodge(0.9), width = 0.2) +
scale_fill_manual(values = c("Past-Focused Statements" = "gray50",
"Future-Focused Statements" = "gray90")) +
labs(x = "", y = "Rating", fill = "") +
scale_y_continuous(limits = c(0, 4), breaks = seq(0, 4, by = 0.5)) +
theme_minimal() +
theme(
legend.position = "top",
axis.title.y = element_text(margin = margin(r = 10)),
panel.grid.major.x = element_blank(),
panel.grid.minor.y = element_blank()
)
print(exp2.plot)
Inferential statistics
According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjects factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2).
# reproduce the above results here
#reshape
df.exp2.anova <- df.exp2 %>%
select(participant,group,statement_type,agreement_rating) %>%
mutate(statement_type = factor(statement_type),
group = factor(group),
subject = ifelse(group == "Spaniards",
participant + 40,
participant))
#mixed effect anova
anova_result <- aov_ez(id = "subject",
dv = "agreement_rating",
data = df.exp2.anova,
between = "group",
within = "statement_type",
type = 3,
na.rm = TRUE)
summary(anova_result)##
## Univariate Type III Repeated-Measures ANOVA Assuming Sphericity
##
## Sum Sq num Df Error SS den Df F value Pr(>F)
## (Intercept) 1536.97 1 15.388 76 7591.0606 < 2.2e-16 ***
## group 0.35 1 15.388 76 1.7195 0.1937007
## statement_type 3.76 1 37.591 76 7.5937 0.0073282 **
## group:statement_type 8.26 1 37.591 76 16.7057 0.0001073 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1nice(anova_result)## Anova Table (Type 3 tests)
##
## Response: agreement_rating
## Effect df MSE F ges p.value
## 1 group 1, 76 0.20 1.72 .007 .194
## 2 statement_type 1, 76 0.49 7.59 ** .066 .007
## 3 group:statement_type 1, 76 0.49 16.71 *** .135 <.001
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1# why is the df now 76?Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001,
# reproduce the above results here
df.exp2.ttest <- df.exp2 %>%
select(participant,group,statement_type,agreement_rating) %>%
pivot_wider(names_from = statement_type,
values_from = agreement_rating) %>%
rename("past_agreement" = "Past-Focused Statements",
"future_agreement" = "Future-Focused Statements")
ttest_past <- t.test(past_agreement ~ group, data = df.exp2.ttest,
var.equal = TRUE)
print(ttest_past)##
## Two Sample t-test
##
## data: past_agreement by group
## t = 3.6595, df = 76, p-value = 0.0004641
## alternative hypothesis: true difference in means between group Moroccans and group Spaniards is not equal to 0
## 95 percent confidence interval:
## 0.2528409 0.8567162
## sample estimates:
## mean in group Moroccans mean in group Spaniards
## 3.261072 2.706294# why is the df now 76?and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001.(de la Fuente et al., 2014, p. 1685)
# reproduce the above results here
ttest_future <- t.test(future_agreement ~ group, data = df.exp2.ttest,
var.equal = TRUE)
print(ttest_future)##
## Two Sample t-test
##
## data: future_agreement by group
## t = -3.0635, df = 78, p-value = 0.003003
## alternative hypothesis: true difference in means between group Moroccans and group Spaniards is not equal to 0
## 95 percent confidence interval:
## -0.5678305 -0.1205029
## sample estimates:
## mean in group Moroccans mean in group Spaniards
## 3.133333 3.477500Step 5: Reflection
Were you able to reproduce the results you attempted to reproduce? If not, what part(s) were you unable to reproduce?
Unfortunately no :(. I hit the 6 hour limit during this one. I particularly struggled with getting the outputs to match the exact same results mentioned in the paper statements (would get df=76 instead of df=78 sometimes?). I think it’s maybe because I wasn’t able to handle the duplicates in the same way as the paper? I removed the duplicate participant IDs in my analyses, but not sure if its the same one that the paper removed (or if they even removed them).
How difficult was it to reproduce your results?
It was very difficult to reproduce the result due to the duplicate error (and maybe how I reshaped the data is different than how the authors did it).
What aspects made it difficult? What aspects made it easy?
DIFFICULT: I had some participants with 2 scores, I ended up randomly removing one and I’m not sure why because I didn’t know the differences between the duplicates (did they just submit twice, or are they two different participants?). I would have needed more information to decide which one to remove. It was also a bit difficult to get the graph to look exactly the same, but I think thats just my limitations in understanding plotting code (like creating a border for the bars, getting the y-axis to look exactly the same as the paper’s figure)
EASY: The authors did a good job labeling the data (everything was clearly labeled and it was easy to mutate/reshape the data). Just wish I knew if they had any exclusion criteria to help with better identifying the duplicates.