For this exercise, please try to reproduce the results from Experiment 2 of the associated paper (de la Fuente, Santiago, Roman, Dumitrache, & Casasanto, 2014). The PDF of the paper is included in the same folder as this Rmd file.

Methods summary:

Researchers tested the question of whether temporal focus differs between Moroccan and Spanish cultures, hypothesizing that Moroccans are more past-focused, whereas Spaniards are more future-focused. Two groups of participants (\(N = 40\) Moroccan and \(N=40\) Spanish) completed a temporal-focus questionnaire that contained questions about past-focused (“PAST”) and future-focused (“FUTURE”) topics. In response to each question, participants provided a rating on a 5-point Likert scale on which lower scores indicated less agreement and higher scores indicated greater agreement. The authors then performed a mixed-design ANOVA with agreement score as the dependent variable, group (Moroccan or Spanish, between-subjects) as the fixed-effects factor, and temporal focus (past or future, within-subjects) as the random effects factor. In addition, the authors performed unpaired two-sample t-tests to determine whether there was a significant difference between the two groups in agreement scores for PAST questions, and whether there was a significant difference in scores for FUTURE questions.

Target outcomes:

Below is the specific result you will attempt to reproduce (quoted directly from the results section of Experiment 2):

According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjectS factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2). Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001, and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001. (de la Fuente et al., 2014, p. 1685).

Step 1: Load packages

library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files

# #optional packages/functions:
# library(afex) # anova functions
# library(ez) # anova functions 2
# library(scales) # for plotting
# std.err <- function(x) sd(x)/sqrt(length(x)) # standard error

Step 2: Load data

# Just Experiment 2
data_path <- 'data/DeLaFuenteEtAl_2014_RawData.xls'
d <- read_excel(data_path, sheet=3)

Step 3: Tidy data

d_tidy <- d |>
  mutate(item_num = as.numeric(strsplit(item, "\\.")[[1]][1]),
         item_text = strsplit(item, "\\.")[[1]][2]) |>
  select(-item)
colnames(d_tidy)[4] = "agreement"

Step 4: Run analysis

Pre-processing

d_mutated <- d_tidy |>
  mutate(group = ifelse(group == "young Spaniard", "Spaniards", "Moroccans"),
         subscale = ifelse(subscale == "PAST", "Past-Focused Statements", "Future-Focused Statements")) |>
  mutate(group = factor(group, levels=c("Spaniards", "Moroccans")),
         subscale = factor(subscale, levels=c("Past-Focused Statements", 
                                              "Future-Focused Statements")),
         participant = ifelse(group == "Spaniards", participant + 40, participant))
d_agg <- d_mutated |>
  group_by(participant, group, subscale) |>
  summarize(mean_agreement = mean(agreement), .groups="keep")

Descriptive statistics

Try to recreate Figure 2 (fig2.png, also included in the same folder as this Rmd file):

d_agg |>
  rev() |>
  group_by(group, subscale) |>
  summarize(mean_rating = mean(mean_agreement),
            se_rating = sd(mean_agreement) / sqrt(n()), .groups="keep") |>
  ggplot(aes(x=subscale, y=mean_rating, fill=subscale)) +
  facet_grid(. ~ group) +
  geom_bar(stat="identity") + 
  geom_errorbar(aes(ymin=mean_rating - se_rating/2, ymax=mean_rating + se_rating/2)) +
  coord_cartesian(ylim=c(2,4)) +
  theme(axis.text.x = element_blank())

Inferential statistics

According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjects factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2).

# reproduce the above results here
res <-  aov(mean_agreement ~ group * subscale + Error(participant/subscale), data=d_agg)
summary(res)

## 
## Error: participant
##       Df Sum Sq Mean Sq
## group  1  1.098   1.098
## 
## Error: participant:subscale
##          Df Sum Sq Mean Sq
## subscale  1  9.362   9.362
## 
## Error: Within
##                 Df Sum Sq Mean Sq F value  Pr(>F)   
## group            1   0.07  0.0681   0.188 0.66495   
## subscale         1   1.57  1.5691   4.337 0.03895 * 
## group:subscale   1   3.08  3.0835   8.523 0.00403 **
## Residuals      154  55.72  0.3618                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

This is not close to what they report in the paper, but I haven’t been able to figure out what is different. I also don’t understand their degrees of freedom, since they report 78.

Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001,

# reproduce the above results here
library(tidymodels)
d_agg |>
  filter(subscale == "Past-Focused Statements") |>
  t_test(formula = mean_agreement ~ group,
         order=c("Moroccans", "Spaniards"))

## # A tibble: 1 x 7
##   statistic  t_df  p_value alternative estimate lower_ci upper_ci
##       <dbl> <dbl>    <dbl> <chr>          <dbl>    <dbl>    <dbl>
## 1      4.00  76.9 0.000143 two.sided      0.618    0.311    0.926

and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001.(de la Fuente et al., 2014, p. 1685)

# reproduce the above results here
d_agg |>
  filter(subscale == "Future-Focused Statements") |>
  t_test(formula = mean_agreement ~ group,
         order=c("Moroccans", "Spaniards"))

## # A tibble: 1 x 7
##   statistic  t_df p_value alternative estimate lower_ci upper_ci
##       <dbl> <dbl>   <dbl> <chr>          <dbl>    <dbl>    <dbl>
## 1     -3.36  73.0 0.00123 two.sided     -0.372   -0.593   -0.152

The t-statistics have their signs flipped here compared to the original results, but that’s just due to the order of the factor levels.

Step 5: Reflection

I spent about 3 hours on this and was not able to correctly reproduce the results.

Were you able to reproduce the results you attempted to reproduce? If not, what part(s) were you unable to reproduce?

I was not quite able to reproduce the results I attempted to reproduce. My figure was about the same as the figure from the original paper, but it looks like my error bars are narrower. My t-test results were almost the same as those from the original paper, but my ANOVA results were very different. I think the differences in t-tests were due to issues with the data, but differences in ANOVA results probably had something to do with errors in my implementation.

How difficult was it to reproduce your results?

It was quite difficult to reproduce my results. I struggled with preparing the data and figuring out how to run the mixed ANOVA. It also looks like there were some problems with the data that impeded my attempts to reproduce this.

What aspects made it difficult? What aspects made it easy?

It looks like one of the participants’ data was mislabeled. Rows 501 through 521 are labelled as participant 40, but participants 24 and 25 are on either side of it. I filled this in with a mixture of 24s and 25s based on when the item numbers go back to 1. However, even after that my t-test results are slightly different (differences of ~0.04 in the F-statistics). I also struggled to figure out how to do a mixed ANOVA.

Reproducibility Report: Group B Choice 3