For this exercise, please try to reproduce the results from Experiment 2 of the associated paper (de la Fuente, Santiago, Roman, Dumitrache, & Casasanto, 2014). The PDF of the paper is included in the same folder as this Rmd file.

Methods summary:

Researchers tested the question of whether temporal focus differs between Moroccan and Spanish cultures, hypothesizing that Moroccans are more past-focused, whereas Spaniards are more future-focused. Two groups of participants (\(N = 40\) Moroccan and \(N=40\) Spanish) completed a temporal-focus questionnaire that contained questions about past-focused (“PAST”) and future-focused (“FUTURE”) topics. In response to each question, participants provided a rating on a 5-point Likert scale on which lower scores indicated less agreement and higher scores indicated greater agreement. The authors then performed a mixed-design ANOVA with agreement score as the dependent variable, group (Moroccan or Spanish, between-subjects) as the fixed-effects factor, and temporal focus (past or future, within-subjects) as the random effects factor. In addition, the authors performed unpaired two-sample t-tests to determine whether there was a significant difference between the two groups in agreement scores for PAST questions, and whether there was a significant difference in scores for FUTURE questions.

Target outcomes:

Below is the specific result you will attempt to reproduce (quoted directly from the results section of Experiment 2):

According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjectS factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2). Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001, and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001. (de la Fuente et al., 2014, p. 1685).

Step 1: Load packages

library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files

# #optional packages/functions:
library(afex) # anova functions
library(ez) # anova functions 2
library(scales) # for plotting
library(effectsize)
std.err <- function(x) sd(x)/sqrt(length(x)) # standard error

Step 2: Load data

# Just Experiment 2
data_path <- 'data/DeLaFuenteEtAl_2014_RawData-OSF-corrected_EHedit.xls'
d <- read_excel(data_path, sheet=3)

Step 3: Tidy data

colnames(d)
## [1] "group"                                                    
## [2] "participant"                                              
## [3] "subscale"                                                 
## [4] "item"                                                     
## [5] "Agreement (0=complete disagreement; 5=complete agreement)"
unique(d$group)
## [1] "Moroccan"       "young Spaniard"
data <- d %>% 
  rename("agreement" = "Agreement (0=complete disagreement; 5=complete agreement)") %>% 
  select(c("group", "participant", "subscale", "agreement")) %>% 
  mutate(group = if_else(group == "young Spaniard", "Spanish", group)) %>% 
  mutate(participant = paste(participant, group))


df <- data %>% 
  group_by(participant, subscale) %>%
  summarize(agreement = mean(agreement, na.rm = TRUE), group = first(group))

Step 4: Run analysis

Pre-processing

spain_past <- df %>% 
  filter(group == "Spanish", subscale == "PAST")

spain_future <- df %>% 
  filter(group == "Spanish", subscale == "FUTURE")

mor_past <- df %>% 
  filter(group == "Moroccan", subscale == "PAST") 

mor_future <- df %>% 
  filter(group == "Moroccan", subscale == "FUTURE")

Descriptive statistics

Try to recreate Figure 2 (fig2.png, also included in the same folder as this Rmd file):

vis_df <- data.frame(
  Group = factor(rep(c("Spaniards", "Moroccans"), each = 2),
                 levels = c("Spaniards", "Moroccans")),   # order on x-axis
  Scale = factor(rep(c("Past", "Future"), 2),
                 levels = c("Past", "Future")),           # Past bar left, Future right
  Rating = c(mean(spain_past$agreement),  mean(spain_future$agreement),
             mean(mor_past$agreement),    mean(mor_future$agreement)),
  SE = c(std.err(spain_past$agreement),   std.err(spain_future$agreement),
         std.err(mor_past$agreement),     std.err(mor_future$agreement))
)

pd <- position_dodge(0.9)

ggplot(vis_df, aes(x = Group, y = Rating, fill = Scale)) +
  geom_bar(stat = "identity", position = pd, width = 0.7, colour = "black") +
  geom_errorbar(
    aes(ymin = Rating - SE, ymax = Rating + SE),
    width = 0.15,
    position = pd
  ) +
  scale_fill_manual(
    values = c("grey60", "white"),
    name = NULL,
    labels = c("Past-Focused Statements", "Future-Focused Statements")
  ) +
  coord_cartesian(ylim = c(2, 4)) +  
  labs(y = "Rating", x = NULL) +
  theme_classic(base_size = 14) +
  theme(
    legend.position = "top",
    legend.background = element_blank(),
    legend.key.size = grid::unit(1.2, "lines"),
    axis.text.x = element_text(size = 12),
    axis.text.y = element_text(size = 12)
  )

Inferential statistics

According to a mixed analysis of variance (ANOVA) with group (Spanish vs. Moroccan) as a between-subjects factor and temporal focus (past vs. future) as a within-subjects factor, temporal focus differed significantly between Spaniards and Moroccans, as indicated by a significant interaction of temporal focus and group, F(1, 78) = 19.12, p = .001, ηp2 = .20 (Fig. 2).

# reproduce the above results here
anova_result <- aov_ez(
  id = "participant",
  dv = "agreement",
  data = df,
  between = "group",
  within = "subscale"
)
anova_result
## Anova Table (Type 3 tests)
## 
## Response: agreement
##           Effect    df  MSE         F  ges p.value
## 1          group 1, 78 0.21    2.88 + .011    .094
## 2       subscale 1, 78 0.51   8.10 ** .069    .006
## 3 group:subscale 1, 78 0.51 19.14 *** .148   <.001
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1
eta_squared(anova_result, partial = TRUE)
## # Effect Size for ANOVA (Type III)
## 
## Parameter      | Eta2 (partial) |       95% CI
## ----------------------------------------------
## group          |           0.04 | [0.00, 1.00]
## subscale       |           0.09 | [0.02, 1.00]
## group:subscale |           0.20 | [0.08, 1.00]
## 
## - One-sided CIs: upper bound fixed at [1.00].

Moroccans showed greater agreement with past-focused statements than Spaniards did, t(78) = 4.04, p = .001,

# reproduce the above results here
t.test(mor_past$agreement, spain_past$agreement)
## 
##  Welch Two Sample t-test
## 
## data:  mor_past$agreement and spain_past$agreement
## t = 4.0034, df = 76.872, p-value = 0.0001428
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.3106955 0.9256681
## sample estimates:
## mean of x mean of y 
##  3.293182  2.675000

and Spaniards showed greater agreement with future-focused statements than Moroccans did, t(78) = −3.32, p = .001.(de la Fuente et al., 2014, p. 1685)

# reproduce the above results here
t.test(spain_future$agreement,mor_future$agreement)
## 
##  Welch Two Sample t-test
## 
## data:  spain_future$agreement and mor_future$agreement
## t = 3.3637, df = 73.02, p-value = 0.001228
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.1517912 0.5932088
## sample estimates:
## mean of x mean of y 
##    3.4925    3.1200

Step 5: Reflection

Were you able to reproduce the results you attempted to reproduce? If not, what part(s) were you unable to reproduce?

I was not able to reproduce the results that I attempted to reproduce despite extensive effort. I was able to reproduce Fig. 2, although the exact numbers in the study were not provided by the authors. I also replaced the original data provided in the problem set with the corrected data (as shown in the results above) and still was not able to exactly replicate results. Overall, while the direction of the results was the same as those reported in the paper and within a rounding error’s distance away from those reported by the authors, the exact numbers for both the mixed ANOVA and t-test were different from that reported by de la Fuente et al. 

How difficult was it to reproduce your results?

It was difficult to reproduce the results, and I was unable to do so.

What aspects made it difficult? What aspects made it easy?

First, it was difficult since the authors provided little information on how data across multiple participants were aggregated. Based on the reported results, it seems like for each participant they averaged over the agreement scores across items; however, it is unclear if this aggregation was a mean as I did in the replication or another method (e.g., median). In addition, in my first attempts, the numbers were not reproducing, which I attributed to miscoding in the data. The authors report that they had 40 participants in both the Moroccan and Spanish; however, in the loaded data, it appears that participant ID 25 in the Moroccan group is missing, which might explain why the results do not match those reported in the paper. I then replaced the original data file with the corrected version and still was not able to exactly reproduce the results listed in the paper.

It was easy that the column names for the data and the provided values were easily mapped to the concept they were encoding.