Reproducibility Report: Group A Choice 2
For this exercise, please try to reproduce the results from Study 1 of the associated paper (Maglio & Polman, 2014). The PDF of the paper is included in the same folder as this Rmd file.
Methods summary:
Researchers recruited 202 volunteers at a subway station in Toronto, Ontario, Canada. Half of the sample was traveling East, while the other half was traveling West. In a 2 (orientation: toward, away from) X 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) design, each participant was randomly asked to estimate how far one of the four stations felt to them (1= very close, 7= very far). Authors conducted a 2 X 4 ANOVA on distance estimates, and then tested differences in distance estimates between East and West-bound groups for each individual station.
Target outcomes:
Below is the specific result you will attempt to reproduce (quoted directly from the results section of Study 1):
We carried out a 2 (orientation: toward, away from) × 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) analysis of variance (ANOVA) on closeness ratings, which revealed no main effect of orientation, F < 1, and a main effect of station, F(3, 194) = 24.10, p < .001, ηp 2 = .27. This main effect was qualified by the predicted interaction between orientation and station, F(3, 194) = 16.28, p < .001, ηp2 = .20. We decomposed this interaction by the subjective-distance ratings between participants traveling east and west for each of the four subway stations. Westbound participants rated the stations to the west of Bay Street as closer than did eastbound participants; this effect was obtained for both the station one stop to the west (St. George, p < .001, ηp2 = .28) and the station two stops to the west (Spadina, p = .001, ηp2 = .20). The opposite pattern held true for stations to the east of Bay Street. Eastbound participants rated the stations to the east of Bay Street as closer than did westbound participants; this effect was obtained for both the station one stop to the east (Bloor-Yonge, p = .053, ηp2 = .08) and the station two stops to the east (Sherbourne, p < .001, ηp2 = .24). Figure 1 summarizes these results.
Step 1: Load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(magrittr) # double pipe
library(stats) # anova
library(effectsize) # eta^2
# #optional packages:
# library(lsr)
# library(ggthemes)Step 2: Load data
# Just Study 1
d <- read_excel ("data/S1_Subway.xlsx")Step 3: Tidy data
The data are already tidy as provided by the authors.
Step 4: Run analysis
Pre-processing
names(d) <- c("direction", "distance", "stnNumber", "stnName")
d$direction %<>% as.factor()
d$stnName %<>% factor(levels = c("SPAD", "STG", "B-Y", "SHER"))Inferential statistics
We carried out a 2 (orientation: toward, away from) × 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) analysis of variance (ANOVA) on closeness ratings, which revealed no main effect of orientation, F < 1, and a main effect of station, F(3, 194) = 24.10, p < .001, ηp 2 = .27. This main effect was qualified by the predicted interaction between orientation and station, F(3, 194) = 16.28, p < .001, ηp2 = .20.
# reproduce the main effects of orientation and station, and the interaction between orientation and station
fit <- aov(distance ~ direction * stnName, data = d)
anovaTest <- anova(fit)
effSize <- rbind(effectsize(anovaTest, ci = 0.95), NA)
combTable <- cbind(anovaTest, effSize) %>% subset(select = -c(Parameter, CI))
print(combTable)## Df Sum Sq Mean Sq F value Pr(>F)
## direction 1 0.7128713 0.7128713 0.6644037 4.160091e-01
## stnName 3 75.1575503 25.0525168 23.3492148 6.011283e-13
## direction:stnName 3 52.4131149 17.4710383 16.2831954 1.765498e-09
## Residuals 194 208.1521070 1.0729490 NA NA
## Eta_Sq_partial CI_low CI_high
## direction 0.003413072 0.0000000 0.03812055
## stnName 0.265284110 0.1603966 0.35725570
## direction:stnName 0.201151614 0.1031846 0.29158921
## Residuals NA NA NAse <- function(x) sd(x) / sqrt(length(x))
summaryD <- d %>% group_by(direction, stnNumber) %>%
summarise(meanDist = mean(distance), distSE = se(distance), .groups = "drop")
graph <- ggplot(summaryD, aes(x = stnNumber, y = meanDist, color = direction)) +
geom_line() +
ylim(0, 5) +
labs(x = "Station", y = "Subjective distance", color = "Direction") +
scale_color_hue(labels = c("Traveling east", "Traveling west")) +
scale_x_continuous(breaks = c(1, 2, 3, 4), labels = c("Spadina", "St. George", "Bloor-Yonge", "Sherbourne")) +
geom_errorbar(aes(ymin = meanDist - 2 * distSE, ymax = meanDist + 2 * distSE), width = .06)
# note: original graph caption suggested that error bars were ±1 SE, but they are actually ±2 SE (i.e. CI)
plot(graph)
Station-specific analyses
We decomposed this interaction by the subjective-distance ratings between participants traveling east and west for each of the four subway stations. Westbound participants rated the stations to the west of Bay Street as closer than did eastbound participants; this effect was obtained for both the station one stop to the west (St. George, p < .001, ηp2 = .28) and the station two stops to the west (Spadina, p = .001, ηp2 = .20). The opposite pattern held true for stations to the east of Bay Street. Eastbound participants rated the stations to the east of Bay Street as closer than did westbound participants; this effect was obtained for both the station one stop to the east (Bloor-Yonge, p = .053, ηp2 = .08) and the station two stops to the east (Sherbourne, p < .001, ηp2 = .24). Figure 1 summarizes these results.
St. George
# reproduce results for St. George
simpleMainEffect <- function(thisStn) {
dStn <- subset(d, stnName == thisStn)
anovaStn <- anova(lm(distance ~ direction, dStn))
esStn <- rbind(effectsize(anovaStn, ci = 0.95), NA)
combStn <- cbind(anovaStn, esStn) %>% subset(select = -c(Parameter, CI))
return(combStn)
}
simpleMainEffect("STG")## Df Sum Sq Mean Sq F value Pr(>F) Eta_Sq_partial CI_low
## direction 1 16.25207 16.2520664 18.79278 7.22865e-05 0.2772092 0.08823822
## Residuals 49 42.37538 0.8648038 NA NA NA NA
## CI_high
## direction 0.4595728
## Residuals NASpadina
## reproduce results for Spadina
simpleMainEffect("SPAD")## Df Sum Sq Mean Sq F value Pr(>F) Eta_Sq_partial CI_low
## direction 1 13.10048 13.100483 11.96623 0.001131006 0.1962763 0.03677735
## Residuals 49 53.64462 1.094788 NA NA NA NA
## CI_high
## direction 0.3833364
## Residuals NABloor-Yonge
## reproduce results for Bloor-Yonge
simpleMainEffect("B-Y")## Df Sum Sq Mean Sq F value Pr(>F) Eta_Sq_partial CI_low
## direction 1 4.156747 4.156747 3.945477 0.05284615 0.0774451 0
## Residuals 47 49.516722 1.053547 NA NA NA NA
## CI_high
## direction 0.2517543
## Residuals NASherbourne
## reproduce results for Sherbourne
simpleMainEffect("SHER")## Df Sum Sq Mean Sq F value Pr(>F) Eta_Sq_partial CI_low
## direction 1 19.30618 19.306184 15.10816 0.0003052116 0.2356667 0.05976062
## Residuals 49 62.61538 1.277865 NA NA NA NA
## CI_high
## direction 0.4214565
## Residuals NAStep 5: Reflection
Were you able to reproduce the results you attempted to reproduce? If not, what part(s) were you unable to reproduce?
The results were largely reproducible, apart from a slight difference in F value for the main effect of direction. As a slight aside, the original paper presented the simple main effect for the Bloor-Yonge station in a manner that suggested that the hypothesised effect was found; however, the p value is quite high (.053), especially considering that some correction for multiple comparisons is needed (which would lower the alpha value).
How difficult was it to reproduce your results?
It was not very difficult as the data were already tidy and the analysis comprised simple ANOVAs.
What aspects made it difficult? What aspects made it easy?
Tidy and available data, clear descriptions of the analysis performed, and comprehensive reporting made the reproduction relatively easy.