For this exercise, please try to reproduce the results from Study 1 of the associated paper (Maglio & Polman, 2014). The PDF of the paper is included in the same folder as this Rmd file.
Researchers recruited 202 volunteers at a subway station in Toronto, Ontario, Canada. Half of the sample was traveling East, while the other half was traveling West. In a 2 (orientation: toward, away from) X 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) design, each participant was randomly asked to estimate how far one of the four stations felt to them (1= very close, 7= very far). Authors conducted a 2 X 4 ANOVA on distance estimates, and then tested differences in distance estimates between East and West-bound groups for each individual station.
Below is the specific result you will attempt to reproduce (quoted directly from the results section of Study 1):
We carried out a 2 (orientation: toward, away from) × 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) analysis of variance (ANOVA) on closeness ratings, which revealed no main effect of orientation, F < 1, and a main effect of station, F(3, 194) = 24.10, p < .001, ηp 2 = .27. This main effect was qualified by the predicted interaction between orientation and station, F(3, 194) = 16.28, p < .001, ηp2 = .20. We decomposed this interaction by the subjective-distance ratings between participants traveling east and west for each of the four subway stations. Westbound participants rated the stations to the west of Bay Street as closer than did eastbound participants; this effect was obtained for both the station one stop to the west (St. George, p < .001, ηp2 = .28) and the station two stops to the west (Spadina, p = .001, ηp2 = .20). The opposite pattern held true for stations to the east of Bay Street. Eastbound participants rated the stations to the east of Bay Street as closer than did westbound participants; this effect was obtained for both the station one stop to the east (Bloor-Yonge, p = .053, ηp2 = .08) and the station two stops to the east (Sherbourne, p < .001, ηp2 = .24). Figure 1 summarizes these results.
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
#optional packages:
library(lsr)
library(ggthemes)# Just Study 1
d <- read_excel ("data/S1_Subway.xlsx")The data are already tidy as provided by the authors.
# result from first sentence in inferential stats section
main_model <- lm(DISTANCE ~ DIRECTION * STN_NAME, data = d)
anova(main_model) # Df for residuals is 194 ## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION 1 0.713 0.7129 0.6644 0.416
## STN_NAME 3 75.158 25.0525 23.3492 6.011e-13 ***
## DIRECTION:STN_NAME 3 52.413 17.4710 16.2832 1.765e-09 ***
## Residuals 194 208.152 1.0729
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1# for station-specific analyses, subset data by station
d_st_george <- d %>% filter(STN_NAME == "STG")
d_spadina <- d %>% filter(STN_NAME == "SPAD")
d_bloor_yonge <- d %>% filter(STN_NAME == "B-Y")
d_sherbourne <- d %>% filter(STN_NAME == "SHER")We carried out a 2 (orientation: toward, away from) × 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) analysis of variance (ANOVA) on closeness ratings, which revealed no main effect of orientation, F < 1, and a main effect of station, F(3, 194) = 24.10, p < .001, ηp 2 = .27. This main effect was qualified by the predicted interaction between orientation and station, F(3, 194) = 16.28, p < .001, ηp2 = .20.
# reproduce the main effect of orientation
anova(main_model)["DIRECTION", ]## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION 1 0.71287 0.71287 0.6644 0.416# F(1, 194) = .66 (which is <1)
# reproduce the main effect of station
anova(main_model)["STN_NAME", ]## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## STN_NAME 3 75.158 25.052 23.349 6.011e-13 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1# F(3, 194) = 23.35 (slightly different), p < .001
eta_all <- effectsize::eta_squared(main_model, partial = TRUE)
eta_stn <- eta_all[eta_all$Parameter == "STN_NAME", ]
eta_stn # partial eta squared = .27 ## # Effect Size for ANOVA (Type I)
##
## Parameter | Eta2 (partial) | 95% CI
## -----------------------------------------
## STN_NAME | 0.27 | [0.18, 1.00]
##
## - One-sided CIs: upper bound fixed at [1.00].# reproduce the interaction between orientation and station
anova(main_model)["DIRECTION:STN_NAME", ]## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION:STN_NAME 3 52.413 17.471 16.283 1.765e-09 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1# F(3, 194) = 16.28, p < .001
eta_interact <- eta_all[eta_all$Parameter == "DIRECTION:STN_NAME", ]
eta_interact # partial eta squared = .20## # Effect Size for ANOVA (Type I)
##
## Parameter | Eta2 (partial) | 95% CI
## --------------------------------------------------
## DIRECTION:STN_NAME | 0.20 | [0.12, 1.00]
##
## - One-sided CIs: upper bound fixed at [1.00].We decomposed this interaction by the subjective-distance ratings between participants traveling east and west for each of the four subway stations. Westbound participants rated the stations to the west of Bay Street as closer than did eastbound participants; this effect was obtained for both the station one stop to the west (St. George, p < .001, ηp2 = .28) and the station two stops to the west (Spadina, p = .001, ηp2 = .20). The opposite pattern held true for stations to the east of Bay Street. Eastbound participants rated the stations to the east of Bay Street as closer than did westbound participants; this effect was obtained for both the station one stop to the east (Bloor-Yonge, p = .053, ηp2 = .08) and the station two stops to the east (Sherbourne, p < .001, ηp2 = .24). Figure 1 summarizes these results.
# reproduce results for St. George
model_st_george <- lm (DISTANCE ~ DIRECTION, data = d_st_george)
anova(model_st_george) # p < .001 ## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION 1 16.252 16.2521 18.793 7.229e-05 ***
## Residuals 49 42.375 0.8648
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1eta_st_george <- effectsize::eta_squared(model_st_george, partial = TRUE)
eta_st_george # partial eta squared = .28 ## # Effect Size for ANOVA
##
## Parameter | Eta2 | 95% CI
## -------------------------------
## DIRECTION | 0.28 | [0.11, 1.00]
##
## - One-sided CIs: upper bound fixed at [1.00].## reproduce results for Spadina
model_spadina <- lm (DISTANCE ~ DIRECTION, data = d_spadina)
anova(model_spadina) # p = .001 ## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION 1 13.100 13.1005 11.966 0.001131 **
## Residuals 49 53.645 1.0948
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1eta_spadina <- effectsize::eta_squared(model_spadina, partial = TRUE)
eta_spadina # partial eta squared = .20 ## # Effect Size for ANOVA
##
## Parameter | Eta2 | 95% CI
## -------------------------------
## DIRECTION | 0.20 | [0.06, 1.00]
##
## - One-sided CIs: upper bound fixed at [1.00].## reproduce results for Bloor-Yonge
model_bloor_yonge <- lm (DISTANCE ~ DIRECTION, data = d_bloor_yonge)
anova(model_bloor_yonge) # p = .053 ## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION 1 4.157 4.1567 3.9455 0.05285 .
## Residuals 47 49.517 1.0535
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1eta_bloor_yonge <- effectsize::eta_squared(model_bloor_yonge, partial = TRUE)
eta_bloor_yonge # partial eta squared = .08 ## # Effect Size for ANOVA
##
## Parameter | Eta2 | 95% CI
## -------------------------------
## DIRECTION | 0.08 | [0.00, 1.00]
##
## - One-sided CIs: upper bound fixed at [1.00].## reproduce results for Sherbourne
model_sherbourne <- lm (DISTANCE ~ DIRECTION, data = d_sherbourne)
anova(model_sherbourne) # p < .001 ## Analysis of Variance Table
##
## Response: DISTANCE
## Df Sum Sq Mean Sq F value Pr(>F)
## DIRECTION 1 19.306 19.3062 15.108 0.0003052 ***
## Residuals 49 62.615 1.2779
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1eta_sherbourne <- effectsize::eta_squared(model_sherbourne, partial = TRUE)
eta_sherbourne # partial eta squared = .24 ## # Effect Size for ANOVA
##
## Parameter | Eta2 | 95% CI
## -------------------------------
## DIRECTION | 0.24 | [0.08, 1.00]
##
## - One-sided CIs: upper bound fixed at [1.00].Were you able to reproduce the results you attempted to reproduce? If not, what part(s) were you unable to reproduce?
All results I attempted to reproduce were reproduced, a part from the F value for the main effect of station; the original F value reported was 24.10, but the F value I found was 23.35.
How difficult was it to reproduce your results?
Conceptually, identifying how to reproduce each of these results (i.e., which statistical tests to run) was relatively simple, despite some initial challenges with understanding the variable names. However, I used ChatGPT for assistance with some code.
What aspects made it difficult? What aspects made it easy?
The language from the paper made it somewhat easy to figure out which variables were the outcomes and which were the predictors when building the different models. It also helped that the dataset only had a few columns; this made up for the fact that the variable names were initially not very intuitive. The coding aspect was slightly difficult; I looked to ChatGPT for help with the code for extracting each of the main effects and effect sizes from the larger ANOVA model and for reproducing all the partial eta squared values.