Reproducing Manylabs Results
anjie & jiayi
8/6/2021
Data Prepping
Analysis Script1.Preparing SPSS Dataset for AnalysesSPSS syntax/* Open “ML2.Slate1.Random3rdDE.xlsx”in SPSS.
loading the data
The following syntax prepares the dataset for the subsequent analyses. RECODE miya1.1 (1=1) INTO Condition. RECODE miya2.1 (1=2) INTO Condition.
d <- d %>%
select(.id, ResponseID, Country, Language, Weird, starts_with("miya")) %>%
mutate(
condition = case_when(
miya1.1 == 1 ~ "condition_1",
miya2.1 == 1 ~ "condition_2"
)
) EXECUTE.DO IF (Condition = 1).RECODE miya1.4 miya1.6 miya1.7 miya1.8 (ELSE=Copy) INTO TrueAttitude Average Constraint Persuasive.END IF. DO IF (Condition = 2).RECODE miya2.4 miya2.6 miya2.7 miya2.8 (ELSE=Copy) INTO TrueAttitude Average Constraint Persuasive. END IF.EXECUTE
d_condition1 <- d %>%
filter(condition == "condition_1") %>%
rename(
true_attitude = miya1.4,
average = miya1.6,
constraint = miya1.7,
persuasive = miya1.8
)
d_condition2 <- d %>%
filter(condition == "condition_2") %>%
rename(
true_attitude = miya2.4,
average = miya2.6,
constraint = miya2.7,
persuasive = miya2.8
)
d_cleaned <- bind_rows(d_condition1, d_condition2) %>%
select(.id, ResponseID, Country, Language, Weird, condition, true_attitude, average, constraint, persuasive)d_cleaned ## # A tibble: 2,419 x 10
## .id ResponseID Country Language Weird condition true_attitude average
## <chr> <chr> <chr> <chr> <dbl> <chr> <dbl> <dbl>
## 1 ML2_Slate… R_0jsBYzHr… Brazil Portugu… 0 conditio… 13 12
## 2 ML2_Slate… R_bKrfQzSl… Brazil Portugu… 0 conditio… 14 12
## 3 ML2_Slate… R_0x1MrDkW… Brazil Portugu… 0 conditio… 15 8
## 4 ML2_Slate… R_3Wtat6Ch… Brazil Portugu… 0 conditio… 8 8
## 5 ML2_Slate… R_7Ql9fUfH… Brazil Portugu… 0 conditio… NA NA
## 6 ML2_Slate… R_0fhpY7Ci… Brazil Portugu… 0 conditio… 15 8
## 7 ML2_Slate… R_6m37xbex… Brazil Portugu… 0 conditio… 9 6
## 8 ML2_Slate… R_7PUKCjwt… Brazil Portugu… 0 conditio… 15 11
## 9 ML2_Slate… R_3Jyl9cxL… Brazil Portugu… 0 conditio… 12 1
## 10 ML2_Slate… R_7V7f89ef… Brazil Portugu… 0 conditio… 8 9
## # … with 2,409 more rows, and 2 more variables: constraint <dbl>,
## # persuasive <dbl>Check assumption
Run a 2 (Country: US vs. Japan) ANOVA with the perceived persuasiveness as the DV to test if the perceived persuasiveness differs between US and Japan.
d_cleaned %>%
filter(Country %in% c("USA", "Japan")) %>%
ggplot(aes(x = Country, y = persuasive, color = Country)) +
geom_jitter(alpha = .3, width = .3) +
stat_summary(fun.data = "mean_cl_boot") +
facet_wrap(~condition) +
theme_classic()
d_cleaned %>%
filter(Country %in% c("USA", "China")) %>%
ggplot(aes(x = Country, y = persuasive, color = Country)) +
geom_jitter(alpha = .3, width = .3) +
stat_summary(fun.data = "mean_cl_boot") +
facet_wrap(~condition) +
theme_classic()
We first confirmed that the essay used in Many Labs 2 was equally low in perceived persuasiveness in the two groups, F(1, 1661) = 0.008, p = .927 (a requirement to test the cross-cultural difference in correspondence bias)
haha i don’t know how to run anova in r but here’s a tutorial: https://www.scribbr.com/statistics/anova-in-r/
USE ALL.COMPUTE filter_\(=(Country = 'USA' or country = 'Japan') .VARIABLE LABELS filter_\) “Country = ‘USA’ or country = ‘Japan’ (FILTER)”. VALUE LABELS filter_$ 0 ‘Not Selected’ 1 ‘Selected’. FORMATS filter_$ (f1.0).FILTER BY filter_$.EXECUTE.
UNIANOVA Persuasive BY Country
METHOD=SSTYPE(3)
INTERCEPT=INCLUDE
PRINT=DESCRIPTIVE
CRITERIA=ALPHA(.05) /DESIGN=Country.
Japan
us_japan_comparison_d <- d_cleaned %>%
filter(Country %in% c("USA", "Japan"))
us_japan_comparison_d %>%
group_by(Country) %>%
summarise(n = n())## # A tibble: 2 x 2
## Country n
## <chr> <int>
## 1 Japan 38
## 2 USA 794aov(persuasive ~ Country, data = us_japan_comparison_d) %>%
summary()## Df Sum Sq Mean Sq F value Pr(>F)
## Country 1 0 0.0015 0.001 0.98
## Residuals 823 2023 2.4584
## 7 observations deleted due to missingnesshaha mission failed (success?) can’t reproduce their results because the spreadsheet doesn’t really contain the same number of participants (we also tried to load the data in slate2 but there’s not even one single participant from japan there so lol)
China
us_china_comparison_d <- d_cleaned %>%
filter(Country %in% c("USA", "China"))
us_china_comparison_d %>%
ggplot(aes(x = Country, y = persuasive, color = Country)) +
geom_jitter(alpha = .3, width = .3) +
stat_summary(fun.data = "mean_cl_boot") +
facet_wrap(~condition) +
theme_classic()
> We first confirmed that the essay used in Many Labs 2 was equally low in perceived persuasiveness in the two groups, F(1, 1661) = 0.008, p = .927 (a requirement to test the cross-cultural difference in correspondence bias)
Now let’s see if US China meets the requirement here
aov(persuasive ~ Country, data = us_china_comparison_d) %>%
summary()## Df Sum Sq Mean Sq F value Pr(>F)
## Country 1 12.5 12.519 5.384 0.0205 *
## Residuals 914 2125.2 2.325
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 10 observations deleted due to missingnessso the assumption that CN and US participants’ perceived persuasiveness of the articles was not met, but we are brave enough to move on from the violation of assumption. let’s see what’s the next test
Actual correspondence bias test
Japan
The following syntax tests cultural differences in the magnitude of CB.
USE ALL.COMPUTE filter_\(=(Country = 'USA' or country = 'Japan'). VARIABLE LABELS filter_\) “Country = ‘USA’ or country = ‘Japan’ (FILTER)”. VALUE LABELS filter_$ 0 ‘Not Selected’ 1 ‘Selected’.FORMATS filter_$ (f1.0). FILTER BY filter_$.EXECUTE.UNIANOVA TrueAttitude BY Condition Country WITH Constraint
METHOD=SSTYPE(3)
INTERCEPT=INCLUDE
PLOT=PROFILE(ConditionCountry)
/EMMEANS=TABLES(ConditionCountry) WITH(Constraint=MEAN)
/PRINT=DESCRIPTIVE
/CRITERIA=ALPHA(.05)
/DESIGN=Constraint Condition Country Condition*Country.
There was a significant Culture x Condition interaction on the judgement of true attitude, controlling for perceived constraint, F(1, 820) = 4.23, p = .04, d = 0.7
us_japan_comparison_d %>%
ggplot(aes(x = Country, y = true_attitude, color = condition)) +
geom_jitter(alpha = .1, width = .3) +
stat_summary(fun.data = "mean_cl_boot") +
facet_wrap(~condition) +
theme_classic()
aov(true_attitude ~ Country * condition + constraint, data = us_japan_comparison_d) %>%
summary()## Df Sum Sq Mean Sq F value Pr(>F)
## Country 1 5 5 0.387 0.5341
## condition 1 7185 7185 556.732 <2e-16 ***
## constraint 1 25 25 1.971 0.1607
## Country:condition 1 55 55 4.234 0.0399 *
## Residuals 820 10582 13
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 7 observations deleted due to missingnessChina
can we see the same thing in China despite the violation of assumption?
us_china_comparison_d %>%
ggplot(aes(x = Country, y = true_attitude, color = condition)) +
geom_jitter(alpha = .1, width = .3) +
stat_summary(fun.data = "mean_cl_boot") +
facet_wrap(~condition) +
theme_classic()
aov(true_attitude ~ Country * condition + constraint, data = us_china_comparison_d) %>%
summary()## Df Sum Sq Mean Sq F value Pr(>F)
## Country 1 9 9 0.665 0.415126
## condition 1 9239 9239 708.760 < 2e-16 ***
## constraint 1 84 84 6.451 0.011255 *
## Country:condition 1 153 153 11.740 0.000639 ***
## Residuals 911 11875 13
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 10 observations deleted due to missingnessactually when controlling for persuasiveness it’s still significant?
aov(true_attitude ~ Country * condition + constraint + persuasive, data = us_china_comparison_d) %>%
summary()## Df Sum Sq Mean Sq F value Pr(>F)
## Country 1 9 9 0.692 0.405597
## condition 1 9196 9196 705.583 < 2e-16 ***
## constraint 1 88 88 6.721 0.009679 **
## persuasive 1 51 51 3.891 0.048848 *
## Country:condition 1 149 149 11.407 0.000763 ***
## Residuals 906 11808 13
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 14 observations deleted due to missingness