True and False Vignettes - General MTurk Sample
Download & load packages
In this study, participants read 26 stories about everyday topics (e.g., microwaving your food); 13 were true claims about the topic (e.g., microwave radiation does not cause cancer) and 13 contained false claims (e.g., microwave radiation causes cancer). Stories were presented in a random order. Unlike many other studies on this topic, participants also had to answer a comprehension check question after each story (on the following page) to ensure they were reading. The comprehension check question was unrelated to the true or false topic of the story.
Participants then completed what they were told to be a general knowledge assessment, which involved answering 39 claims as either true or false (binary, forced choice). Statements were presented one at a time in a random order. 26 of those statements at test were topically related to the previously-seen stories, and 13 were new claims. The statement judged at test was either presented in its accurate (e.g., microwave radiation does not cause cancer, true or false?) or inaccurate form ( microwave radiation causes cancer, true or false?). Thus, the study followed a 3 (story accuracy: true, false, or control / not read) x 2 (test statement accuracy: true or false) within subjects design. Story accuracy and test statement accuracy were counterbalanced across 6 possible conditions, which participants were randomly and evenly assigned to at the beginning of the experiment by Qualtrics.
Load data
longdata <- readr::read_csv("vignette-longdata.csv", col_names = TRUE)## Parsed with column specification:
## cols(
## .default = col_double(),
## ID = col_character(),
## ReadingCheck1 = col_character(),
## Gender1 = col_character(),
## Race1 = col_character(),
## Comments = col_character(),
## Item = col_character(),
## Response = col_logical(),
## Topic = col_character(),
## Story_Accuracy = col_character(),
## Test_Accuracy = col_logical()
## )## See spec(...) for full column specifications.Table with descriptive statistics by story type (true, false, or control)
summary <- longdata %>%
group_by(Story_Accuracy) %>%
summarise(Prop = sum(Incorrect==1, na.rm = TRUE)/n(),
sd = sqrt((sum(Incorrect==1, na.rm = TRUE)/n())*(1-(sum(Incorrect==1,na.rm = TRUE)/n()))),
n = n(),
SE = ((sum(Incorrect==1,na.rm = TRUE)/n())*(1-(sum(Incorrect==1,na.rm = TRUE)/n())))/sqrt(n())) %>%
print()## # A tibble: 3 x 5
## Story_Accuracy Prop sd n SE
## * <chr> <dbl> <dbl> <int> <dbl>
## 1 FALSE 0.383 0.486 546 0.0101
## 2 NEUTRAL 0.291 0.454 546 0.00883
## 3 TRUE 0.231 0.421 546 0.00760true <- within(longdata,Story_Accuracy == "TRUE",longdata$Incorrect)Graph 1 - How do error rates differ as a function of reading false as compared to true or no information at exposure?
# split by story accuracy
longdata %>%
group_by(Story_Accuracy) %>%
summarise(Prop = sum(Incorrect==1, na.rm = TRUE)/n(),
sd = sqrt((sum(Incorrect==1, na.rm = TRUE)/n())*(1-(sum(Incorrect==1,na.rm = TRUE)/n()))),
n = n(),
SE = ((sum(Incorrect==1,na.rm = TRUE)/n())*(1-(sum(Incorrect==1,na.rm = TRUE)/n())))/sqrt(n())) %>%
ggplot() + aes(x = Story_Accuracy, y = Prop, fill = Story_Accuracy) +
geom_bar(stat = "summary", fun.y = "mean", position = "dodge") + labs(x = "Accuracy of Claim in Stroy", y = "Proportion of Judgment Errors at Test") +
theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank(),panel.background = element_blank(), axis.line = element_line(colour = "black"), legend.title = element_blank()) +
geom_errorbar(aes(ymin=Prop-SE, ymax=Prop+SE), position=position_dodge(width=0.9), width=.1) +
geom_text(aes(label=round(Prop,digits=2)), position=position_dodge(width=.9), vjust=5) +
scale_fill_manual(values=c("#636363","#bdbdbd","#f0f0f0")) %>%
print()## <ggproto object: Class ScaleDiscrete, Scale, gg>
## aesthetics: fill
## axis_order: function
## break_info: function
## break_positions: function
## breaks: waiver
## call: call
## clone: function
## dimension: function
## drop: TRUE
## expand: waiver
## get_breaks: function
## get_breaks_minor: function
## get_labels: function
## get_limits: function
## guide: legend
## is_discrete: function
## is_empty: function
## labels: waiver
## limits: NULL
## make_sec_title: function
## make_title: function
## map: function
## map_df: function
## n.breaks.cache: NULL
## na.translate: TRUE
## na.value: NA
## name: waiver
## palette: function
## palette.cache: NULL
## position: left
## range: <ggproto object: Class RangeDiscrete, Range, gg>
## range: NULL
## reset: function
## train: function
## super: <ggproto object: Class RangeDiscrete, Range, gg>
## reset: function
## scale_name: manual
## train: function
## train_df: function
## transform: function
## transform_df: function
## super: <ggproto object: Class ScaleDiscrete, Scale, gg>
Graph 2 - How do error rates differ as a function of reading false as compared to true or no information at exposure, AND having rated true or false statements at test?
longdata %>%
group_by(Story_Accuracy, Test_Accuracy) %>%
summarise(Prop = sum(Incorrect==1, na.rm = TRUE)/n(),
sd = sqrt((sum(Incorrect==1, na.rm = TRUE)/n())*(1-(sum(Incorrect==1,na.rm = TRUE)/n()))),
n = n(),
SE = ((sum(Incorrect==1,na.rm = TRUE)/n())*(1-(sum(Incorrect==1,na.rm = TRUE)/n())))/sqrt(n())) %>%
ggplot() + aes(x = Story_Accuracy, y = Prop, fill = Test_Accuracy) +
geom_bar(stat = "summary", fun.y = "mean", position = "dodge") +
labs(x = "Accuracy of Claim in Stroy", y = "Proportion of Judgment Errors at Test", fill = "Accuracy Test Statement") +
theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank(),panel.background = element_blank(), axis.line = element_line(colour = "black")) +
geom_errorbar(aes(ymin=Prop-SE, ymax=Prop+SE), position=position_dodge(width=0.9), width=.1) +
geom_text(aes(label=round(Prop,digits=2)), position=position_dodge(width=.9), vjust=5) +
scale_fill_manual(values=c("#636363","#f0f0f0")) %>%
print()## `summarise()` has grouped output by 'Story_Accuracy'. You can override using the `.groups` argument.## <ggproto object: Class ScaleDiscrete, Scale, gg>
## aesthetics: fill
## axis_order: function
## break_info: function
## break_positions: function
## breaks: waiver
## call: call
## clone: function
## dimension: function
## drop: TRUE
## expand: waiver
## get_breaks: function
## get_breaks_minor: function
## get_labels: function
## get_limits: function
## guide: legend
## is_discrete: function
## is_empty: function
## labels: waiver
## limits: NULL
## make_sec_title: function
## make_title: function
## map: function
## map_df: function
## n.breaks.cache: NULL
## na.translate: TRUE
## na.value: NA
## name: waiver
## palette: function
## palette.cache: NULL
## position: left
## range: <ggproto object: Class RangeDiscrete, Range, gg>
## range: NULL
## reset: function
## train: function
## super: <ggproto object: Class RangeDiscrete, Range, gg>
## reset: function
## scale_name: manual
## train: function
## train_df: function
## transform: function
## transform_df: function
## super: <ggproto object: Class ScaleDiscrete, Scale, gg>
Analyzing error rates as a function of exposure to true, false, or no story
# setting "neutral (or control condition) as referent condition for subsequent analyses
longdata <- longdata %>%
within(., Story_Accuracy <- as.factor(Story_Accuracy)) %>%
within(., Story_Accuracy <- relevel(Story_Accuracy, ref = "NEUTRAL"))
glm1 <- glmer(Incorrect ~ Story_Accuracy + (1|ID) + (1|Item), family = binomial, data = longdata)
summary(glm1)## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: Incorrect ~ Story_Accuracy + (1 | ID) + (1 | Item)
## Data: longdata
##
## AIC BIC logLik deviance df.resid
## 1732.9 1759.9 -861.4 1722.9 1633
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.3215 -0.5829 -0.3434 0.6676 8.4100
##
## Random effects:
## Groups Name Variance Std.Dev.
## ID (Intercept) 0.719 0.8479
## Item (Intercept) 1.166 1.0799
## Number of obs: 1638, groups: ID, 42; Item, 39
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -1.1873 0.2440 -4.865 1.14e-06 ***
## Story_AccuracyFALSE 0.5120 0.1471 3.480 0.000501 ***
## Story_AccuracyTRUE -0.4359 0.1567 -2.781 0.005419 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) S_AFAL
## Stry_AFALSE -0.324
## Stry_AcTRUE -0.285 0.485# Participants produce significantly more errors after exposure to false as compared to control/neutral info
# Participants produce significantly fewer errors after exposure to true as compared to control/neutral infoInteraction between influence of reading false stories and extent to which participants possess accurate prior knowledge of the claim in the story (from prior norming)
longdata %>%
ggplot() +
aes(x = Prop_Known, color = factor(Story_Accuracy), group = Story_Accuracy, y = Correct) +
labs(x = "How Well Known Claim is", y = "Proportion Correct Answer at Test") + geom_smooth(method = "lm") + labs(color = "Accuracy of Claim in Story") + theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank(),panel.background = element_blank(), axis.line = element_line(colour = "black")) + scale_color_manual(values=c("#E7E944","#FF5733","#61DD7E")) %>% print()## <ggproto object: Class ScaleDiscrete, Scale, gg>
## aesthetics: colour
## axis_order: function
## break_info: function
## break_positions: function
## breaks: waiver
## call: call
## clone: function
## dimension: function
## drop: TRUE
## expand: waiver
## get_breaks: function
## get_breaks_minor: function
## get_labels: function
## get_limits: function
## guide: legend
## is_discrete: function
## is_empty: function
## labels: waiver
## limits: NULL
## make_sec_title: function
## make_title: function
## map: function
## map_df: function
## n.breaks.cache: NULL
## na.translate: TRUE
## na.value: NA
## name: waiver
## palette: function
## palette.cache: NULL
## position: left
## range: <ggproto object: Class RangeDiscrete, Range, gg>
## range: NULL
## reset: function
## train: function
## super: <ggproto object: Class RangeDiscrete, Range, gg>
## reset: function
## scale_name: manual
## train: function
## train_df: function
## transform: function
## transform_df: function
## super: <ggproto object: Class ScaleDiscrete, Scale, gg>
glm2 <- longdata %>%
glmer(Incorrect ~ Story_Accuracy*Prop_Known + (1|ID) + (1|Item), family = binomial, data = .)
summary(glm2)## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: Incorrect ~ Story_Accuracy * Prop_Known + (1 | ID) + (1 | Item)
## Data: .
##
## AIC BIC logLik deviance df.resid
## 1686.1 1729.3 -835.1 1670.1 1630
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.3223 -0.5654 -0.3283 0.6417 7.8616
##
## Random effects:
## Groups Name Variance Std.Dev.
## ID (Intercept) 0.7409 0.8608
## Item (Intercept) 0.2459 0.4959
## Number of obs: 1638, groups: ID, 42; Item, 39
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 3.3624 0.5653 5.948 2.71e-09 ***
## Story_AccuracyFALSE -0.9435 0.5941 -1.588 0.11227
## Story_AccuracyTRUE -1.6730 0.6218 -2.690 0.00714 **
## Prop_Known -6.3266 0.7781 -8.130 4.28e-16 ***
## Story_AccuracyFALSE:Prop_Known 2.1243 0.8370 2.538 0.01115 *
## Story_AccuracyTRUE:Prop_Known 1.8399 0.8912 2.065 0.03896 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) St_AFALSE St_ATRUE Prp_Kn S_AFALSE:
## Stry_AFALSE -0.582
## Stry_AcTRUE -0.561 0.529
## Prop_Known -0.940 0.593 0.575
## S_AFALSE:P_ 0.578 -0.968 -0.524 -0.630
## S_ATRUE:P_K 0.544 -0.513 -0.967 -0.592 0.542Significant story accuracy x well-known interactions suggest people’s validity judgment responses are more influenced by …
1- reading true stories when the claim is lesser known
2- reading false stories when the claim is more well-known