Overview
This document reports the analysis pipeline and results for a study
testing directionally motivated reasoning in the evaluation of
belief-congruent and belief-incongruent news messages, adapting the
incentivized news-evaluation paradigm from Thaler (2024) and Borghi et
al. (2026).
Participants completed a belief-updating task on six topics (four
political: COVID-19 vaccines, climate change, media bias, and same-sex
adoption; two general-knowledge: cats, brain). On each topic, they
stated a numeric guess intended to reflect the median of their
subjective belief distribution, then received a message stating that the
correct answer was higher or lower than their guess, attributed with
equal prior probability to a True News source (always accurate) or a
Fake News source (always inaccurate). Participants then rated how likely
it was that the message came from the True News source (0-100).
The manipulated factor was message congruence: whether the feedback
direction was consistent (congruent) or inconsistent (incongruent) with
the participant’s previously stated attitude on that topic. Because a
Bayesian reasoner whose guess is correctly placed at their own
subjective median should find higher- and lower-than-guess messages
equally diagnostic of source veracity, a systematic difference in
perceived truthfulness as a function of message congruence constitutes
evidence of directionally motivated reasoning. All hypotheses, exclusion
criteria, and confirmatory/ sensitivity/exploratory models were
preregistered on AsPredicted prior to data collection.
Project
organization
|
Script
|
What it does
|
Output(s) produced
|
|
R/00_functions.R
|
Shared constants: answer keys for scoring
CPT/attention-check/understanding items, the topic-to-column map, and
helper functions (duplicate-column coalescing, choice-text number
parsing, safe lmer fitting).
|
(no direct output – sourced by every other script)
|
|
R/01_import_and_clean.R
|
Reads the raw survey export, resolves duplicate-named columns, recovers
numeric values from mixed choice-text formatting, computes
participant-level scores (CPT, AOT, humility, understanding), and
applies both a confirmatory and an exploratory exclusion rule.
|
output/tables/participants_clean.rds
|
|
R/02_build_long_format.R
|
Reshapes the six topics from wide to long format (one row per
participant x topic); builds congruence, domain, and prior-strength
effect codes; flags valid response times.
|
output/tables/task_long_rebuilt.rds, task_long_rebuilt.csv
|
|
R/03_confirmatory_models.R
|
Fits the preregistered confirmatory model and both preregistered
sensitivity analyses (random-slope model, aggregate paired comparison),
on both exclusion samples.
|
output/tables/confirmatory_results.rds, confirmatory_model_*_tidy.csv
|
|
R/04_exploratory_models.R
|
Fits the 7 remaining preregistered exploratory models: general-topic
effect, domain comparison, confidence, conflict, response time, task
understanding (pre/post), everyday evidence-updating, prior attitude
strength.
|
output/tables/exploratory_results.rds, exploratory_*_tidy.csv
|
|
R/05_individual_differences.R
|
Computes the aggregate motivated-reasoning score per participant and the
preregistered validity correlations/moderation analyses (AOT, humility,
CPT).
|
output/tables/individual_differences_results.rds,
individual_differences_data.csv
|
|
R/06_figures.R
|
Generates the descriptive figures used in this report and the
manuscript.
|
output/figures/fig1-3*.png
|
|
R/07_topic_interaction_check.R (supplementary, not preregistered)
|
Not part of the preregistration. Quantifies how much including topic
changes the congruence estimate (a congruence-only model), and tests
whether the pooled congruence effect is homogeneous across the four
political topics via a Congruence x Topic interaction, its omnibus
F-test, and per-topic simple slopes.
|
output/tables/topic_interaction_check_results.rds, supp_*.csv
|
|
R/run_all.R
|
Convenience driver: sources 01-07 in order.
|
(all of the above)
|
The pipeline can be reproduced by running R/run_all.R
from the project root, or by knitting this document.
Data coding and
exclusion notes
|
Issue
|
Resolution
|
|
Preregistered exclusion criteria
|
The preregistration specifies a target of N = 365 and treats
attention-check failure as an exploratory robustness check rather than a
confirmatory exclusion criterion.
|
|
Sample eligibility
|
Geolocation coordinates in the raw export confirm the sample was
recruited entirely from the continental United States, consistent with
the preregistered eligibility criteria; incidental GBP currency
labelling on two items is a template artifact.
|
|
Duplicate / incomplete Prolific submissions
|
Ten Prolific IDs were associated with more than one submission attempt;
in every case exactly one attempt was complete and the remainder were
abandoned, incomplete attempts, so no genuine duplicate completions
occur in the sample. The first complete submission (by start time) is
retained wherever more than one exists.
|
|
CPT and task-understanding item scoring
|
These multiple-choice items are exported as choice text rather than
numeric codes; scoring matches responses against the exact
correct-answer text (see KEY in
00_functions.R).
|
|
Demographic attention-check coding
|
The instructed-response item (ai_check) is scored using the
numeric value corresponding to the instructed response, consistent with
the response pattern of the large majority of finished respondents.
|
|
CPT bat-and-ball item wording
|
The bat-and-ball item’s question stem is denominated in US dollars, but
its response options are labelled in British pence, a wording
inconsistency inherited from an earlier study template. The item is
scored by matching the intended numeric answer irrespective of the
currency label; this is noted as an instrument limitation.
|
|
Sample size
|
Data collection yielded 366 complete submissions.
|
Several Likert-type items in the raw export mix bare numeric
responses with endpoint-labelled text (e.g.,
"1 (Strongly Disagree)" vs. plain "2"),
affecting prior attitudes, AOT items, humility, confidence, and conflict
ratings. parse_leading_number() in
00_functions.R extracts the leading numeric value for all
such items.
Sample and
exclusions
428 total submissions (366 complete, 62 incomplete)
|
Rule
|
Participants retained
|
Participants excluded
|
|
Confirmatory (complete submissions only)
|
366
|
62
|
|
Exploratory (+ both attention checks)
|
364
|
64
|
Sample
characteristics (N = 366, confirmatory sample)
|
Characteristic
|
Value
|
|
Age, M (SD)
|
42.7 (13.7)
|
|
Gender: Female / Male / Other
|
188 / 176 / 2
|
|
Education: Bachelor’s or higher
|
61.2%
|
|
Political ideology (0-10, general), M (SD)
|
4.42 (3.04)
|
|
Ideological self-categorization: Liberal / Moderate / Conservative /
Other
|
156 / 78 / 115 / 17
|
Confirmatory
analysis
Confirmatory model, N = 366 participants, 1464 observations
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
57.052
|
0.805
|
70.838
|
366
|
0.000
|
|
|
congruence_ec
|
9.789
|
1.014
|
9.658
|
1098
|
0.000
|
|
|
topic1
|
1.558
|
0.877
|
1.777
|
1098
|
0.076
|
|
|
topic2
|
1.949
|
0.877
|
2.222
|
1098
|
0.026
|
|
|
topic3
|
-0.223
|
0.877
|
-0.254
|
1098
|
0.800
|
Sensitivity
analyses
Random-slope sensitivity model
|
effect
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
fixed
|
|
(Intercept)
|
57.052
|
0.805
|
70.838
|
366.000
|
0.000
|
|
fixed
|
|
congruence_ec
|
9.774
|
1.054
|
9.276
|
362.624
|
0.000
|
|
fixed
|
|
topic1
|
1.576
|
0.873
|
1.805
|
1096.261
|
0.071
|
|
fixed
|
|
topic2
|
1.603
|
0.873
|
1.837
|
1096.259
|
0.066
|
|
fixed
|
|
topic3
|
-0.034
|
0.873
|
-0.039
|
1096.167
|
0.969
|
|
ran_pars
|
ResponseId
|
sd__(Intercept)
|
12.140
|
|
|
|
|
|
ran_pars
|
ResponseId
|
sd__congruence_ec
|
6.744
|
|
|
|
|
|
ran_pars
|
ResponseId
|
cor__(Intercept).congruence_ec
|
0.563
|
|
|
|
|
|
ran_pars
|
Residual
|
sd__Observation
|
18.976
|
|
|
|
|
Aggregate-score paired t-test (congruent vs. incongruent mean rating)
|
estimate
|
statistic
|
p.value
|
parameter
|
conf.low
|
conf.high
|
method
|
alternative
|
|
9.669
|
9.067
|
0
|
365
|
7.572
|
11.767
|
Paired t-test
|
two.sided
|
Exploratory
analyses
General-topic model
(cats/brain)
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
61.020
|
0.959
|
63.625
|
366
|
0.000
|
|
|
congruence_ec
|
4.167
|
1.239
|
3.362
|
366
|
0.001
|
|
|
topic1
|
0.189
|
0.620
|
0.305
|
366
|
0.761
|
Political
vs. general domain comparison
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
59.083
|
0.760
|
77.770
|
443.752
|
0.000
|
|
|
congruence_ec
|
6.978
|
0.878
|
7.943
|
1830.000
|
0.000
|
|
|
domain_ec
|
-4.252
|
1.388
|
-3.063
|
1830.000
|
0.002
|
|
|
topic1
|
1.653
|
0.948
|
1.743
|
1830.000
|
0.082
|
|
|
topic2
|
2.043
|
0.948
|
2.155
|
1830.000
|
0.031
|
|
|
topic3
|
-0.128
|
0.949
|
-0.135
|
1830.000
|
0.893
|
|
|
topic4
|
-3.190
|
0.948
|
-3.365
|
1830.000
|
0.001
|
|
|
congruence_ec:domain_ec
|
5.623
|
1.757
|
3.200
|
1830.000
|
0.001
|
Confidence
(1-7)
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
4.317
|
0.067
|
64.323
|
366
|
0.000
|
|
|
congruence_ec
|
0.222
|
0.057
|
3.928
|
1098
|
0.000
|
|
|
topic1
|
0.068
|
0.049
|
1.383
|
1098
|
0.167
|
|
|
topic2
|
0.262
|
0.049
|
5.346
|
1098
|
0.000
|
|
|
topic3
|
-0.171
|
0.049
|
-3.496
|
1098
|
0.000
|
Conflict (1-7)
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
3.244
|
0.081
|
39.807
|
366
|
0.000
|
|
|
congruence_ec
|
-0.212
|
0.061
|
-3.509
|
1098
|
0.000
|
|
|
topic1
|
-0.082
|
0.052
|
-1.562
|
1098
|
0.118
|
|
|
topic2
|
-0.177
|
0.052
|
-3.388
|
1098
|
0.001
|
|
|
topic3
|
0.140
|
0.052
|
2.675
|
1098
|
0.008
|
Log response
time
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
2.024
|
0.038
|
53.017
|
1038.809
|
0.000
|
|
|
congruence_ec
|
-0.027
|
0.030
|
-0.898
|
1095.689
|
0.369
|
|
|
topicclimate
|
-0.003
|
0.043
|
-0.071
|
1095.839
|
0.944
|
|
|
topicmedia
|
0.044
|
0.043
|
1.026
|
1095.858
|
0.305
|
|
|
topicadoption
|
0.044
|
0.043
|
1.025
|
1095.850
|
0.305
|
Pre-task
understanding interaction
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
57.052
|
0.800
|
71.304
|
366
|
0.000
|
|
|
congruence_ec
|
9.789
|
1.013
|
9.660
|
1098
|
0.000
|
|
|
pre_understanding_c
|
2.547
|
1.158
|
2.200
|
366
|
0.028
|
|
|
topic1
|
1.559
|
0.877
|
1.777
|
1098
|
0.076
|
|
|
topic2
|
1.949
|
0.877
|
2.223
|
1098
|
0.026
|
|
|
topic3
|
-0.228
|
0.877
|
-0.259
|
1098
|
0.795
|
|
|
congruence_ec:pre_understanding_c
|
0.923
|
1.465
|
0.630
|
1098
|
0.529
|
Post-task
understanding interaction
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
57.052
|
0.805
|
70.878
|
366
|
0.000
|
|
|
congruence_ec
|
9.788
|
1.013
|
9.661
|
1098
|
0.000
|
|
|
post_understanding_c
|
0.432
|
0.666
|
0.648
|
366
|
0.517
|
|
|
topic1
|
1.533
|
0.877
|
1.748
|
1098
|
0.081
|
|
|
topic2
|
1.952
|
0.877
|
2.226
|
1098
|
0.026
|
|
|
topic3
|
-0.193
|
0.878
|
-0.220
|
1098
|
0.826
|
|
|
congruence_ec:post_understanding_c
|
-0.847
|
0.838
|
-1.011
|
1098
|
0.312
|
Everyday
evidence-updating orientation interaction
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
55.392
|
1.286
|
43.077
|
366
|
0.000
|
|
|
congruence_ec
|
11.008
|
1.622
|
6.786
|
1098
|
0.000
|
|
|
everyday_evidence_ec
|
4.247
|
2.572
|
1.652
|
366
|
0.099
|
|
|
topic1
|
1.551
|
0.877
|
1.769
|
1098
|
0.077
|
|
|
topic2
|
1.950
|
0.877
|
2.224
|
1098
|
0.026
|
|
|
topic3
|
-0.211
|
0.877
|
-0.241
|
1098
|
0.810
|
|
|
congruence_ec:everyday_evidence_ec
|
-3.121
|
3.244
|
-0.962
|
1098
|
0.336
|
Prior attitude
strength interaction
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
57.103
|
0.803
|
71.114
|
365.221
|
0.000
|
|
|
congruence_ec
|
9.833
|
1.006
|
9.776
|
1097.311
|
0.000
|
|
|
prior_strength_c
|
1.393
|
0.742
|
1.878
|
1461.209
|
0.061
|
|
|
topic1
|
1.639
|
0.871
|
1.882
|
1097.823
|
0.060
|
|
|
topic2
|
1.793
|
0.876
|
2.047
|
1103.691
|
0.041
|
|
|
topic3
|
0.108
|
0.885
|
0.122
|
1113.301
|
0.903
|
|
|
congruence_ec:prior_strength_c
|
5.362
|
1.330
|
4.033
|
1271.857
|
0.000
|
Individual-difference /
task-validity analyses
Validity correlations, N = 366
|
Correlation
|
r
|
p
|
|
MR score x AOT
|
0.051
|
0.330
|
|
MR score x Intellectual humility
|
-0.082
|
0.119
|
|
MR score x CPT (no directional prediction)
|
-0.049
|
0.352
|
CPT x AOT moderation of the motivated-reasoning score
|
term
|
estimate
|
std.error
|
statistic
|
p.value
|
|
(Intercept)
|
9.750
|
1.068
|
9.125
|
0.000
|
|
CPT_c
|
-0.667
|
0.624
|
-1.069
|
0.286
|
|
AOT_c
|
0.965
|
0.880
|
1.096
|
0.274
|
|
CPT_c:AOT_c
|
-0.592
|
0.520
|
-1.140
|
0.255
|
CPT x intellectual humility moderation of the motivated-reasoning score
|
term
|
estimate
|
std.error
|
statistic
|
p.value
|
|
(Intercept)
|
9.631
|
1.064
|
9.048
|
0.000
|
|
CPT_c
|
-0.600
|
0.622
|
-0.965
|
0.335
|
|
humility_c
|
-2.929
|
1.848
|
-1.585
|
0.114
|
|
CPT_c:humility_c
|
1.408
|
1.156
|
1.218
|
0.224
|
Supplementary analysis
(not preregistered): topic heterogeneity in the congruence effect
This section is not part of the preregistration. It was conducted
post hoc to examine two questions about the confirmatory model’s
topic terms: (a) how much does including topic
actually change the congruence estimate, and (b) does the pooled
congruence effect apply equally to all four political topics, or does it
mask meaningful topic-to-topic variation?
Congruence-only model
(no topic term)
Congruence-only model (topic omitted)
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
57.052
|
0.805
|
70.838
|
366
|
0
|
|
|
congruence_ec
|
9.669
|
1.020
|
9.479
|
1098
|
0
|
This congruence estimate (9.67) is close to the confirmatory model’s
estimate with topic included (9.79, see above), a difference of 0.12
points, indicating that adjusting for topic barely moves the pooled
congruence effect.
Congruence x Topic
interaction model
Congruence x Topic interaction model, fixed effects
|
group
|
term
|
estimate
|
std.error
|
statistic
|
df
|
p.value
|
|
|
(Intercept)
|
57.184
|
0.800
|
71.481
|
365.479
|
0.000
|
|
|
congruence_ec
|
9.786
|
1.005
|
9.733
|
1096.838
|
0.000
|
|
|
topic1
|
1.562
|
0.871
|
1.793
|
1097.235
|
0.073
|
|
|
topic2
|
1.925
|
0.871
|
2.211
|
1097.252
|
0.027
|
|
|
topic3
|
-0.110
|
0.872
|
-0.126
|
1097.578
|
0.900
|
|
|
congruence_ec:topic1
|
6.214
|
1.958
|
3.174
|
1426.249
|
0.002
|
|
|
congruence_ec:topic2
|
4.967
|
1.958
|
2.537
|
1426.239
|
0.011
|
|
|
congruence_ec:topic3
|
-7.472
|
1.959
|
-3.814
|
1426.107
|
0.000
|
Omnibus (Type III, Satterthwaite) test of each term
|
Term
|
Sum Sq
|
Mean Sq
|
NumDF
|
DenDF
|
F value
|
Pr(>F)
|
|
congruence_ec
|
34975.664
|
34975.664
|
1
|
1096.838
|
94.731
|
0.000
|
|
topic
|
6419.045
|
2139.682
|
3
|
1097.315
|
5.795
|
0.001
|
|
congruence_ec:topic
|
8547.444
|
2849.148
|
3
|
1405.092
|
7.717
|
0.000
|
The Congruence x Topic interaction is significant, indicating the
pooled 9.79-point confirmatory estimate is a topic-averaged
effect, not a uniform one. Per-topic simple slopes make this
concrete:
Simple congruence effect (slope) within each political topic
|
topic
|
congruence_ec.trend
|
SE
|
df
|
lower.CL
|
upper.CL
|
|
vaccine
|
16.000
|
2.209
|
1386.401
|
11.667
|
20.333
|
|
climate
|
14.754
|
2.209
|
1386.401
|
10.421
|
19.086
|
|
media
|
2.315
|
2.211
|
1386.333
|
-2.023
|
6.653
|
|
adoption
|
6.076
|
2.207
|
1386.478
|
1.747
|
10.406
|
The congruence effect is large for vaccines and climate, moderate for
adoption, and near zero (and not significantly different from zero) for
media bias – despite media bias being one of the four topics driving the
significant, positive confirmatory result. This qualifies, but does not
overturn, the confirmatory conclusion: congruent messages are rated as
more truthful on average across the four political topics, but the
strength of that tendency varies by topic.
References
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting
linear mixed-effects models using lme4. Journal of Statistical
Software, 67(1), 1-48.
Borghi, O., Tappin, B. M., Smets, K., & Tsakiris, M. (2026). Mind
over bias: How is cognitive control related to politically motivated
reasoning? Cognition, 268, 106373.
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017).
lmerTest package: Tests in linear mixed effects models. Journal of
Statistical Software, 82(13), 1-26.
Thaler, M. (2024). The fake news effect: Experimentally identifying
motivated reasoning using trust in news. American Economic Journal:
Microeconomics, 16(2), 1-38.
Session info
## R version 4.6.0 (2026-04-24)
## Platform: aarch64-apple-darwin23
## Running under: macOS Tahoe 26.0
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## time zone: Europe/Istanbul
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] emmeans_2.0.3 lmerTest_3.2-1 lme4_2.0-1
## [4] Matrix_1.7-5 broom_1.0.13 broom.mixed_0.2.9.7
## [7] kableExtra_1.4.1 knitr_1.51 lubridate_1.9.5
## [10] forcats_1.0.1 stringr_1.6.0 dplyr_1.2.1
## [13] purrr_1.2.2 readr_2.2.0 tidyr_1.3.2
## [16] tibble_3.3.1 ggplot2_4.0.3 tidyverse_2.0.0
##
## loaded via a namespace (and not attached):
## [1] tidyselect_1.2.1 viridisLite_0.4.3 farver_2.1.2
## [4] S7_0.2.2 fastmap_1.2.0 digest_0.6.39
## [7] timechange_0.4.0 estimability_1.5.1 lifecycle_1.0.5
## [10] magrittr_2.0.5 compiler_4.6.0 rlang_1.2.0
## [13] sass_0.4.10 tools_4.6.0 utf8_1.2.6
## [16] yaml_2.3.12 labeling_0.4.3 bit_4.6.0
## [19] xml2_1.5.2 RColorBrewer_1.1-3 withr_3.0.2
## [22] numDeriv_2016.8-1.1 grid_4.6.0 xtable_1.8-8
## [25] future_1.70.0 globals_0.19.1 scales_1.4.0
## [28] MASS_7.3-65 cli_3.6.6 mvtnorm_1.3-7
## [31] rmarkdown_2.31 crayon_1.5.3 ragg_1.5.2
## [34] reformulas_0.4.4 generics_0.1.4 otel_0.2.0
## [37] rstudioapi_0.18.0 tzdb_0.5.0 minqa_1.2.8
## [40] cachem_1.1.0 splines_4.6.0 parallel_4.6.0
## [43] vctrs_0.7.3 boot_1.3-32 jsonlite_2.0.0
## [46] hms_1.1.4 pbkrtest_0.5.5 bit64_4.8.2
## [49] listenv_0.10.1 systemfonts_1.3.2 jquerylib_0.1.4
## [52] glue_1.8.1 parallelly_1.47.0 nloptr_2.2.1
## [55] codetools_0.2-20 stringi_1.8.7 gtable_0.3.6
## [58] furrr_0.4.0 pillar_1.11.1 htmltools_0.5.9
## [61] R6_2.6.1 textshaping_1.0.5 Rdpack_2.6.6
## [64] vroom_1.7.1 evaluate_1.0.5 lattice_0.22-9
## [67] png_0.1-9 rbibutils_2.4.1 backports_1.5.1
## [70] bslib_0.11.0 Rcpp_1.1.1-1.1 svglite_2.2.2
## [73] coda_0.19-4.1 nlme_3.1-169 xfun_0.57
## [76] pkgconfig_2.0.3
