Introduction

In this analysis, we aim to analyze and understand why respondents would to share (not share) the posts they intended to share (not share) when they were doing the survey. For each survey in the main experiment, we ask a why share and a why not share question at the end of the survey to one of the posts they wanted to share and one of the posts they did not want to share. Therefore, each person will answer two why share questions and why not share questions. Respondents do not get asked this question in the followup survey we conducted. Note that if a respondent decides to share all the posts in a particular survey, s/he will not see the why not share question. Similar situation applies for respondents who do not want to share any of the posts.

See here for an example of the Why Not Share question.

Data

Load Packages

Read Data

Data Preprocessing

We use the complete set of users who completed the main experiment for this analysis, regardless of whether they completed our followup survey or not. The total number of users in this analysis is 8684.

WordCloud

We first conduct WordCloud analysis to see if there is any interesting pattern.

We separately look at Why Share Pre, Why Not Share Pre, Why Share Post, and Why Not Share Post.
We further subset by accuracy nudge to understand if there is any potentially interesting patterns by the accuracy nudge condition.
For the Why Not Share questions, which are the more interesting questions of interest, we additionally look at how patterns different by treatment groups and by the veracity of the posts. We also look at the interaction between misleading posts and treatment courses for patterns.

Here, we should the word clouds and the top 20 most frequent word or phrases (bigram) alongside the frequencies.

Summary of Learning

Most participants mention about raising awareness for the Why Share question in both pre- and post- survey.
Compared to Why Not Share Pre, there are significantly more people mentioning “misleading”, “misinformation”, and “disinformation” in the Why Not Share Post questions, suggesting that participants are using the vocabulary we taught in the course to answer the question.
In the Why Not Share questions, many people answer fact-specific reasons. For example, about “vaccine”, “antibiotics”, “cancer”, etc.
We do not find differences of patterns between accuracy nudge conditions. In addition, we do not see significant differences of patterns among different treatment groups.

Why Share Pre

Overall

WordCloud with Word Frequency >= 100

word	freq
people	1869
awareness	1173
true	905
can	773
health	747
create	636
create awareness	589
know	531
information	485
help	462
mental	406
get	373
will	369
share	339
aware	329
much	319
mental health	318
many	309
make	306
post	301

Accuracy Inter

WordCloud with Word Frequency >= 100

word	freq
people	909
awareness	559
true	454
can	359
health	328
create	317
create awareness	294
information	255
know	244
help	219
mental	203
get	182
share	166
aware	160
will	159
mental health	156
make	148
much	145
post	143
many	142

Accuracy After

WordCloud with Word Frequency >= 100

word	freq
people	960
awareness	614
true	451
health	419
can	414
create	319
create awareness	295
know	287
help	243
information	230
will	210
mental	203
get	191
much	174
share	173
aware	169
many	167
sleep	166
mental health	162
make	158

Why Share Post

Overall

WordCloud with Word Frequency >= 100

word	freq
people	1333
true	1095
awareness	1026
health	734
can	681
information	645
create	585
create awareness	541
know	400
mental	383
much	357
help	353
mental health	311
share	298
will	294
get	273
post	263
truth	246
good	239
aware	235

Accuracy Inter

WordCloud with Word Frequency >= 100

word	freq
people	684
true	514
awareness	493
health	345
can	333
information	331
create	277
create awareness	261
know	200
mental	190
accurate	189
much	186
help	176
mental health	156
share	146
get	135
truth	132
will	132
post	128
many	115

Accuracy After

WordCloud with Word Frequency >= 100

word	freq
people	649
true	581
awareness	533
health	389
can	348
information	314
create	308
create awareness	280
know	200
mental	193
help	177
much	171
will	162
mental health	155
share	152
get	138
good	137
post	135
aware	128
cause	122

Why Not Share Pre

Overall

WordCloud with Word Frequency >= 50

word	freq
true	856
information	844
dont	749
sure	741
people	699
post	462
can	386
share	386
accurate	346
misleading	341
false	334
will	333
think	313
vaccine	278
vaccines	255
know	253
health	228
cause	226
believe	210
antibiotics	201

Baseline Posts

WordCloud with Word Frequency >= 50

word	freq
sure	312
information	261
true	248
dont	228
people	147
post	124
share	123
accurate	120
can	107
false	96
think	93
will	91
know	84
misleading	81
research	66
vaccine	64
im sure	62
cause	61
statement	60
antibiotics	58

Misleading Posts

WordCloud with Word Frequency >= 50

word	freq
true	608
information	583
people	552
dont	521
sure	429
post	338
can	279
share	263
misleading	260
will	242
false	238
vaccines	238
accurate	226
think	220
vaccine	214
health	173
know	169
cause	165
covid	162
believe	154

Accuracy Inter

WordCloud with Word Frequency >= 50

word	freq
information	471
sure	466
true	451
dont	374
people	319
accurate	304
post	228
share	222
misleading	180
can	179
false	168
think	152
will	143
know	123
vaccine	111
vaccines	107
cause	103
want	102
health	101
antibiotics	100

Accuracy After

WordCloud with Word Frequency >= 50

word	freq
true	405
people	380
dont	375
information	373
sure	275
post	234
can	207
will	190
vaccine	167
false	166
share	164
misleading	161
think	161
vaccines	148
know	130
health	127
cause	123
believe	117
cancer	116
like	116

Facts Baseline

WordCloud with Word Frequency >= 25

word	freq
true	184
sure	165
information	156
dont	147
people	145
post	102
can	88
share	88
misleading	76
accurate	73
think	68
will	65
false	62
know	57
vaccine	55
antibiotics	50
cause	50
cancer	47
smoking	45
health	42

Reasoning

WordCloud with Word Frequency >= 25

word	freq
information	178
true	172
dont	147
sure	133
people	126
post	94
misleading	79
accurate	76
share	73
false	72
think	70
can	68
know	56
will	53
vaccines	47
believe	44
like	43
statement	43
vaccine	41
antibiotics	39

Combo

WordCloud with Word Frequency >= 25

word	freq
information	194
people	155
sure	155
true	151
dont	147
post	81
will	75
share	70
misleading	69
accurate	68
can	66
false	63
vaccine	63
vaccines	55
health	54
think	54
know	51
may	49
cancer	46
believe	45

Emotions

WordCloud with Word Frequency >= 25

word	freq
true	196
dont	171
sure	165
information	155
people	135
post	105
can	100
false	82
share	82
will	73
vaccine	69
misleading	65
accurate	61
think	60
vaccines	60
believe	50
cause	50
may	49
covid	48
statement	47

No-course Baseline

WordCloud with Word Frequency >= 25

word	freq
information	161
true	153
people	138
dont	137
sure	123
post	80
share	73
accurate	68
will	67
can	64
think	61
health	57
false	55
vaccines	54
misleading	52
vaccine	50
cause	45
know	43
get	38
sharing	36

Facts Baseline (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
true	118
sure	102
information	90
dont	86
people	86
post	58
can	50
share	50
misleading	42
think	41
accurate	39
will	38
antibiotics	37
false	35
vaccine	33
cancer	32
cause	32
know	32
want	29
health	28

Reasoning (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	121
dont	97
true	95
sure	90
people	84
post	63
share	51
false	50
misleading	49
can	47
think	46
accurate	39
will	38
know	37
vaccines	31
like	30
antibiotics	28
may	27
believe	25
vaccine	25

Combo (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	112
true	96
dont	89
people	89
sure	87
post	50
misleading	46
will	45
vaccine	43
can	41
share	41
false	38
accurate	36
know	35
vaccines	35
health	32
think	32
may	30
believe	27
cancer	27

Emotions (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
true	120
sure	108
dont	101
people	89
information	85
can	64
post	61
share	54
false	46
accurate	41
vaccines	40
will	40
misleading	39
vaccine	39
believe	38
think	36
cancer	31
cause	30
know	29
statement	29

No-course Baseline (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	100
true	85
people	84
sure	75
dont	74
post	55
can	44
share	43
accurate	42
health	39
misleading	39
will	37
think	36
vaccine	36
false	35
vaccines	34
cause	31
know	27
sharing	26
get	25

Why Not Share Post

Overall

WordCloud with Word Frequency >= 50

word	freq
information	1182
true	954
sure	926
misinformation	666
dont	622
people	506
misleading	503
accurate	450
post	400
false	350
share	319
can	312
vaccine	284
think	266
health	237
will	235
vaccines	233
know	215
may	192
statement	185

Baseline Posts

WordCloud with Word Frequency >= 50

word	freq
sure	425
information	414
true	340
misinformation	236
dont	226
accurate	183
misleading	150
post	124
people	119
false	117
can	107
share	106
think	99
know	88
sure information	87
im sure	84
health	70
statement	68
may	64
research	64

Misleading Posts

WordCloud with Word Frequency >= 50

word	freq
information	768
true	614
sure	501
misinformation	430
dont	396
people	387
misleading	353
post	276
accurate	267
false	233
vaccine	221
share	213
vaccines	206
can	205
will	175
health	167
think	167
cancer	133
may	128
know	127

Accuracy Inter

WordCloud with Word Frequency >= 50

word	freq
information	632
sure	520
true	480
accurate	353
dont	337
misinformation	334
people	270
misleading	241
post	204
share	181
false	169
can	151
health	134
think	132
vaccine	123
will	121
may	113
sure information	103
know	102
want	99

Accuracy After

WordCloud with Word Frequency >= 50

word	freq
information	550
true	474
sure	406
misinformation	332
dont	285
misleading	262
people	236
post	196
false	181
can	161
vaccine	161
vaccines	144
share	138
think	134
will	114
know	113
health	103
cause	102
statement	99
covid	98

Facts Baseline

WordCloud with Word Frequency >= 25

word	freq
information	218
true	213
sure	182
dont	155
misinformation	119
people	112
post	94
accurate	93
false	84
think	75
can	68
share	67
vaccines	67
misleading	59
vaccine	59
know	57
will	56
cancer	46
health	46
antibiotics	44

Reasoning

WordCloud with Word Frequency >= 25

word	freq
information	248
true	180
misleading	149
sure	144
misinformation	125
dont	114
accurate	97
people	95
post	86
can	72
false	69
share	55
vaccine	50
health	47
statement	47
think	46
know	39
source	39
will	35
cancer	34

Combo

WordCloud with Word Frequency >= 25

word	freq
information	259
misinformation	187
true	187
sure	185
misleading	146
dont	112
accurate	102
people	93
post	71
false	65
share	62
can	52
vaccine	51
will	48
health	44
think	43
may	41
im sure	37
know	36
sure information	36

Emotions

WordCloud with Word Frequency >= 25

word	freq
information	262
sure	260
true	222
misinformation	191
dont	142
people	106
post	84
misleading	82
accurate	80
false	74
can	69
share	66
vaccine	58
sure information	56
know	53
health	50
evidence	49
think	48
research	47
will	47

No-course Baseline

WordCloud with Word Frequency >= 25

word	freq
information	195
sure	155
true	152
people	100
dont	99
accurate	78
share	69
misleading	67
vaccine	66
post	65
false	58
vaccines	58
think	54
can	51
health	50
will	49
misinformation	44
cancer	39
cause	39
covid	36

Facts Baseline (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
true	140
information	139
sure	106
dont	103
people	90
misinformation	76
post	68
vaccines	61
false	54
accurate	49
share	49
think	48
vaccine	48
will	45
misleading	44
can	38
health	34
cancer	33
know	33
covid	31

Reasoning (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	163
true	111
misleading	102
sure	76
misinformation	75
people	72
dont	67
post	60
accurate	54
can	50
false	46
share	39
health	35
vaccine	34
covid	29
statement	29
vaccines	29
cancer	26
may	26
lung	25

Combo (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	173
misinformation	119
true	116
misleading	101
sure	93
dont	76
people	73
accurate	66
false	45
post	45
vaccine	38
will	36
share	35
can	34
vaccines	34
think	29
cancer	28
health	28
may	28
misleading information	28

Emotions (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	159
true	141
sure	135
misinformation	129
dont	85
people	74
misleading	56
post	52
vaccine	50
false	48
can	46
accurate	45
share	35
evidence	33
health	33
know	32
sure information	32
will	32
vaccines	31
proven	30

No-course Baseline (Misleading Posts)

WordCloud with Word Frequency >= 25

word	freq
information	134
true	106
sure	91
people	78
dont	65
share	55
accurate	53
post	51
vaccine	51
vaccines	51
misleading	50
will	41
false	40
can	37
health	37
think	36
covid	32
misinformation	31
cancer	29
cause	29

Heuristics Analysis

For heuristics analysis, we attempt with several directions.

We try out the heuristics we used for the reflective questions in the followup questions and see whether there is significant differences. (Reflective Question Heuristics)
Given that many people mention the words, “misleading”, “misinformation”, and “disinformation”, we would like to the distribution of participants mentioning it. (Misinformation/Misleading Heuristics)
Given that many people reply about fact specific (e.g. vaccine, cancer, antibiotics), we would like to the distribution of participants mentioning it. (Fact-Based Heuristics)

We only conduct this analysis on Why Not Share - Post question as we believe this is the most important question for us to understand whether we can dig out any underlying mechanism.

Summary of Learning

For (Reflective Question Heuristics), we don’t find any significant differences among treatment courses, suggesting the reflective question heuristics might not the best for evaluation of this question.
For (Misinformation/Misleading Heuristics), we find that Treatment participants mention the misinformation/misleading keywords significantly more than No-course Baseline and Facts Baseline. No-course Baseline group’s behavior is expected as the group does not receive any information so should not know to use misinformation as part of their answer.
For (Fact-Based Heuristics), we do see that No-course Baseline and Facts Baseline mention facts specific responses significantly more than the treatment courses group, which is quite interesting.

Reflective Question Heuristics

treatment	percentage_mentioned	count_mentioned	total_in_group
combo	0.0908046	158	1740
control	0.0964373	157	1628
emotion	0.1236383	227	1836
reminder	0.1111732	199	1790
tactics	0.0905325	153	1690

Conduct Hypothesis Test

All treatment courses aggregated vs No-course Baseline
All treatment courses aggregated vs Facts Baseline
Emotions vs No-course Baseline
Reasoning vs No-course Baseline
Combo vs No-course Baseline
Reminder vs No-course Baseline

	estimates	std.err	CI_lw	CI_up	ts	p_val	p_val_holm
Test 1 - All Treatment Courses v. No-course Baseline	0.0057275	0.0084249	-0.0081349	Inf	0.6798278	0.2483352	0.9933408
Test 2 - All Treatment Courses v. Facts Baseline	-0.0090084	0.0085239	-0.0230332	Inf	-1.0568402	0.8546651	1.0000000
Test 3 - Emotions v. No-course Baseline	0.0272010	0.0106115	0.0097419	Inf	2.5633464	0.0052044	0.0312262
Test 4 - Reasoning v. No-course Baseline	-0.0059048	0.0101146	-0.0225465	Inf	-0.5837887	0.7202988	1.0000000
Test 5 - Combo v. No-course Baseline	-0.0056327	0.0100515	-0.0221705	Inf	-0.5603906	0.7123747	1.0000000
Test 6 - Facts Baseline v. No-course Baseline	0.0147358	0.0104303	-0.0024251	Inf	1.4127923	0.0789040	0.3945200

Responses containing heuristics

Responses not containing heuristics

Misinformation/Misleading/Disinformation Heuristics

treatment	percentage_mentioned	count_mentioned	total_in_group
combo	0.2183908	380	1740
control	0.0743243	121	1628
emotion	0.1759259	323	1836
reminder	0.1173184	210	1790
tactics	0.1828402	309	1690

Conduct Hypothesis Test

All treatment courses aggregated vs No-course Baseline
All treatment courses aggregated vs Facts Baseline
Emotions vs No-course Baseline
Reasoning vs No-course Baseline
Combo vs No-course Baseline
Reminder vs No-course Baseline

	estimates	std.err	CI_lw	CI_up	ts	p_val	p_val_holm
Test 1 - All Treatment Courses v. No-course Baseline	0.1178519	0.0084719	0.1039137	Inf	13.910956	0e+00	0e+00
Test 2 - All Treatment Courses v. Facts Baseline	0.0748578	0.0093472	0.0594792	Inf	8.008570	0e+00	0e+00
Test 3 - Emotions v. No-course Baseline	0.1016016	0.0110133	0.0834812	Inf	9.225364	0e+00	0e+00
Test 4 - Reasoning v. No-course Baseline	0.1085159	0.0114345	0.0897020	Inf	9.490248	0e+00	0e+00
Test 5 - Combo v. No-course Baseline	0.1440665	0.0118509	0.1245674	Inf	12.156567	0e+00	0e+00
Test 6 - Facts Baseline v. No-course Baseline	0.0429941	0.0100085	0.0265270	Inf	4.295745	9e-06	9e-06

Responses containing heuristics

Responses not containing heuristics

Fact-Based Heuristics

treatment	percentage_mentioned	count_mentioned	total_in_group
combo	0.1683908	293	1740
control	0.2106880	343	1628
emotion	0.1873638	344	1836
reminder	0.2162011	387	1790
tactics	0.1751479	296	1690

Conduct Hypothesis Test

All treatment courses aggregated vs No-course Baseline
All treatment courses aggregated vs Facts Baseline
Emotions vs No-course Baseline
Reasoning vs No-course Baseline
Combo vs No-course Baseline
Reminder vs No-course Baseline

	estimates	std.err	CI_lw	CI_up	ts	p_val	p_val_holm
Test 1 - All Treatment Courses v. No-course Baseline	-0.0335136	0.0113974	-Inf	-0.0147598	-2.9404606	0.0016533	0.0066130
Test 2 - All Treatment Courses v. Facts Baseline	-0.0390268	0.0110640	-Inf	-0.0208224	-3.5273760	0.0002131	0.0012786
Test 3 - Emotions v. No-course Baseline	-0.0233241	0.0136083	-Inf	-0.0009343	-1.7139602	0.0433140	0.0866280
Test 4 - Reasoning v. No-course Baseline	-0.0355400	0.0137021	-Inf	-0.0129957	-2.5937604	0.0047678	0.0143033
Test 5 - Combo v. No-course Baseline	-0.0422972	0.0135181	-Inf	-0.0200557	-3.1289353	0.0008849	0.0044244
Test 6 - Facts Baseline v. No-course Baseline	0.0055132	0.0140333	-Inf	0.0286022	0.3928619	0.6527769	0.6527769

Responses containing heuristics

Responses not containing heuristics

Comparing Gender

WordCloud

Emotions

WordCloud with Word Frequency >= 25

Man

word	freq
information	164
sure	155
true	144
misinformation	120
dont	90
people	62
post	60
misleading	51
accurate	47
false	44
can	40
know	40
vaccine	38
may	34
share	34
sure information	34
evidence	33
research	33
health	28
might	28

Woman

word	freq
sure	104
information	98
true	78
misinformation	70
dont	52
people	44
accurate	33
share	32
misleading	31
false	30
can	29
post	24
health	22
sure information	22
want	22
think	21
will	21
vaccine	20
cant	19
im sure	19

Reasoning

WordCloud with Word Frequency >= 25

Man

word	freq
information	168
true	116
misleading	99
sure	86
misinformation	76
dont	71
accurate	59
people	54
post	50
can	46
false	46
statement	37
share	33
source	32
think	31
health	30
vaccine	29
may	26
cancer	25
will	25

Woman

word	freq
information	79
true	64
sure	58
misleading	50
misinformation	49
dont	43
people	41
accurate	38
post	36
can	26
false	23
share	22
vaccine	21
know	18
health	17
want	17
antibiotics	15
covid	15
think	15
sure information	13

Combo

WordCloud with Word Frequency >= 25

Man

word	freq
information	175
true	114
sure	110
misinformation	109
misleading	101
accurate	74
dont	68
people	54
post	49
false	42
share	36
can	34
vaccine	34
will	31
health	27
may	25
misleading information	25
im sure	23
sure information	23
think	23

Woman

word	freq
information	84
misinformation	78
sure	75
true	73
misleading	45
dont	44
people	39
accurate	28
share	26
false	23
post	21
know	20
think	20
can	18
health	17
will	17
may	16
sure information	16
vaccine	16
im sure	14

Contain Information Keyword Heuristics

## # A tibble: 6 × 5
## # Groups:   treatment [3]
##   treatment gender percentage_mentioned count_mentioned total_in_group
##   <chr>     <chr>                 <dbl>           <int>          <int>
## 1 combo     Man                   0.432             496           1147
## 2 combo     Woman                 0.469             276            589
## 3 emotion   Man                   0.401             474           1182
## 4 emotion   Woman                 0.423             275            650
## 5 tactics   Man                   0.409             445           1087
## 6 tactics   Woman                 0.404             242            599

##                                      estimates    std.err CI_lw       CI_up
## Test 1 - Man vs Woman (Emotion)   -0.022061695 0.02407238  -Inf 0.017561494
## Test 2 - Man vs Woman (Reasoning)  0.005376947 0.02500585  -Inf 0.046538801
## Test 3 - Man vs Woman (Combo)     -0.036158399 0.02525190  -Inf 0.005409962
##                                           ts      p_val p_val_holm
## Test 1 - Man vs Woman (Emotion)   -0.9164735 0.17979254  0.3595851
## Test 2 - Man vs Woman (Reasoning)  0.2150276 0.58510933  0.5851093
## Test 3 - Man vs Woman (Combo)     -1.4319079 0.07621768  0.2286530

WhyShare Analysis IAM Project

January 30, 2023

Introduction

Data

WordCloud

Summary of Learning

Heuristics Analysis

Summary of Learning

Reflective Question Heuristics

Responses containing heuristics

Responses not containing heuristics

Misinformation/Misleading/Disinformation Heuristics

Responses containing heuristics

Responses not containing heuristics

Fact-Based Heuristics

Responses containing heuristics

Responses not containing heuristics

Comparing Gender

WordCloud

Emotions

Reasoning

Combo

Contain Information Keyword Heuristics