Text Analysis

Summary

• The purpose of this script is to analyze the text data from the follow-up survey. In the follow-up, we asked participants three free questions, which are as follows:

1. Can you remember any example in the last two weeks where thinking about our previous chat changed your behavior? followup_reflection_1
2. Have you noticed any misinformation/disinformation in your timeline in the past 6 weeks? followup_reflection_2
3. If you noticed a close friend sharing posts that you thought might have misinformation, what might you say to them that would help them make different choices in the future? followup_reflection_3

• During the cleaning process, we noted that these text responses were not clean, and sometimes contained links, emojis, and curse words. We created a series of binary variable to indicate the presence of a link, and in this script, we will do additional cleaning before running our text analysis.

• For responses that contains a link, we have variable abnormal_followup_reflection_1, abnormal_followup_reflection_2, and abnormal_followup_reflection_3 to indicate. We will remove these links before running our text analysis.

• We will use the data generated from the previous script, misinfo_followup_clean_wide, which is stored in ~fb_misinfo_interventions/data/processed/.

• Our main goal is to determine how much people remembered from our intervention, especially the emotional course.

Load Data

• The current dataset is stored at ./data/chatfuel/processed/. The current working directory is ~fb_misinfo_interventions/code.

• In this dataset, there are 19950 participants who started the follow-up and passed a series of high-quality data checks from the follow-up cleaning script.

Variable dictionary

library(Hmisc)
dictionary <- label(data) %>% data.frame()
dictionary <- dictionary %>% 
  mutate(variable_names = rownames(dictionary)) %>%
  select(variable_names, '.')

rownames(dictionary) <- NULL
dictionary %>% arrange(dictionary[,1]) %>% kable(digits = 3, col.names = c("Variable Name", "Description")) |>
      kable_styling(bootstrap_options = c("striped", "hover")) |>
      kableExtra::scroll_box( height = "500px")

Variable Name	Description
MisinfoChat_start_time_followup	Follow up Start Time
MisinfoQuiz_start_time_followup	Follow up Quiz Start Time
abnormal_followup_reflection_1	Follow up Abnormal Reflection 1
abnormal_followup_reflection_2	Follow up Abnormal Reflection 2
abnormal_followup_reflection_3	Follow up Abnormal Reflection 3
analytic_id	Analytic ID
attention_check_passed_followup_coded_num
completed_quiz_followup_coded_num	Completed Followup Misinformation Quiz as Binary Variable
duration_complete_quiz_followup	Follow up Quiz Duration
followup_entry_point	Follow up Entry Point
followup_reflection_1	Follow up Reflection 1
followup_reflection_2	Follow up Reflection 2
followup_reflection_3	Follow up Reflection 3
manipulation_discernment_followup	Follow up Manipulation Discernment
mean_manipulative_misinfo_followup	Follow up Mean Misinfo Manipulative Score
mean_manipulative_nonmisinfo_followup	Follow up Mean Nonmisinfo Manipulation Score
mean_reliable_misinfo_followup	Follow up Mean Misinfo Reliable Score
mean_reliable_nonmisinfo_followup	Follow up Mean Nonmisinfo Reliability Score
mean_share_misinfo_followup	Follow up Mean Misinfo Share Score
mean_share_nonmisinfo_followup	Follow up Mean Nonmisinfo Sharing Score
misinfoQuiz_end_time_followup	Follow up Quiz End Time
quiz_completed_followup	Follow up Quiz Completion
reliability_discernment_followup	Follow up Reliability Discernment
sharing_discernment_followup	Follow up Sharing Discernment
time_since_completed_intervention	Follow up Time Since Completed Intervention in Main Survey
time_since_completed_quiz	Follow up Time Since Completed Misinformation Quiz in Main Survey
user_payment_amount_followup	Follow up Payment Amount

Data Cleaning and Analysis

• We will clean the text data before running our text analysis. We will remove emojis and links.

Reflection Question 1

• The table below will show randomly 10 raw non-missing responses to the question “Can you remember any example in the last two weeks where thinking about our previous chat changed your behavior?”

data %>% 
  select(followup_reflection_1) %>% filter(!is.na(followup_reflection_1)) %>%
  sample_n(10) %>% 
  kable(digits = 3, col.names = c("Reflection 1")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "500px")

Reflection 1
Can’t i
Yes
No
Yes
Alot
Yes
Not at all
I pay attention to what I read on social media
Yes
I do remember the misleading info

reflection_1 <- data %>% select(analytic_id, followup_reflection_1, abnormal_followup_reflection_1) %>% filter(followup_reflection_1 != "") %>% filter(abnormal_followup_reflection_1 == 0)

n_reflection_1 <- as.character(nrow(reflection_1))


reflection_1 %>% write_csv(here::here("data", "chatfuel", "processed", "reflection_1_clean.csv"))

After filtering out the missing responses and responses only containing a link, we have 18243 responses to the question “Can you remember any example in the last two weeks where thinking about our previous chat changed your behavior?”

reflection_1 <- reflection_1 %>%
  mutate(
    word_count = str_count(followup_reflection_1, boundary("word")),
    char_count = str_length(followup_reflection_1)
  )

# merging with the main data set to get treatment intervention and time block
reflection_1 <- reflection_1 %>% left_join(main %>% select(analytic_id, arm_coded, time_since_first_start), by = "analytic_id")

The mean, standard deviation, and total words count and character count for Question 1

reflection_1 %>% select(word_count, char_count) %>% summary() %>% kable(digits = 3, col.names = c("Word Count", "Character Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "500px")

	Word Count	Character Count
	Min. : 0.00	Min. : 1.0
	1st Qu.: 1.00	1st Qu.: 3.0
	Median : 1.00	Median : 3.0
	Mean : 3.91	Mean : 19.5
	3rd Qu.: 3.00	3rd Qu.: 10.0
	Max. :133.00	Max. :799.0

By treatment arm

reflection_1 %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    mean_word_count = mean(word_count),
    sd_word_count = sd(word_count),
    se_word_count = sd(word_count)/sqrt(n),
    mean_char_count = mean(char_count),
    sd_char_count = sd(char_count),
    se_char_count = sd(char_count)/sqrt(n)
  ) %>% kable(digits = 3, col.names = c("Arm", "N", "Mean Word Count", "SD Word Count", "SE Word Count", "Mean Char Count", "SD Char Count", "SE Char Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Mean Word Count	SD Word Count	SE Word Count	Mean Char Count	SD Char Count	SE Char Count
Game	1992	3.166	5.729	0.128	15.076	32.081	0.719
Long Baseline	3012	3.836	7.196	0.131	19.195	40.757	0.743
Original Baseline	4133	4.000	6.933	0.108	19.893	38.835	0.604
SMS	6814	4.117	8.027	0.097	20.854	45.611	0.553
Video	2292	3.878	7.026	0.147	19.027	38.866	0.812

By time block

• The table below will show the mean, standard deviation, and total words count and character count for Question 1 by time block in days since the first start of the experiment.

reflection_1 %>% 
  group_by(time_since_first_start) %>% 
  summarise(
    n= n(),
    mean_word_count = mean(word_count),
    sd_word_count = sd(word_count),
    se_word_count = sd(word_count)/sqrt(n),
    mean_char_count = mean(char_count),
    sd_char_count = sd(char_count),
    se_char_count = sd(char_count)/sqrt(n)
  ) %>% kable(digits = 3, col.names = c("Day since first start", "N", "Mean Word Count", "SD Word Count", "SE Word Count", "Mean Char Count", "SD Char Count", "SE Char Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Day since first start	N	Mean Word Count	SD Word Count	SE Word Count	Mean Char Count	SD Char Count	SE Char Count
0	534	4.006	7.304	0.316	19.985	42.044	1.819
1	748	4.124	7.270	0.266	20.414	40.015	1.463
2	1140	3.901	6.530	0.193	19.326	36.605	1.084
3	1019	3.741	6.631	0.208	18.293	36.988	1.159
4	560	3.859	7.179	0.303	19.036	40.469	1.710
5	538	3.840	6.638	0.286	19.028	36.882	1.590
6	644	3.683	7.083	0.279	17.961	39.120	1.542
7	689	4.271	8.311	0.317	21.350	47.452	1.808
8	504	3.579	6.461	0.288	17.736	37.111	1.653
9	529	3.560	6.449	0.280	17.682	36.130	1.571
10	539	3.436	5.916	0.255	16.850	33.008	1.422
11	401	3.825	6.571	0.328	18.933	36.566	1.826
12	386	4.249	9.298	0.473	21.137	51.628	2.628
13	201	3.761	6.516	0.460	19.229	37.692	2.659
14	258	3.516	7.725	0.481	17.136	42.724	2.660
15	264	3.678	6.860	0.422	18.083	38.239	2.353
16	245	3.310	6.555	0.419	15.882	37.725	2.410
17	222	3.802	7.327	0.492	18.986	42.698	2.866
18	11	1.455	1.508	0.455	5.455	8.490	2.560
19	9	5.222	7.694	2.565	28.000	47.539	15.846
25	295	4.922	9.466	0.551	25.661	55.720	3.244
26	1742	4.001	7.433	0.178	19.986	42.323	1.014
27	991	4.079	7.791	0.247	20.550	43.729	1.389
28	757	3.374	7.153	0.260	16.629	40.832	1.484
29	668	4.009	7.779	0.301	20.150	43.439	1.681
30	555	3.701	6.713	0.285	18.827	38.962	1.654
31	467	3.867	7.182	0.332	19.704	41.040	1.899
32	492	4.268	8.011	0.361	21.268	44.301	1.997
33	432	3.743	7.684	0.370	18.343	42.573	2.048
34	549	4.375	7.123	0.304	22.373	40.221	1.717
35	419	4.907	9.727	0.475	25.317	54.509	2.663
36	149	2.859	4.801	0.393	13.550	26.917	2.205
37	141	3.461	5.580	0.470	16.823	30.853	2.598
38	189	3.698	6.291	0.458	18.519	35.942	2.614
39	227	3.916	6.973	0.463	19.317	39.081	2.594
40	123	4.081	6.748	0.608	20.423	37.397	3.372
41	133	3.481	7.519	0.652	17.338	41.529	3.601
42	80	3.850	6.002	0.671	19.800	36.025	4.028
43	249	3.980	7.772	0.493	20.237	44.361	2.811
44	141	5.014	9.655	0.813	26.504	55.381	4.664
45	3	13.000	20.785	12.000	66.667	110.274	63.667

Bar graph of word count and character count time block

Word Count

reflection_1 %>% 
  ggplot(aes(x = time_since_first_start, y = word_count)) +
  geom_bar(stat = "summary", fun = "mean", fill = "skyblue", color = "black") +
  labs(title = "Mean Word Count by Time Block", x = "Time Block (days since first start)", y = "Mean Word Count") +
  theme_minimal()

Character Count

reflection_1 %>% 
  ggplot(aes(x = time_since_first_start, y = char_count)) +
  geom_bar(stat = "summary", fun = "mean", fill = "skyblue", color = "black") +
  labs(title = "Mean Character Count by Time Block", x = "Time Block (days since first start)", y = "Mean Character Count") +
  theme_minimal()

Reflection Question 2

• The table below will show randomly 10 raw non-missing responses to the question “Have you noticed any misinformation/disinformation in your timeline in the past 6 weeks?”

data %>% 
  select(followup_reflection_2) %>% filter(!is.na(followup_reflection_2)) %>%
  sample_n(10) %>% 
  kable(digits = 3, col.names = c("Reflection 2")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "500px")

Reflection 2
Not at all
No
Yes
Yes
Yes
Yes
No
Yes
No
Yes

reflection_2 <- data %>% select(analytic_id, followup_reflection_2, abnormal_followup_reflection_2) %>% filter(followup_reflection_2 != "") %>% filter(abnormal_followup_reflection_2 == 0)

n_reflection_2 <- as.character(nrow(reflection_2))



reflection_2 %>% write_csv(here::here("data", "chatfuel", "processed", "reflection_2_clean.csv"))

After filtering out the missing responses and responses only containing a link, we have 18290 responses to the question “Have you noticed any misinformation/disinformation in your timeline in the past 6 weeks?”

reflection_2 <- reflection_2 %>%
  mutate(
    word_count = str_count(followup_reflection_2, boundary("word")),
    char_count = str_length(followup_reflection_2)
  )

# merging with the main data set to get treatment intervention and time block
reflection_2 <- reflection_2 %>% left_join(main %>% select(analytic_id, arm_coded, time_since_first_start), by = "analytic_id")

The mean, standard deviation, and total words count and character count for Question 2

reflection_2 %>% select(word_count, char_count) %>% summary() %>% kable(digits = 3, col.names = c("Word Count", "Character Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "500px")

	Word Count	Character Count
	Min. : 0.000	Min. : 1.000
	1st Qu.: 1.000	1st Qu.: 2.000
	Median : 1.000	Median : 3.000
	Mean : 1.481	Mean : 5.307
	3rd Qu.: 1.000	3rd Qu.: 3.000
	Max. :54.000	Max. :294.000

By treatment arm

reflection_2 %>% 
  group_by(arm_coded) %>% 
  summarise(
    n= n(),
    mean_word_count = mean(word_count),
    sd_word_count = sd(word_count),
    se_word_count = sd(word_count)/sqrt(n),
    mean_char_count = mean(char_count),
    sd_char_count = sd(char_count),
    se_char_count = sd(char_count)/sqrt(n)
  ) %>% kable(digits = 3, col.names = c("Arm", "N", "Mean Word Count", "SD Word Count", "SE Word Count", "Mean Char Count", "SD Char Count", "SE Char Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Mean Word Count	SD Word Count	SE Word Count	Mean Char Count	SD Char Count	SE Char Count
Game	2007	1.401	1.705	0.038	4.752	9.436	0.211
Long Baseline	3019	1.559	2.473	0.045	5.780	13.825	0.252
Original Baseline	4139	1.499	2.212	0.034	5.381	12.424	0.193
SMS	6818	1.483	2.273	0.028	5.390	12.423	0.150
Video	2307	1.407	1.990	0.041	4.791	11.285	0.235

By time block

• The table below will show the mean, standard deviation, and total words count and character count for Question 2 by time block in days since the first start of the experiment.

reflection_2 %>% 
  group_by(time_since_first_start) %>% 
  summarise(
    n = n(),
    mean_word_count = mean(word_count),
    sd_word_count = sd(word_count),
    se_word_count = sd(word_count)/sqrt(n),
    mean_char_count = mean(char_count),
    sd_char_count = sd(char_count),
    se_char_count = sd(char_count)/sqrt(n)
  ) %>% kable(digits = 3, col.names = c("Day since first start", "N", "Mean Word Count", "SD Word Count", "SE Word Count",  "Mean Char Count", "SD Char Count", "SE Char Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Day since first start	N	Mean Word Count	SD Word Count	SE Word Count	Mean Char Count	SD Char Count	SE Char Count
0	533	1.415	1.954	0.085	4.992	11.230	0.486
1	749	1.543	2.327	0.085	5.561	12.303	0.450
2	1137	1.499	2.204	0.065	5.353	12.224	0.363
3	1027	1.374	1.568	0.049	4.682	8.473	0.264
4	560	1.388	1.775	0.075	4.737	9.823	0.415
5	536	1.552	2.262	0.098	5.685	13.208	0.571
6	646	1.450	2.076	0.082	5.099	11.647	0.458
7	694	1.631	2.695	0.102	6.084	15.512	0.589
8	507	1.653	3.173	0.141	6.229	18.367	0.816
9	532	1.383	1.482	0.064	4.654	8.231	0.357
10	541	1.240	0.797	0.034	3.887	4.306	0.185
11	405	1.467	2.192	0.109	5.205	12.289	0.611
12	388	1.418	1.735	0.088	4.892	9.935	0.504
13	204	1.358	1.370	0.096	4.490	8.509	0.596
14	257	1.467	2.163	0.135	5.385	13.221	0.825
15	267	1.431	2.631	0.161	4.948	12.505	0.765
16	247	1.211	0.794	0.051	3.676	4.000	0.255
17	221	1.312	1.115	0.075	4.407	6.631	0.446
18	11	2.182	3.601	1.086	8.909	18.743	5.651
19	9	1.111	0.333	0.111	3.333	1.414	0.471
25	296	1.429	1.645	0.096	5.203	9.495	0.552
26	1740	1.524	2.506	0.060	5.575	13.750	0.330
27	998	1.414	1.913	0.061	4.990	10.417	0.330
28	758	1.464	1.879	0.068	5.152	10.353	0.376
29	668	1.530	2.237	0.087	5.647	12.788	0.495
30	555	1.569	3.293	0.140	5.850	18.124	0.769
31	469	1.537	2.285	0.106	5.648	13.482	0.623
32	491	1.542	1.883	0.085	5.823	10.496	0.474
33	437	1.357	1.672	0.080	4.668	9.374	0.448
34	549	1.610	2.822	0.120	5.969	13.828	0.590
35	421	1.463	2.311	0.113	5.449	13.765	0.671
36	153	1.379	1.303	0.105	4.784	7.189	0.581
37	142	1.704	2.825	0.237	6.345	14.349	1.204
38	187	1.513	1.733	0.127	5.406	9.066	0.663
39	227	1.511	2.083	0.138	5.568	10.733	0.712
40	123	2.561	5.817	0.525	11.813	33.072	2.982
41	132	1.780	2.982	0.260	6.659	16.200	1.410
42	80	1.138	0.522	0.058	3.612	3.366	0.376
43	250	1.452	1.542	0.098	5.272	9.047	0.572
44	140	1.286	1.183	0.100	4.250	6.134	0.518
45	3	3.000	3.464	2.000	15.333	21.362	12.333

Bar graph of word count and character count time block

Word Count

reflection_2 %>% 
  ggplot(aes(x = time_since_first_start, y = word_count)) +
  geom_bar(stat = "summary", fun = "mean", fill = "yellow", color = "black") +
  labs(title = "Mean Word Count by Time Block", x = "Time Block (days since first start)", y = "Mean Word Count") +
  theme_minimal()

Character Count

reflection_2 %>% 
  ggplot(aes(x = time_since_first_start, y = char_count)) +
  geom_bar(stat = "summary", fun = "mean", fill = "yellow", color = "black") +
  labs(title = "Mean Character Count by Time Block", x = "Time Block (days since first start)", y = "Mean Character Count") +
  theme_minimal()

Reflection Question 3

• The table below will show randomly 10 raw non-missing responses to the question “If you noticed a close friend sharing posts that you thought might have misinformation, what might you say to them that would help them make different choices in the future?”

data %>% 
  select(followup_reflection_3) %>% filter(!is.na(followup_reflection_3)) %>%
  sample_n(10) %>% 
  kable(digits = 3, col.names = c("Reflection 3")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "500px")

Reflection 3
I will tell them to stop
I will kindly advise them not to post such posts and enlighten them about misinformation
Will tell them never to post any thing that has misinformation any more it might coz harm to others
Stay away from such things
To be very keen on the sources they rely on to get information, that the information they consume is verified and validated and not to spread ear says so as to reduce the spread of misinformation.
inspire them to make better choices
I will encourage them always to analyse the situation unlike giving answers that are not true.
Not to share things things they aren’t Sure about
Stop spreading things that you yourself are not sure about
They should not post whats not true

reflection_3 <- data %>% select(analytic_id, followup_reflection_3, abnormal_followup_reflection_3) %>% filter(followup_reflection_3 != "") %>% filter(abnormal_followup_reflection_3 == 0)

n_reflection_3 <- as.character(nrow(reflection_3))

reflection_3 %>% write_csv(here::here("data", "chatfuel", "processed", "reflection_3_clean.csv"))

After filtering out the missing responses and responses only containing a link, we have 18212 responses to the question “If you noticed a close friend sharing posts that you thought might have misinformation, what might you say to them that would help them make different choices in the future?”

reflection_3 <- reflection_3 %>%
  mutate(
    word_count = str_count(followup_reflection_3, boundary("word")),
    char_count = str_length(followup_reflection_3)
  )

# merging with the main data set to get treatment intervention and time block
reflection_3 <- reflection_3 %>% left_join(main %>% select(analytic_id, arm_coded, time_since_first_start, phase_coded), by = "analytic_id")

The mean, standard deviation, and total words count and character count for Question 3

reflection_3 %>% select(word_count, char_count) %>% summary() %>% kable(digits = 3, col.names = c("Word Count", "Character Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "500px")

	Word Count	Character Count
	Min. : 0.0	Min. : 1.00
	1st Qu.: 5.0	1st Qu.: 30.00
	Median : 10.0	Median : 54.00
	Mean : 11.1	Mean : 61.21
	3rd Qu.: 15.0	3rd Qu.: 82.00
	Max. :137.0	Max. :807.00

By treatment arm

reflection_3 %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    mean_word_count = mean(word_count),
    sd_word_count = sd(word_count),
    se_word_count = sd(word_count)/sqrt(n),
    mean_char_count = mean(char_count),
    sd_char_count = sd(char_count), 
    se_char_count = sd(char_count)/sqrt(n)

  ) %>% kable(digits = 3, col.names = c("Arm", "N", "Mean Word Count", "SD Word Count", "SE Word Count", "Mean Char Count", "SD Char Count", "SE Char Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Mean Word Count	SD Word Count	SE Word Count	Mean Char Count	SD Char Count	SE Char Count
Game	1992	9.767	7.498	0.168	53.345	42.050	0.942
Long Baseline	3008	11.416	9.574	0.175	63.207	55.052	1.004
Original Baseline	4119	11.220	8.163	0.127	61.723	45.926	0.716
SMS	6801	11.402	8.447	0.102	63.069	47.945	0.581
Video	2292	10.743	8.545	0.178	58.986	48.475	1.013

By time block

• The table below will show the mean, standard deviation, and total words count and character count for Question 3 by time block in days since the first start of the experiment.

reflection_3 %>% 
  group_by(time_since_first_start) %>% 
  summarise(
    n = n(),
    mean_word_count = mean(word_count),
    sd_word_count = sd(word_count),
    se_word_count = sd(word_count)/sqrt(n),
    mean_char_count = mean(char_count),
    sd_char_count = sd(char_count),
    se_char_count = sd(char_count)/sqrt(n)
    
  ) %>% kable(digits = 3, col.names = c("Day since first start", "N", "Mean Word Count", "SD Word Count", "SE Word Count", "Mean Char Count", "SD Char Count", "SE Char Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Day since first start	N	Mean Word Count	SD Word Count	SE Word Count	Mean Char Count	SD Char Count	SE Char Count
0	533	10.852	7.446	0.323	59.902	41.131	1.782
1	750	11.521	7.712	0.282	63.663	43.051	1.572
2	1131	10.880	7.322	0.218	59.421	41.446	1.232
3	1023	10.707	7.979	0.249	58.852	45.058	1.409
4	561	10.781	8.006	0.338	58.943	45.257	1.911
5	534	10.251	6.994	0.303	55.867	39.199	1.696
6	642	10.611	7.637	0.301	58.542	42.642	1.683
7	694	11.193	8.274	0.314	61.582	48.163	1.828
8	506	10.935	8.447	0.376	60.814	48.944	2.176
9	526	10.367	7.341	0.320	56.973	42.426	1.850
10	537	10.939	10.108	0.436	60.369	57.770	2.493
11	401	10.636	7.437	0.371	58.401	42.116	2.103
12	386	11.459	9.851	0.501	62.083	55.516	2.826
13	203	11.680	8.482	0.595	64.202	46.370	3.255
14	253	9.783	6.803	0.428	53.194	38.894	2.445
15	263	9.928	7.193	0.444	54.627	40.400	2.491
16	243	9.329	7.690	0.493	50.905	41.346	2.652
17	219	10.420	8.310	0.562	58.183	48.338	3.266
18	11	13.455	14.535	4.383	69.091	71.605	21.590
19	9	13.000	6.928	2.309	71.778	39.414	13.138
25	296	11.841	10.312	0.599	65.003	58.970	3.428
26	1734	11.375	8.012	0.192	62.915	45.253	1.087
27	992	11.324	8.383	0.266	62.655	47.060	1.494
28	758	11.520	9.772	0.355	63.532	56.066	2.036
29	667	11.822	11.380	0.441	65.573	66.458	2.573
30	554	11.841	9.561	0.406	65.514	53.550	2.275
31	469	11.644	9.382	0.433	65.049	53.930	2.490
32	491	11.566	8.578	0.387	63.770	49.143	2.218
33	435	11.968	10.505	0.504	66.097	58.824	2.820
34	548	11.819	7.975	0.341	65.677	45.306	1.935
35	417	11.403	10.714	0.525	62.523	61.329	3.003
36	150	10.087	6.695	0.547	55.800	36.745	3.000
37	141	11.277	8.226	0.693	63.191	46.252	3.895
38	184	10.837	7.470	0.551	59.783	43.441	3.203
39	224	10.768	7.610	0.508	59.741	43.106	2.880
40	122	9.656	6.335	0.574	53.664	36.365	3.292
41	133	11.361	6.573	0.570	63.519	38.139	3.307
42	79	9.823	6.201	0.698	54.658	35.130	3.952
43	250	11.396	9.899	0.626	62.536	55.393	3.503
44	140	10.786	8.531	0.721	60.593	46.664	3.944
45	3	8.333	11.846	6.839	42.667	65.271	37.684

Bar graph of word count and character count time block

Word Count

reflection_3 %>% 
  ggplot(aes(x = time_since_first_start, y = word_count)) +
  geom_bar(stat = "summary", fun = "mean", fill = "lavender", color = "black") +
  labs(title = "Mean Word Count by Time Block", x = "Time Block (days since first start)", y = "Mean Word Count") +
  theme_minimal()

Character Count

reflection_3 %>% 
  ggplot(aes(x = time_since_first_start, y = char_count)) +
  geom_bar(stat = "summary", fun = "mean", fill = "lavender", color = "black") +
  labs(title = "Mean Character Count by Time Block", x = "Time Block (days since first start)", y = "Mean Character Count") +
  theme_minimal()

ChatGPT analysis for the first two reflection question

• For the first two reflective questions, we asked ChatGPT to determine whether the response can be classified as yes/no/unsure.

• The two dataset is stored at ~fb_misinfo_interventions/data/chatfuel/processed/reflection_1_clean_classified.csv and ~fb_misinfo_interventions/data/chatfuel/processed/reflection_2_clean_classified.csv .

reflection_1_classified <- read_csv(here::here("data", "chatfuel", "processed", "reflection_1_clean_classified.csv"))

reflection_2_classified <- read_csv(here::here("data", "chatfuel", "processed", "reflection_2_clean_classified.csv"))

# Merging by analytic_id to get treatment arm 

main_1 <- fread(here::here("data", "chatfuel", "processed", "misinfo_clean_wide.csv.gz"))

reflection_1_classified <- reflection_1_classified %>% left_join(main_1 %>% select(analytic_id, arm_coded, phase_coded), by = "analytic_id")

reflection_2_classified <- reflection_2_classified %>% left_join(main_1 %>% select(analytic_id, arm_coded, phase_coded), by = "analytic_id")

Reflection 1

• This table will show the count of the classification for the first reflection question by treatment arm.

# group by treatment arm and classification

reflection_1_classified %>% 
  group_by(arm_coded, classification) %>% 
  summarise(
    n = n()
  ) %>% kable(digits = 3, col.names = c("Arm", "Classification", "Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	Classification	Count
Game	no	476
Game	unsure	373
Game	yes	1143
Long Baseline	no	559
Long Baseline	unsure	616
Long Baseline	yes	1837
Original Baseline	no	830
Original Baseline	unsure	893
Original Baseline	yes	2410
SMS	no	1138
SMS	unsure	1368
SMS	yes	4308
Video	no	466
Video	unsure	486
Video	yes	1340

Affirmative Response

• In this section, we defined affirmative == 1 if the ChatGPT classification is yes. Answers “no” and “unsure” are being treated as 0. The sample consists only of those who responded and classification was available.

• The table contains a column for treatment arm, the number of observation per treatment group excluding missing values, the share of affirmative responses, and the standard error of the share of affirmative responses.

reflection_1_classified %>% 
  mutate(
    affirmative = case_when(
      classification == "yes" ~ 1,
      classification == "no" ~ 0,
      TRUE ~ NA,
      )
  ) %>% 
  select(arm_coded, affirmative) %>%
  drop_na() %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    p = mean(affirmative),
    se = sd(affirmative) / sqrt(n)
    
  ) %>% kable(
    digits = 3,
    col.names = c("Arm", "N", "Share Affirmative", "SE")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Share Affirmative	SE
Game	1619	0.706	0.011
Long Baseline	2396	0.767	0.009
Original Baseline	3240	0.744	0.008
SMS	5446	0.791	0.006
Video	1806	0.742	0.010

reflection_1_classified %>% 
  mutate(
    affirmative_includingNA = case_when(
      classification == "yes" ~ 1,
      TRUE ~ 0,
      )
  ) %>% 
  select(arm_coded, affirmative_includingNA) %>%
  drop_na() %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    p = mean(affirmative_includingNA),
    se = sd(affirmative_includingNA) / sqrt(n)
    
  ) %>% kable(
    digits = 3,
    col.names = c("Arm", "N", "Share Affirmative (including NAs)", "SE")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Share Affirmative (including NAs)	SE
Game	1992	0.574	0.011
Long Baseline	3012	0.610	0.009
Original Baseline	4133	0.583	0.008
SMS	6814	0.632	0.006
Video	2292	0.585	0.010

Keep only Phase 2

reflection_1_classified %>% 
  filter(phase_coded == "Phase 2") %>%
  mutate(
    affirmative_includingNA = case_when(
      classification == "yes" ~ 1,
      TRUE ~ 0,
      )
  ) %>% 
  select(arm_coded, affirmative_includingNA) %>%
  drop_na() %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    p = mean(affirmative_includingNA),
    se = sd(affirmative_includingNA) / sqrt(n)
    
  ) %>% kable(
    digits = 3,
    col.names = c("Arm", "N", "Share Affirmative (including NAs)", "SE")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Share Affirmative (including NAs)	SE
Long Baseline	3012	0.610	0.009
Original Baseline	1386	0.595	0.013
SMS	4404	0.635	0.007

Reflection 2

# group by treatment arm and classification

reflection_2_classified %>% 
  group_by(arm_coded, classification) %>% 
  summarise(
    n = n()
  ) %>% kable(digits = 3, col.names = c("Arm", "Classification", "Count")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	Classification	Count
Game	no	914
Game	unsure	168
Game	yes	925
Long Baseline	no	980
Long Baseline	unsure	243
Long Baseline	yes	1796
Original Baseline	no	1573
Original Baseline	unsure	348
Original Baseline	yes	2218
SMS	no	2012
SMS	unsure	565
SMS	yes	4241
Video	no	1032
Video	unsure	157
Video	yes	1118

• Answer yes across treatment arms, count and percentage

reflection_2_classified %>% 
  mutate(
      affirmative = case_when(
        classification == "yes" ~ 1,
        classification == "no" ~ 0,
        TRUE ~ NA,
        )
    ) %>% 
    select(arm_coded, affirmative) %>%
    drop_na() %>% 
    group_by(arm_coded) %>% 
    summarise(
      n = n(),
      p = mean(affirmative),
      se = sd(affirmative) / sqrt(n)
      
    ) %>% kable(
      digits = 3,
      col.names = c("Arm", "N", "Share Affirmative", "SE")) |>
    kable_styling(bootstrap_options = c("striped", "hover")) |>
    kableExtra::scroll_box( height = "400px")

Arm	N	Share Affirmative	SE
Game	1839	0.503	0.012
Long Baseline	2776	0.647	0.009
Original Baseline	3791	0.585	0.008
SMS	6253	0.678	0.006
Video	2150	0.520	0.011

reflection_2_classified %>% 
  mutate(
    affirmative_includingNA = case_when(
      classification == "yes" ~ 1,
      TRUE ~ 0,
      )
  ) %>% 
  select(arm_coded, affirmative_includingNA) %>%
  drop_na() %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    p = mean(affirmative_includingNA),
    se = sd(affirmative_includingNA) / sqrt(n)
    
  ) %>% kable(
    digits = 3,
    col.names = c("Arm", "N", "Share Affirmative (including NAs)", "SE")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Share Affirmative (including NAs)	SE
Game	2007	0.461	0.011
Long Baseline	3019	0.595	0.009
Original Baseline	4139	0.536	0.008
SMS	6818	0.622	0.006
Video	2307	0.485	0.010

Keep only Phase 2

reflection_2_classified %>% 
  filter(phase_coded == "Phase 2") %>%
  mutate(
    affirmative_includingNA = case_when(
      classification == "yes" ~ 1,
      TRUE ~ 0,
      )
  ) %>% 
  select(arm_coded, affirmative_includingNA) %>%
  drop_na() %>% 
  group_by(arm_coded) %>% 
  summarise(
    n = n(),
    p = mean(affirmative_includingNA),
    se = sd(affirmative_includingNA) / sqrt(n)
    
  ) %>% kable(
    digits = 3,
    col.names = c("Arm", "N", "Share Affirmative (including NAs)", "SE")) |>
  kable_styling(bootstrap_options = c("striped", "hover")) |>
  kableExtra::scroll_box( height = "400px")

Arm	N	Share Affirmative (including NAs)	SE
Long Baseline	3019	0.595	0.009
Original Baseline	1391	0.547	0.013
SMS	4409	0.633	0.007

ChatGPT generated key themes, contrasting the sms course and baseline

Key Themes by Treatment Group:

SMS Group:

Advising caution:

• “I will advise them to be cautious with online sharing of posts” (48 mentions)
• “I will advise them” (48 mentions)

Giving advice:

• “Advice them” (36 mentions)

Suggesting to stop:

• “They should stop” (33 mentions)
• “To stop” (30 mentions)

Game Group:

Giving advice:

-“Advice them” (21 mentions)

Cautioning:

• “Caution them” (12 mentions)

Advising caution:

• “I will advise them” (12 mentions)
• “I will advice them” (12 mentions)

Suggesting to stop:

• “To stop” (11 mentions)

Original Baseline Group:

Advising caution:

• “I will advise them” (32 mentions)

Telling to stop:

• “will tell them to stop” (30 mentions)
• “Tell them to stop” (21 mentions)

Correcting behavior:

• “I will correct them” (28 mentions)

Suggesting to stop:

• “They should stop it” (25 mentions)

Video Group:

Advising caution:

• “I will advise them” (20 mentions)
• “I will advise them to stop” (15 mentions)

Giving advice:

• “I will advice them” (20 mentions)

Telling to stop:

• “I will tell them to stop” (15 mentions)

Correcting behavior:

• “Correct them” (12 mentions)

Long Baseline Group:

Advising caution:

• “I will advise them to be cautious with online sharing of posts” (30 mentions)
• “I will advise them” (20 mentions)

Telling to stop:

• “I will tell them to stop” (20 mentions)
• “Will tell them to stop” (15 mentions)

Suggesting to stop:

• “To stop” (14 mentions)

Summary:

1.Advising caution: Common across all groups, with some groups providing more specific advice about being cautious with online sharing. 2.Giving advice: A prominent theme in SMS, Game, and Video groups, showing a direct approach in guiding friends. 3.Suggesting to stop: Present in all groups, emphasizing the action of halting the spread of misinformation. 4.Correcting behavior: More common in the Original Baseline and Video groups, indicating a focus on rectifying the behavior directly.

Contrasting SMS and Baseline

The most common words in the responses for both the SMS and Original Baseline groups include:

SMS Group:

1. to (4699)
2. them (3207)
3. i (3036)
4. the (2956)
5. information (1884)
6. they (1769)
7. and (1763)
8. it (1713)
9. before (1704)
10. will (1648)

Original Baseline Group:

1. to (2817)
2. them (1977)
3. i (1895)
4. the (1732)
5. and (1063)
6. will (1030)
7. information (1024)
8. it (996)
9. they (869)
10. before (863)
11. While these words are useful, they are too generic to derive meaningful themes. Let’s analyze the phrases containing these common words in context to extract more specific themes. ​​

Contrast Between SMS and Baseline Groups:

1. Emphasis on Caution: Both groups frequently mention advising caution. However, the SMS group provides more specific advice about being cautious with online sharing.
2. Direct Advice: The SMS group more often uses phrases like “Advice them” or “I will advise them to be cautious,” whereas the Baseline group uses “I will advise them” more generically.
3. Stopping the Behavior: Both groups suggest stopping the sharing of misinformation, but the SMS group has slightly more varied phrases for this suggestion.
4. Correction: The Baseline group mentions “correcting” their friends more often than the SMS group, indicating a slight difference in approach to addressing misinformation.

These themes reflect differences in how participants from each group approach conversations about misinformation with their friends. The SMS group appears to focus more on caution and specific advice, while the Baseline group uses a more generic advisory approach and places emphasis on correcting behavior.

Running regression analysis / language model

• In this session, we will run a series of regression analysis for the third reflection question, predicting treatment group using the following key words: “stop”, “think”, “first”, “check”, “evaluate”, “identify”, “investigate”, “analyze”, “research”, “pause”, “question”, “verify”, “verified”, “identified”, “prove”, “proved”, “differentiate”, “distinguish”, “spot”, “confirm”, “confirmed”, “researched”, “analyzed”, “before”, “ask myself”, “asked myself”, “emotion”, “emotions”, “anger”, “angry”, “fear”, “afraid”, “superior”, “superiority”, “feel”, “feeling”.

Probit regression result

• We restrict our sample to Phase 2 and SMS and Long Baseline treatment groups. The probit regression estimators are reported as follows.

words <- c("stop", "think", "first", "check", "evaluate", "identify", "investigate", "analyze", "research", "pause", "question", "verify", "verified", "identified", "prove", "proved",  "differentiate", "distinguish", "spot", "confirm", "confirmed", "researched", "analyzed", "before", "ask myself", "asked myself", "emotion", "emotions", "anger", "angry", "fear", "afraid", "superior", "superiority", "feel", "feeling")


# Function to count matches
count_matches <- function(text, word) {
  str_count(tolower(text), word)
}

reflection_3_analysis <- reflection_3 %>%
  select(followup_reflection_3, arm_coded, phase_coded)

for (word in words) {
  reflection_3_analysis <- reflection_3_analysis %>%
    mutate(!!paste0("count_", gsub(" ", "_", word)) := sapply(followup_reflection_3, count_matches, word = word))
}

# Create dummy variables for treatment group

reflection_3_analysis <- reflection_3_analysis %>% filter(phase_coded == "Phase 2") %>% filter(arm_coded == "Long Baseline" | arm_coded == "SMS" ) %>%
  mutate(
    sms = if_else(arm_coded == "SMS", 1, 0),
    long_baseline = if_else(arm_coded == "Long Baseline", 1, 0)
  )

# Running regression analysis of count of words on each treatment group 

# Select columns that start with "count"
count_vars <- grep("^count", names(reflection_3_analysis), value = TRUE)

# Create the formula for the probit regression
formula_sms <- as.formula(paste("sms ~", paste(count_vars, collapse = " + ")))



# Fit the probit model
probit_model_sms <- feglm(formula_sms, family = binomial(link = "probit"), data = reflection_3_analysis)



# View the summary of the model
etable(probit_model_sms)

##                       probit_model_sms
## Dependent Var.:                    sms
##                                       
## Constant            0.2124*** (0.0197)
## count_stop            -0.0578 (0.0432)
## count_think           -0.0118 (0.0801)
## count_first           -0.0043 (0.0624)
## count_check            0.0297 (0.0578)
## count_evaluate         0.4269 (0.6843)
## count_identify        -0.2250 (0.4284)
## count_investigate      0.1481 (0.1792)
## count_analyze          0.1036 (0.5327)
## count_research        -0.0461 (0.0659)
## count_pause            0.6806 (0.6111)
## count_question        0.4630. (0.2474)
## count_verify          -0.0796 (0.0619)
## count_verified         0.0342 (0.1296)
## count_prove          -0.4827* (0.2176)
## count_proved           0.0031 (0.6088)
## count_differentiate    0.4621 (0.6829)
## count_distinguish       -4.965 (92.31)
## count_spot               4.399 (92.31)
## count_confirm         0.1787* (0.0849)
## count_confirmed        0.1142 (0.2477)
## count_researched       0.6853 (0.6433)
## count_before         0.1114** (0.0412)
## count_emotion         -0.1988 (0.4445)
## count_emotions         0.7675 (0.5920)
## count_anger            0.0982 (0.0981)
## count_angry              4.233 (44.88)
## count_fear             0.6699 (0.4434)
## count_feel           -0.4795. (0.2789)
## count_feeling            4.889 (37.63)
## ___________________ __________________
## S.E. type                          IID
## Observations                     7,401
## Squared Cor.                   0.00729
## Pseudo R2                      0.00599
## BIC                           10,206.7
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Running LASSO

• We will reserve 70% of the data for training and 30% for testing.

• WORK IN PROGRESS.

# Split the data into training and testing sets
set.seed(123)
train_index <- sample(1:nrow(reflection_3_analysis), 0.7 * nrow(reflection_3_analysis))
train_data <- reflection_3_analysis[train_index, ]
test_data <- reflection_3_analysis[-train_index, ]

# Convert the predictor variables to a matrix
x_train <- train_data %>%
  select(starts_with("count_")) %>%
  as.matrix()

# Response variable
y_train <- train_data$sms

# Fit the LASSO model
lasso_model <- cv.glmnet(
  x = x_train,
  y = y_train,
  family = "binomial",
  alpha = 1
)


optimal_lambda <- lasso_model$lambda.min
lasso_coefficients <- coef(lasso_model, s = optimal_lambda)

# Print coefficients
print(lasso_coefficients)

## 37 x 1 sparse Matrix of class "dgCMatrix"
##                             s1
## (Intercept)         0.38384934
## count_stop          .         
## count_think         .         
## count_first         .         
## count_check         .         
## count_evaluate      .         
## count_identify      .         
## count_investigate   .         
## count_analyze       .         
## count_research      .         
## count_pause         .         
## count_question      .         
## count_verify        .         
## count_verified      .         
## count_identified    .         
## count_prove         .         
## count_proved        .         
## count_differentiate .         
## count_distinguish   .         
## count_spot          .         
## count_confirm       .         
## count_confirmed     .         
## count_researched    .         
## count_analyzed      .         
## count_before        0.02956316
## count_ask_myself    .         
## count_asked_myself  .         
## count_emotion       .         
## count_emotions      .         
## count_anger         .         
## count_angry         .         
## count_fear          .         
## count_afraid        .         
## count_superior      .         
## count_superiority   .         
## count_feel          .         
## count_feeling       .

Analysis of free responses

Get OpenAI embeddings and predict treatment status from them

if (!file.exists("./data/chatfuel/processed/reflection_3_embeddings.csv.gz")) {

  reflection_3 = reflection_3[sapply(reflection_3$followup_reflection_3, nchar) > 2, ]


  key = "sk-proj-SteF09YKigRpkLaMrb8cT3BlbkFJxfxniLSPT3SVZ6NHi6A7"


  get_embeddings = function(dt){

    embeddings = create_embedding(
                model = "text-embedding-3-small",
                input = dt$followup_reflection_3 ,
                openai_api_key = key
            )

    embeddings_df = do.call(rbind, embeddings$data$embedding) |> as_tibble()

    colnames(embeddings_df) = paste0("emb_", 1:ncol(embeddings_df))

    embeddings_df$analytic_id = dt$analytic_id
    
    return(embeddings_df)
  }

  batch_size <- 1000
  batches <- split(reflection_3, as.integer((seq_len(nrow(reflection_3)) - 1) / batch_size))

  embeddings_batches = list()

  i = 1 
  for (batch in batches){
    print(i)
    embeddings_batches[[i]] = try(get_embeddings(batch))
    i = i +1
  }

  embeddings = embeddings_batches |> bind_rows() |> select(analytic_id, everything())

  embeddings |> fwrite("./data/chatfuel/processed/reflection_3_embeddings.csv.gz")
} else {
    embeddings = fread("./data/chatfuel/processed/reflection_3_embeddings.csv.gz")

}

library(glmnet)
main$analytic_id <- as.character(main$analytic_id)
embeddings = embeddings |> left_join(main |> select(analytic_id, arm_coded, phase_coded))

embeddings = embeddings |>
  filter(phase_coded == "Phase 2") |>
  mutate(treatment = ifelse(arm_coded == "SMS", 1, 0))


# Train-test split

train_id = sample(embeddings$analytic_id, size = floor(0.7 * nrow(embeddings)))
train_set = embeddings |> filter(analytic_id %in% train_id)
test_set = embeddings |> filter(!(analytic_id %in% train_id))

# Run cross-validated lasso on the train set

X = train_set |> select(starts_with("emb")) |> as.matrix()
Y = train_set$treatment


library(doParallel)
registerDoParallel(10)
res_lasso = cv.glmnet(x = X, y = Y, nfolds = 10, parallel = TRUE)

# Get non-zero coefficients from cross-validated LASSO
nonzero_coefs = rownames(coef(res_lasso, s = 'lambda.1se'))[coef(res_lasso, s = 'lambda.1se')[,1]!= 0] 
nonzero_coefs = nonzero_coefs[grepl("emb", nonzero_coefs)]


# Run OLS using the embedding dimensions identified by LASSO

fml = paste("treatment ~ ", paste(nonzero_coefs, collapse = " + "))

library(fixest)

res_ols = feols(as.formula(fml), test_set)

cat("R2 of the regression is: ", (r2(res_ols))["r2"], "\n")

## R2 of the regression is:  0.03525684

For easier interpretation we will find some examples of sentences for which the predicted values of coming from sms course are largest/smallest.

embeddings$pred_treatment = predict(res_ols, newdata = embeddings)

lowest_sms = reflection_3 |> filter(
    analytic_id %in% (embeddings |> arrange(pred_treatment) |> head(20) |> pull(analytic_id))
) |> select(lowest_predicted_sms = followup_reflection_3)


highest_sms = reflection_3 |> filter(
    analytic_id %in% (embeddings |> arrange(desc(pred_treatment)) |> head(20) |> pull(analytic_id))
) |> select(highest_predicted_sms = followup_reflection_3)


tab = bind_cols(lowest_sms, highest_sms)

tab |> kable()

lowest_predicted_sms	highest_predicted_sms
to stop falsehood	I’ll ask them if they were a hundred percent sure that sharing that information would not cause harm in any way to those receiving it
humans’ survival instinct leads them on primal quest for success and power.	I will ask them to carefully investigate a post before sharing
It’s either you choose to abide or don’t be pressured into thinking otherwise	To clarify what they wrote on social media before sharing it to the community
By changing the world	Will ask them if they are sure of what they are sharing because it can be misleading
That false news spreads like fire and cannot be stopped when it begins to spread.	I will ask him/her if the information is a misinfo or disinfo
To change there bad actitude	I normally do ask them if they have prove of what the share
It will be difficult for them to change in future and should seek other alternatives	Just told them to make proper checks before sharing any post
Try other helpful options to get a living	I will advice them that before they could share a post the fast think they should think about is if their are not manipulated or the post their are sharing is not misinformation
To be patient	Sorry
I will say change your way you think be a better person	I will ask them to delete the misinformation post and find proof of the information first before they post
To have a positive attitude	I’ll ask them to get clarity on a subject matter before posting to avoid misinformation and disinformation.
Always remain positive no matter the circumstances	I ask them to very before sharing
It hard coz people tend to believe what they saw or heard at first hand by that it might be hard to change their mind	I will ask them to first check if the information is true before sharing it
Two of them should be hard working	I will asked them not to share any post that will cause public distractions
Siwill tell them to stop	Make sure they have a proof
Give him power	I will ask them if they have proof
You should stop posting fake news to the individual	Won’t share what u don’t know much about
to desist from such act	𝙁𝙞𝙧𝙨𝙩 𝙢𝙖𝙠𝙚 𝙖 𝙥𝙧𝙤𝙥𝙚𝙧 𝙞𝙣𝙫𝙚𝙨𝙩𝙞𝙜𝙖𝙩𝙞𝙤𝙣 𝙞𝙣 𝙗𝙧𝙞𝙣𝙜𝙞𝙣𝙜 𝙖𝙗𝙤𝙪𝙩 𝙩𝙝𝙚 𝙧𝙚𝙖𝙡 𝙛𝙖𝙘𝙩 𝙤𝙛 𝙩𝙝𝙚 𝙞𝙣𝙛𝙤𝙧𝙖𝙩𝙞𝙤𝙣 𝙗𝙚𝙛𝙤𝙧𝙚 𝙞𝙩 𝙜𝙤𝙚𝙨 𝙫𝙞𝙧𝙖𝙡, 𝙬𝙝𝙞𝙘𝙝 𝙬𝙤𝙪𝙡𝙙 𝙗𝙚 𝙩𝙝𝙚 𝙩𝙧𝙪𝙩𝙝
Be positive	I will tell them not share it because it is a misinformation post
About how can do economic and depend him self from poverty	Before they post any thing online they should ask if it’ll be manipulative, or misleading