Pilot V6
May 03, 2022
Project Overview and Research Question
Given the current technologies, misinformation spreads around the globe at a speed faster than ever before. What we see on the internet not only impacts our mindset and changes our beliefs but also could be transformed into offline personal and societal consequences. This could translate to actions that are physically or mentally harmful to people; some could even be deadly (Mu ̈ller and Schwarz, 2019; Chan et al., 2016). Once misinformation starts to propagate, we face the risk of no longer being able to identify what is the truth. More importantly, misinformation sticks in people’s minds. Numerous psychological experiments have shown that erasing misinformation is challenging even in controlled lab settings (Lewandowsky et al., 2012). Thus, it is important to look into actions and methods that prevent misinformation from entering in people’s minds in the first place, where we “vaccinate” people against misinformation through inoculation. Inoculation methods present weakened versions of the misinformation messages beforehand to build resistance and immunity to false narratives (McGuire et al. 1961). Through inoculation, we are able to prepare ourselves to spot and deal with misinformation targeting our communities and mitigate the potential damage misinformation can cause.
In this project, we would like to “vaccinate” people against misinformation through inoculation, where we develop a text-message-based course to deliver effective treatments to participants.
We want to test out the following research questions :
- Can a text message-based course “inoculate” users against misinformation?
- Hypothesis 1: Courses delivered in text message format treat participants effectively in spotting misinformation (i.e. at least one of the three treatments works, which requires a much smaller sample size than all treatments work)
- Hypothesis 2: Participants who receive analytical treatment should do better in spotting misinformation using analytical techniques. Participants who receive emotional treatment should do better in spotting misinformation using emotional techniques. Participants who receive combo treatment (i.e. mixture of analytical and emotional) should do better in both.
- How can we improve the efficacy and scalability of the course?
- Is the course differentially effective for different types of users (based on age, political affiliations, etc.)?
Current Pilot Scheme (Recruiting Process + Participant Experience Process)
- Step 1: Participants are recruited through Facebook ads, where they will land on the pre-course survey after clicking through one of the ads.
- Step 2: Participants will finish a pre-course survey with the structure listed in the setcion below. Once they finish, they will be asked if they want to join the free text message course and have the option to either take it on Whatsapp or through SMS.
- Step 3: Participants will then be assigned a treatment (control, control_delayed, tactics, emotions, combo) once they fill out their phone number. Then they will be directly enrolled in the course through Zapier.
- Step 4: Once participants start the course, they will receive one message per day for five days, with each day’s content being less than 1200 characters, and some days have interaction question. In the current setting, participants must complete the questions in order to proceed to next day’s content.
- Step 5: Participants will get the link to the post-course survey embedded in the last day of the course (i.e. Day 5) and once again on Day 6, where we congratulate them for taking the course and reminding them to take the post-course survey in order to receive airtime.
- Step 6: Participants enter the post-course survey through the link in the course, and they complete the same format of the questions with a different set of posts. There will be no demographics questions asked in post-course survey
Survey Structure
- Consent
- Pledge: “While there are no right or wrong answers, we will evaluate your participation and discard low quality responses, so think about each question carefully and answer truthfully. Please, confirm you will do so by choosing the right option below.”
- I will pay attention throughout the survey and answer the questions after thinking through them carefully.
- I will NOT pay attention throughout the survey NOR answer the questions after thinking through them carefully.
- Instruction: “You will see a series of social media posts in the following screens. Please, read each one of them carefully and answer the questions at the end of each screen.”
- Section 1 (4 posts)
- Midway Message: “you are about halfway through, keep going!”
- Section 2 (8 posts)
- Demographics (if pre-course survey) / Feedback (if post-course survey)
Note:
- respondents do not know a clear cut difference between section 1 and section 2; the only difference they see are the different questions
Technical Issues in Pilot V6
Zapier
- Validation failed: Distribution already has an enrollment for that user: Duplicate numbers used. Respondents can use the same number in both SMS and Whatsapp, but they cannot use it repeatedly for a single platform
- Invalid Address: Respondents entered a wrong number.
- Currently do not have full Zapier history data downloaded (5000 limit), need to reach out to Zapier support.
Arist
- Some respondents gave feedback that they cannot reply to SMS course messages (i.e. they cannot do the interactive questions and cannot do the course under current setting)
- Potential Solution 1: Disable the setting where we require respondents to reply before being able to proceed
- Caveat: We have no way to know if the respondents actually did the course. They can just skip all the content and click the survey to get airtime.
- Potential Solution 2: Reduce amount of interactive questions (potentially have no interactive questions)
- Caveat: The pilot we did in Cornell had great feedback on interactive questions, who said these questions help them better learn and prefer to have more
- Potential Solution 3: Disable the setting where we require respondents to reply before being able to proceed + Incentivize people to answer the interaction questions
- Caveat 1: Cost
- Caveat 2: Arist individual level data might not be reliable given they just launched 2.0
- Potential Solution 1: Disable the setting where we require respondents to reply before being able to proceed
- Currently do not have full Arist individual level data given the Analytics dashboard doesn’t load anything.
- Course name in Arist cannot be the same. Currently we are using the same name with spaces added in the back to surpass the unique name requirement. However, Arist told us this doesn’t work (caused problems for the messaging system)
- “In short our messenger will stop sending messages to enrollees if the course title is changed to have several spaces at the end. These enrollments will remain in a bad state and need to be reactivated manually by our team.”
- Potential Solution: add symbols to course names
- “Inoculation Against Misinformation”
- “Inoculation” Against Misinformation
- Inoculation - Against Misinformation
- Inoculation_Against_Misinformation
- -Inoculation Against Misinformation-
- ~Inoculation Against Misinformation~
Qualtrics
- Some respondents did not see the topic manipulation question in the post-course survey. This is now fixed.
Ads
- Currently the ads algorithm is optimizing for link clicks. We might want to consider tracking survey completions as the outcome metric. However, the caveat is that ads algorithm does not support tracking for IOS 14.5 users (Note: newest IOS system is 15.x)
- When we first launched the A/B test on 4/11, Facebook did not review all of the ads by the time we scheduled for the A/B test to start. Thus, we stopped the campaign on 4/12, and restarted on 4/13 when all of the ads are indeed reviewed and approved by Facebook. Thus, in the analyses below, we only use data from 4/13 onward. (NOTING HERE so that this doesn’t happen again)
Airtime (Potential Issue)
- We are using Africa’s Talking for airtime distribution, which Vaccine Hesitancy Project in the lab has shown that there might be potential issues. I will monitor after distributing the airtime and post on Facebook to make sure everyone gets the airtime.
Data
Load Packages
Read Data
Data Cleaning (Survey Data)
Variable Encoding
Technical Analysis
Funnel Analysis
Treatment Distribution
We have one control group, one control placebo course, and three treatment courses.
- Plain Control (“Control Delayed”): Respondents receive no course for 5 days and are asked to fill out the post-course survey on Day 5.
- Control Course (“Control Alternative”): Respondents receive a placebo course that remind them misinformation is harmful for five days.
- Tactics Course: Respondents receive a course that focuses on teaching tactical techniques misinformation creators commonly used.
- Emotion Course: Respondents receive a course that focuses on teaching emotional techniques misinformation creators commonly used.
- Combo Course: A combination of Tactics and Emotion course.
Overall
By Messaging Platform
Feedback
| Feedback | Freq |
|---|---|
| a nice éducative and informative survey | 1 |
| add more questions a healthy lifestyle is a healthy nation | 1 |
| am learning alot from you thank you | 1 |
| asante | 1 |
| continue doing these researches to get accurate information | 1 |
| credit when is it being send. | 1 |
| educative survey | 1 |
| excellent survey | 1 |
| good research questions.keep it up | 1 |
| good survey | 1 |
| good survey learnt alot | 1 |
| great | 1 |
| great survey | 2 |
| helpful survey | 1 |
| how do i get more surveys from you | 1 |
| how do you reduce spread of misinformation? we need guidelines | 1 |
| i enjoyed taking the survey | 1 |
| i have enjoyed the surver,im ready to participate next time. | 1 |
| i have learnt a lot, thanks so much | 1 |
| i have learnt more from your survey | 1 |
| i have really learnt through your surveys thanks alot | 1 |
| i like taking the survey sent to me | 1 |
| i love the work you are doing | 1 |
| i really like the survey | 1 |
| i thank you for the platform | 1 |
| i would like to appreciate for your survey and gain knowledge about misconception | 1 |
| i would like to be informed of future surveys | 1 |
| i’m grateful | 1 |
| insightful survey, really opened my eyes | 1 |
| it’s going well | 1 |
| learned a lot about misinformation | 1 |
| looking forword. for more survays | 1 |
| minimise the days for the survey. | 1 |
| misinformation should be addressed because it can cause damage to our society | 1 |
| more survey more airtime. thank you. | 1 |
| next time organize online trainings. | 1 |
| nice lesson🤝 | 1 |
| nice study | 1 |
| nice survey | 1 |
| no | 230 |
| no i don’t | 1 |
| no thank you | 2 |
| no thanks | 5 |
| no. thank you | 1 |
| none | 9 |
| none for today | 1 |
| nope | 2 |
| not actually | 1 |
| not at all | 2 |
| not at the moment | 1 |
| not really | 3 |
| not yet | 1 |
| nothing | 3 |
| some information needs to be proved scientifically so as to avoid manipulation. | 1 |
| splendid survey | 1 |
| straight to point survey | 1 |
| thank for sharing information on disinformation and misinformation | 1 |
| thank you | 4 |
| thank you for educating me about misinformation | 1 |
| thank you for letting me know the difference between misinformation and disinformation | 1 |
| thank you for such an amazing topic. anticipating for more surveys. | 1 |
| thank you for survey | 1 |
| thank you for the opportunity | 1 |
| thank you for the survey | 4 |
| thank you for the survey it has helped me understand more about misinformation | 1 |
| thank you for the survey, i really enjoy it | 1 |
| thank you for the survey. kindly send the incentives though paypal or mpesa | 1 |
| thank you for you for the interview | 1 |
| thank you for your campaign against misinformation | 1 |
| thank you for your survey | 1 |
| thank you for your time.waiting for my airtime | 1 |
| thankfully for giving me this opportunity | 1 |
| thanks | 3 |
| thanks for allowing me participate in the survey | 1 |
| thanks for allowing me to participate in this survey.god bless you | 1 |
| thanks for great information | 1 |
| thanks for making me know much | 1 |
| thanks for the course.it has educative to me. | 1 |
| thanks for the good information | 1 |
| thanks for the information | 1 |
| thanks for the survey its educative and gives alot of details | 1 |
| thanks for the work | 1 |
| thanks for your course.. i really learned a lot | 1 |
| thanks for your support | 1 |
| thanks for your survey | 1 |
| thanks fotlr the survey | 1 |
| thanks you for the survey | 1 |
| thanks, the study was enlightening | 1 |
| thankyou for the survey,i have learnt alot | 1 |
| the course was very educative. i would love more like such | 1 |
| the survey is great | 1 |
| the survey was good and helpful | 1 |
| the survey was very educative . | 1 |
| the survey was very educative please send more on other topic. | 1 |
| think the survey will help some people learn about misinformation and disinformation | 1 |
| this is a great survey | 1 |
| this survey helps not only you but inform us too on medical issues | 1 |
| this thing is it real | 1 |
| this was great | 1 |
| two surveys for one pay. | 1 |
| verry educative | 1 |
| very precise and direct to the point survey | 1 |
| will there be more surveys? | 1 |
| wow | 1 |
| yeah | 1 |
| yes | 3 |
| yes,you people have transformed my brains to always think on the current post on social media | 1 |
| you’re doing a good job | 1 |
| your survey is way too long | 1 |
| your surveys are educative thanks so much. | 1 |
Outcome Analysis
Transform Outcome Variables
Section 1
In section 1, we aim to test whether respondents can distinguish the level of manipulation of the post. We have a pool of 4 manipulative posts as well as their non-manipulative counterparts (i.e. 4 facts). All of the facts are true information; the only thing we varied was the posts’ level of manipulativeness. Wwe validated on Lucid (n = 100) that the manipulative versions on average are in fact preceived as more manipulative. In this section, we show participants two manipulative posts and two non-manipulative posts (with no overlaps of facts) and asked the following questions for each post]
Would you share this post? [NOTE: this outcome is only included so that all sections have the same format (i.e. we do not care about this outcome given all of the posts are true posts)]
- [public] I would share it publicly (e.g. on timeline or feed) [Yes / No]
- [private] I would share it privately (e.g. through a private message) [Yes / No]
Would you do anything else with the post? [Yes / No]
[If Yes to previous question] What else would you do? Please tell us in a few words.
Do you agree or disagree with the following statement about the post? This post is manipulative.
- Completely disagree (-2)
- Disagree (-1)
- Neither agree nor disagree (0)
- Agree (1)
- Completely agree (2)
Sample Post Section 1
Sample Post Section 1
Recall: We have one control group, one control placebo course, and three treatment courses.
- Plain Control (“Control Delayed”): Respondents receive no course for 5 days and are asked to fill out the post-course survey on Day 5.
- Control Course (“Control Alternative”): Respondents receive a placebo course that remind them misinformation is harmful for five days.
- Tactics Course: Respondents receive a course that focuses on teaching tactical techniques misinformation creators commonly used.
- Emotion Course: Respondents receive a course that focuses on teaching emotional techniques misinformation creators commonly used.
- Combo Course: A combination of Tactics and Emotion course.
Manipulativeness
Note: we have the difference-in-difference estimation plot following the raw outcome results below.
Raw Outcome
This is the raw score of people’s opinion on post manipulativeness on average for each group. Each individual’s score is averaged first before taking the group average. Note that manipulativeness rating takes range between -2 and 2, where 2 means very manipulative and -2 means not at all manipulative.
Raw Outcome Point Estimate Table
| treatment | Pre-Survey (Non-Manip) | Pre-Survey (Manip) | Post-Survey (Non-Manip) | Post-Survey (Manip) |
|---|---|---|---|---|
| control_delayed | -0.137 | -0.108 | 0.093 | 0.071 |
| combo | -0.086 | -0.140 | 0.089 | 0.105 |
| control_alternative | -0.069 | -0.181 | 0.047 | 0.152 |
| emotion | -0.115 | -0.143 | 0.133 | 0.262 |
| tactics | -0.049 | -0.239 | 0.093 | 0.192 |
Raw Outcome Distribution Plot (WIP)
Outcome Estimation
We estimate the outcome as compared to control delayed group (i.e. treating control alternative as a treatment) using Difference-in-Difference estimation.
Note that numbers being positive mean that people perceive posts as being more manipulative.
What to look for: We want treatments to have positive coefficients for the manipulative posts.
Section 2
In section 2, we aim to test whether respondents can distinguish whether a post is misinformation or not. We have a pool of 4 misinformation posts with emotion techniques, 4 misinformation posts with tactics, 4 general misinformation posts, 4 factually true posts. We show each respondent two posts from each type (randomized), and asked them the following questions for each:
Note that the facts in all of the posts do not overlap (i.e. we have 16 facts / topics)
Would you share this post? - [public] I would share it publicly (e.g. on timeline or feed) [Yes / No] - [private] I would share it privately (e.g. through a private message) [Yes / No]
Would you do anything else with the post? [Yes / No]
[If Yes to previous question] What else would you do? Please tell us in a few words.
Do you agree or disagree with the following statement about the post? The information presented in this post is accurate.
- Completely disagree (-2)
- Disagree (-1)
- Neither agree nor disagree (0)
- Agree (1)
- Completely agree (2)
Sample Post Section 2
Sample Post Section 2
Sharing
Note: we have the difference-in-difference estimation plot following the raw outcome results below.
Raw Outcome
This is the raw score of sharing behavior on average. The terms in the brackets represent the group of posts. For example “(Tactics)” means that this set of posts contain misinformation that uses techniques learned in the tactics course.
Raw Outcome Point Estimate Table
| treatment | Pre-Survey (Tactics) | Pre-Survey (Emotions) | Pre-Survey (General) | Pre-Survey (True) | Post-Survey (Tactics) | Post-Survey (Emotions) | Post-Survey (General) | Post-Survey (True) |
|---|---|---|---|---|---|---|---|---|
| control_delayed | 0.696 | 0.654 | 0.647 | 0.778 | 0.737 | 0.686 | 0.679 | 0.792 |
| combo | 0.728 | 0.726 | 0.685 | 0.794 | 0.778 | 0.707 | 0.700 | 0.821 |
| control_alternative | 0.744 | 0.678 | 0.673 | 0.777 | 0.789 | 0.719 | 0.712 | 0.822 |
| emotion | 0.747 | 0.733 | 0.715 | 0.815 | 0.832 | 0.806 | 0.781 | 0.850 |
| tactics | 0.717 | 0.713 | 0.702 | 0.784 | 0.765 | 0.707 | 0.676 | 0.802 |
Raw Outcome Distribution Plot (WIP)
Outcome Estimation
We estimate the outcome as compared to control delayed group (i.e. treating control alternative as a treatment) using Difference-in-Difference estimation.
Note that numbers being positive mean that people share more posts.
What to look for: We want treatments to have negative coefficients for the tactics, emotions, general misinformation posts.
Misinformation Post vs True Post
Each Misinformation Post Type vs True Information Post
Accuracy
Note: we have the difference-in-difference estimation plot following the raw outcome results below.
Raw Outcome
This is the raw score of people’s opinion on post accuracy on average for each group. Each individual’s score is averaged first before taking the group average. Accuracy rating takes range between -2 and 2, where 2 means very accurate and -2 means not at all accurate. The terms in the brackets represent the group of posts. For example “(Tactics)” means that this set of posts contain misinformation that uses techniques learned in the tactics course.
Raw Outcome Point Estimate Table
| treatment | Pre-Survey (Tactics) | Pre-Survey (Emotions) | Pre-Survey (General) | Pre-Survey (True) | Post-Survey (Tactics) | Post-Survey (Emotions) | Post-Survey (General) | Post-Survey (True) |
|---|---|---|---|---|---|---|---|---|
| control_delayed | 0.462 | 0.356 | 0.274 | 0.679 | 0.666 | 0.274 | 0.332 | 0.790 |
| combo | 0.618 | 0.422 | 0.339 | 0.710 | 0.637 | 0.382 | 0.411 | 0.820 |
| control_alternative | 0.613 | 0.453 | 0.404 | 0.794 | 0.721 | 0.441 | 0.402 | 0.895 |
| emotion | 0.636 | 0.528 | 0.619 | 0.769 | 1.059 | 0.559 | 0.619 | 0.997 |
| tactics | 0.577 | 0.467 | 0.371 | 0.761 | 0.698 | 0.396 | 0.319 | 0.832 |
Raw Outcome Distribution Plot (WIP)
Outcome Estimation
We estimate the outcome as compared to control delayed group (i.e. treating control alternative as a treatment) using Difference-in-Difference estimation.
Note that numbers being positive mean that people think posts are more accurate.
What to look for: We want treatments to have negative coefficients for the tactics, emotions, general misinformation posts.
Misinformation Post vs True Post
Each Misinformation Post Type vs True Information Post
Power Analysis
Here we conduct power analysis using variance estimated from the data, the budget constraints, and some potential hypothesized effects.
Standard Deviation Estimation
Manipulativeness Outcome
| non-manipulative_post | manipulative_post | |
|---|---|---|
| control_alternative | 1.289592 | 1.287640 |
| tactics | 1.315608 | 1.298153 |
| emotion | 1.296217 | 1.379650 |
| combo | 1.337427 | 1.293228 |
Sharing Outcome
| tactics_post | emotion_post | general_misinfo_post | true_post | |
|---|---|---|---|---|
| control_alternative | 0.3274150 | 0.3229590 | 0.3139937 | 0.3354594 |
| tactics | 0.3322810 | 0.3269294 | 0.3241769 | 0.3221962 |
| emotion | 0.3154273 | 0.3281613 | 0.3044524 | 0.3141088 |
| combo | 0.3347978 | 0.3270001 | 0.3266851 | 0.3293715 |
Accuracy Outcome
| tactics_post | emotion_post | general_misinfo_post | true_post | |
|---|---|---|---|---|
| control_alternative | 1.075075 | 1.055985 | 1.130914 | 1.055506 |
| tactics | 1.159213 | 1.074389 | 1.034656 | 1.139570 |
| emotion | 1.124643 | 1.092479 | 1.106386 | 1.109961 |
| combo | 1.100337 | 1.109713 | 1.084562 | 1.106594 |
MDE Calculation
Manipulativeness Outcome
Sharing Outcome
Accuracy Outcome
Power Calculation
Manipulativeness Outcome
## [1] 1164 10475Sharing Outcome
## [1] 1560 6240Accuracy Outcome
## [1] 716 11449Overall Takeaways (for Outcome and Power Analysis)
- For the manipulativeness outcome, which yields the cleanest results, Tactics and Combo seem to be going in the right direction. That is, people perceive manipulative posts more manipulative after they took the course more than those in the control delayed arm. However, this effect is mitigated when we compare against control alternative (which is a daily reminder that misinformation is bad). Emotion seems to be going in the opposite direction. In any case, results are very noisy to draw any conclusions.
- The sharing outcome is high on average (treatment effects are not really interpretable, too noisy).
- The accuracy outcome is less clean than manipulativeness, and we can’t really conclude anything.
- Despite the noisiness of the estimations, we were able to use the mean and SD of the outcomes to draft some rough power calculations:
- A budget of $30,000 suffices for a sample size of 20,000 at the conservative cost of $1.50 per observation.
- This is more than enough to detect effects of about 10% of the standard deviation of the current outcomes, after correcting for MHT, which requires a sample size of about 10,000.
- However, we want to make modifications to how we are measuring the outcomes, based on the high baseline values of the responses to the sharing questions (also include attention checks and possibly incentivization).
Ads Analysis
We ran an A/B test on three ad sets with three themes
- Logo: all ads include logo of Viral Facts Africa and First Draft and all headlines contain the phrase “supported by WHO”. The images are different color versions of the same image.
- Non-Logo: same images and headlines as the Logo ad set except all the logos and organization names are taken out
- Non-Course: four pre-set social media images with no logos and organization names (the headlines are the same as non-logo set)
Essentially, we are mix-and-matching the five headlines with the images below.
Headline
Note: We will only include the phrase “supported by WHO” if the ads are in the Logo ad set, else we exclude it.
Images (Logo)
Images (Non-Logo)
Images (Non-Course)
!!Note: When we first launched the A/B test on 4/11, Facebook did not review all of the ads by the time we scheduled for the A/B test to start. Thus, we stopped the campaign on 4/12, and restarted on 4/13 when all of the ads are indeed reviewed and approved by Facebook. Thus, in the analyses below, we only use data from 4/13 onward.
Takeaway
- According to Facebook ads manager A/B test result analytics, non-course ads clearly win as compared to ads either with or without logo, suggesting that not mentioning the course in the image gives us more click through rate.(We will still need to do our own calculation as noted in issue #22)
- However, getting higher click through rate does not necessarily translate to pre-survey completion.