Outcome Measurement Survey V4
March 16, 2022
Project Overview and Research Question
Given the current technologies, misinformation spreads around the globe at a speed faster than ever before. What we see on the internet not only impacts our mindset and changes our beliefs but also could be transformed into offline personal and societal consequences. This could translate to actions that are physically or mentally harmful to people; some could even be deadly (Mu ̈ller and Schwarz, 2019; Chan et al., 2016). Once misinformation starts to propagate, we face the risk of no longer being able to identify what is the truth. More importantly, misinformation sticks in people’s minds. Numerous psychological experiments have shown that erasing misinformation is challenging even in controlled lab settings (Lewandowsky et al., 2012). Thus, it is important to look into actions and methods that prevent misinformation from entering in people’s minds in the first place, where we “vaccinate” people against misinformation through inoculation. Inoculation methods present weakened versions of the misinformation messages beforehand to build resistance and immunity to false narratives (McGuire et al. 1961). Through inoculation, we are able to prepare ourselves to spot and deal with misinformation targeting our communities and mitigate the potential damage misinformation can cause.
In this project, we would like to “vaccinate” people against misinformation through inoculation, where we develop a text-message-based course to deliver effective treatments to participants.
We want to test out the following research questions :
- Can a text message-based course “inoculate” users against misinformation?
- Hypothesis 1: Courses delivered in text message format treat participants effectively in spotting misinformation (i.e. at least one of the three treatments works, which requires a much smaller sample size than all treatments work)
- Hypothesis 2: Participants who receive analytical treatment should do better in spotting misinformation using analytical techniques. Participants who receive emotional treatment should do better in spotting misinformation using emotional techniques. Participants who receive combo treatment (i.e. mixture of analytical and emotional) should do better in both.
- How can we improve the efficacy and scalability of the course?
- Is the course differentially effective for different types of users (based on age, political affiliations, etc.)?
Current Pre-Test Scheme
We are interested in investigating what outcome measures we should use in the actual experiment. Thus, we are pre-testing using Qualtrics survey to find questions (i.e. outcomes) and associated display formats that would generate heterogeneous answers.
In this analysis script version, we used Facebook ads for recruitment of Kenyan respondents who are 18 years or older. The ads directly connect to our Qualtric survey. Note: The performance of individual ad is not of interest to this project at this current stage. We are mostly interested in getting as many participants as possible during the pilot stage
Survey Structure
- Consent
- Pledge: “While there are no right or wrong answers, we will evaluate your participation and discard low quality responses, so think about each question carefully and answer truthfully. Please, confirm you will do so by choosing the right option below.”
- I will pay attention throughout the survey and answer the questions after thinking through them carefully.
- I will NOT pay attention throughout the survey NOR answer the questions after thinking through them carefully.
- Instruction: “You will see a series of social media posts in the following screens. Please, read each one of them carefully and answer the questions at the end of each screen.”
- Section 1 (4 posts)
- Midway Message: “you are about halfway through, keep going!”
- Section 2 (8 posts)
- Demographics
Note:
- respondents do not know a clear cut difference between section 1 and section 2; the only difference they see are the different questions
Summary of Learning
Demographics Description
- n = 220
- Median age is 27 with 1st and 3rd quartiles being 24 and 31 (median age in Kenya is about 20 years old) [~same as previous]
- Around 34% respondents are female and 66% are male [~ same as previous]
- around 51% respondents have some college degree
- 39% unemployed and looking for work [~ same as previous]
- mostly right leaning (median 66 on a scale of 100, where 100 means extremely right)
- 48.6% live in mostly rural areas; 51.4% mostly urban [better balance]
- most describe themselves as having locus of control (median 8 on a scale of 10)
- most use social media around 3 (1st quartile) to 7 (3rd quartile) hours on a daily basis, with median being 5 hours [~ same as pilot v2]
- social media shares:
- 0-20% share (12% of total)
- 20-40% share (22% of total)
- 40-60% share (35% of total)
- 60-80% share (20% of total)
- 80-100% share (11% of total)
Ads
- We used $21.26 for ads in total to recruit n = 220 completed responses (~0.097 USD per survey completion)
- Click through rate is at 3.8% (# clicks = 1027) with cost per click being $0.021
- Only dropoff place we can measure is at the consent level, where 220 (21.4%) people pass the consent questions and all complete till the end of the survey
- Note We automatically shutoff qualtrics for collection after ~200 responses therefore the dropoff above will not be accurate as we closed the ad campaign at a later stage.
- Facebook prioritized ads with airtime only headline [See more in Ads Analysis Section]
- Note the campaign objective is click through rate instead of tracking conversions. This is different than the campaign ran before.
Outcome
IPotentail Issue - around 12% of respondents have selected the same answer in manipulativeness / accuracy question on six or more consecutive questions - this is much better than previous version, where 21% had >=6 consecutive same answers
Learning
- After cleaning out consecutively same answer responses (>= 6 posts) observations, we find that the distribution of manipulativeness / accuracy seems to have heterogeneity that we want, suggesting this outcome measure works.
- Section 1: good distribution with heterogeneity
- Section 2: factually true posts are more likely to get recognized as accurate. One emotion post has strong distribution of being recognized as non-accurate
- Sharing distribution looks much better than previous collection
- Section 1: people are less likely to share manipulative posts than non-manipulative posts
- Section 2: people are less likely to share emotion posts; factually true posts are shared most often among all four types
Yet To Be Done
- Analyze topics of free text response
Data
Load Packages
Read Data
Data Cleaning
Variable Encoding
Data Analysis
Section 1
In section 1, we aim to test whether respondents can distinguish the level of manipulation of the post. We have a pool of 4 manipulative posts as well as their non-manipulative counterparts (i.e. 4 facts). We show two manipulative posts and two non-manipulative posts (with no overlaps of facts) and asked the following questions for each post:
Would you share this post? - [public] I would share it publicly (e.g. on timeline or feed) [Yes / No] - [private] I would share it privately (e.g. through a private message) [Yes / No]
Would you do anything else with the post? [Yes / No]
[If Yes to previous question] What else would you do? Please tell us in a few words.
Do you agree or disagree with the following statement about the post? This post is manipulative.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 1
Sample Post Section 1
List of Actions to Take
List of Action People Say
non-manipulative_fact1
| x1_1_a_other | Freq |
|---|---|
| Ask my male friends if the would report chest pains | 1 |
| Ask women around me if they have had chest pains before, and how they handled it. | 1 |
| Create awareness about it | 1 |
| Create awareness in my timeline that i completely agree | 1 |
| I will discuss with friends and family | 1 |
| I would share it publicly with all my friends | 1 |
| Indicate the causes of chest pain | 1 |
| Research first | 1 |
| save for later reference | 1 |
| save it | 1 |
| Seek if its true | 1 |
| share | 1 |
| Share information with my friends | 1 |
| Share it with friends, face-face | 1 |
| Share with friends | 1 |
| Share with my close friends | 1 |
| Share with my friend | 1 |
| To brainstorm my students | 1 |
| try to do more research about it& get to know how to avoid it | 1 |
| Women to report chest pain so that it can be dealt with by doctors | 1 |
non-manipulative_fact2
| x1_2_a_other | Freq |
|---|---|
| Advice my close friends to avoid foods that may cause obesity | 1 |
| Best | 1 |
| Discuss it with people verbally | 1 |
| Encourage my friends to watch their weight | 1 |
| I would love to educate them on how to live a healthy life | 1 |
| I would use it to warn people on the dangers of overweight | 1 |
| Send to groups | 1 |
| Share it via whatsapp | 1 |
| Share it widely | 1 |
| Share on my timeline | 1 |
| To teach other youths meaning of good health | 1 |
| Ways of reducing body weight | 1 |
| Will observe my diet as stated in the post | 1 |
| Yes i agree obese is not healthy lead to body problems | 1 |
non-manipulative_fact3
| x1_3_a_other | Freq |
|---|---|
| Add some more information | 1 |
| Advice men to find alternative precuedures than suicide | 1 |
| Create awareness | 1 |
| Create awareness on the importance of life that God gave us | 1 |
| Give it to students to discuss in class | 1 |
| Give the possible causes of suicides | 1 |
| Have a discussion about the causes | 1 |
| I will engage in conversation with a group discussing on the same | 1 |
| I will share the information | 1 |
| I would educate my people on why it is not good to commit suicide | 1 |
| I would research on the facts | 1 |
| I would tag as many people as possible | 1 |
| Let others know through other social networks | 1 |
| Make my friends understand | 1 |
| Nothing | 1 |
| Reread and share | 1 |
| Send to groups so that they can be helped psychologically | 1 |
| Share information about it with my friends | 1 |
| Share information with my friends | 1 |
| Share it widely | 1 |
| This is a wrong doing | 1 |
non-manipulative_fact4
| x1_4_a_other | Freq |
|---|---|
| Do a follow up research | 1 |
| Do more research | 1 |
| encourage my viewers &follower’s to use vitamin c products to prevent them from ailing | 1 |
| I will share it on my WhatsApp | 1 |
| I will share it with medical professionals to know their opinions | 1 |
| i would discuss the contents with my peers and try to break down its contents for use | 1 |
| I would post it privately | 1 |
| I would send to many people | 1 |
| I would talk to my friends when we are together. | 1 |
| Share | 1 |
| Share it more on social media platforms | 1 |
| Share it other Facebook groups | 1 |
| Share widely | 1 |
| Tag a close friend | 1 |
| Teach people on how to use it | 1 |
| Tell kids | 1 |
| Tell my family | 1 |
| You it to market my page | 1 |
non-manipulative_fact5
| x1_5_a_other | Freq |
|---|---|
| Encourage the use of aloe vera | 1 |
| Find out more about the importance of aloe vera | 1 |
| Give a testimony | 1 |
| Help someone know it’s use | 1 |
| I will do more research | 1 |
| It doesn’t have side effects | 1 |
| Keep it for future use | 1 |
| like the post | 1 |
| Love if it is useful | 1 |
| Put it in a public place for eveeyone to see | 1 |
| Recommend use of aloe vera | 1 |
| Share it family members | 1 |
| Take the advice and orally tell others | 1 |
| Teach people important of aloe vera | 1 |
| The aloe vera plant is medicine | 1 |
| To use aloe Vera product | 1 |
| Try using aloe vera | 1 |
| Use the post to explain the importance of this plant to those who cannot access social media | 1 |
non-manipulative_fact6
| x1_6_a_other | Freq |
|---|---|
| Add some few words | 1 |
| anybody | 1 |
| Create awareness | 1 |
| Fact check on possible causes of migraines | 1 |
| I agree | 1 |
| I will do more research about Aloe Vera | 1 |
| I will post it in my timeline for the benefit of the society | 1 |
| I will post it on social media | 1 |
| I would save it. It’s educative | 1 |
| Not good to skip any diet can cause illness | 1 |
| Post on Facebook | 1 |
| Repost to whatsapp groups | 1 |
| save it | 1 |
| Save on phone | 1 |
| sensitize people | 1 |
| Share | 1 |
| Share it with my friends | 1 |
| Share verbally | 1 |
| Share with friends | 1 |
| Tag my friends | 1 |
| Talk to my friends about it | 1 |
| Tell my friends and family members | 1 |
| Tell people Abt it | 1 |
| use the knowledge to help others | 1 |
non-manipulative_fact7
| x1_7_a_other | Freq |
|---|---|
| Advise my girlfriend | 1 |
| Educate friends through physical address | 1 |
| I will answer all questions attentively | 1 |
| I would advise women to be careful | 1 |
| I would investigate further. | 1 |
| I’ll educate people about it | 1 |
| I’ll keep it for reference | 1 |
| Make it go viral to save life | 1 |
| Share on other social networks | 1 |
| Sharing | 1 |
| Show my friends and family about this | 1 |
| Tell friends | 1 |
| Tell it to those who are not on social media | 1 |
| Understand the information and educate others. | 1 |
| Whatsapp status | 1 |
non-manipulative_fact8
| x1_8_a_other | Freq |
|---|---|
| Find out more about effects of antibiotics when used with other substance | 1 |
| I would save it to share at a later date | 1 |
| Inform people about how to properly use antibio | 1 |
| Inviting other people | 1 |
| Post in all social media platforms | 1 |
| Post it | 1 |
| Report the post | 1 |
| Research the author and information posted | 1 |
| Share | 1 |
| Share on my social media networks | 1 |
| Tell my friends | 1 |
| Tell people that drugs are meant to be taken with water or per doctors instructions. | 1 |
| To follow what doctors prescriptions | 1 |
manipulative_fact1
| x1_1_b_other | Freq |
|---|---|
| Comment on it to share my thoughts | 1 |
| Consult a medical doctor about the complain | 1 |
| Do some research abouy it | 1 |
| Find out more by researching | 1 |
| I will share the information | 1 |
| I will share with my friend,for them to take part also in this survey | 1 |
| I would send to many prople | 1 |
| I would tell women that women and men are all the same. We are all human beings and we should be treated equally. | 1 |
| Make a research on it | 1 |
| Post on my timeline status | 1 |
| Report it | 1 |
| Research more about the project | 1 |
| Share | 1 |
| Share it on all social media account | 1 |
| Take note and talk about it with a friend | 1 |
| Take the survey | 1 |
| Teach people | 1 |
| want people the fact | 1 |
| will educate others | 1 |
manipulative_fact2
| x1_2_b_other | Freq |
|---|---|
| Advise on activities to be done inorder to lose the excess weight | 1 |
| Conduct a research | 1 |
| Create awareness about obesity using this post | 1 |
| Discuss it verbally | 1 |
| elaborate how | 1 |
| Encourage friends and family members to be physical fit by doing some exercise | 1 |
| Exercise can get rid of excessive fat,and avoid choresteral food. | 1 |
| I will share with my friends | 1 |
| I would help by doing research on what one should use to control weight & maintain | 1 |
| I would screen shot it to use as an example while explaining to friends | 1 |
| Keep it self | 1 |
| Nothing | 1 |
| Sensitize my friends on dangers of excess weight. | 1 |
| Share information with my friends | 1 |
| Share with my overwheight friends | 1 |
| Use it to spark a conversation on body weight and health | 1 |
| We should check our weight and diet regularly. | 1 |
manipulative_fact3
| x1_3_b_other | Freq |
|---|---|
| Add more words to support the post like more reasons | 1 |
| discuss the issue with specialists &try find the causes and reasons leading to suicide | 1 |
| Do a research on it | 1 |
| Educate people about it | 1 |
| Girl child has been given much attention | 1 |
| more guidance is need to this men generation | 1 |
| Report the post to Facebook administration | 1 |
| share | 1 |
| Share | 1 |
| Share information | 1 |
| Share it to different social media | 1 |
| Share verbally with others | 1 |
| Tell friends | 1 |
| Try to find out why | 1 |
| Would relate my experiences as a man to the post | 1 |
manipulative_fact4
| x1_4_b_other | Freq |
|---|---|
| Correct the person to avoid using abusive language | 1 |
| Discuss with friends | 1 |
| download | 1 |
| Encourage people take vitamin c | 1 |
| extensive research | 1 |
| Find out more about vitamin c | 1 |
| I will recommend other friends of mine to carry out the survey | 1 |
| I will share on other social platforms | 1 |
| Post | 1 |
| Sharing it with friends as a discussion | 1 |
| Teach on importance of vitamin C in the body | 1 |
manipulative_fact5
| x1_5_b_other | Freq |
|---|---|
| Aloe Vera is a cure | 1 |
| I will tell my friends about the importance of aloe Vera | 1 |
| I would do a proper research and add a Comment to support | 1 |
| I would do more research about Aloe Vera | 1 |
| I would do the said practically | 1 |
| I’ll share. The information with my friends | 1 |
| Nothing | 1 |
| Post it on my timeline coz its something helpful to the society | 1 |
| Public education | 1 |
| save it in my gallery | 1 |
| Share it privately and publicly | 1 |
| utilize the knowledge for personal use | 1 |
| Will tell my friends about it | 1 |
manipulative_fact6
| x1_6_b_other | Freq |
|---|---|
| Assist or donate food | 1 |
| Discuss it verbally | 1 |
| do research | 1 |
| Educate others and remind myself when experiencing a migraine | 1 |
| Encourage everyone not to skip a meal | 1 |
| Explain importance of taking meals to those who cannot get all meals as required | 1 |
| Hang a picture of it in my house | 1 |
| I will share with friends out of social media | 1 |
| I will tag my friends so that they can participate | 1 |
| I would advertise it, so it can reach many people | 1 |
| I would advise people to take of their skin | 1 |
| I would share it with all of my friends and relatives | 1 |
| Share | 1 |
| Share it in my Twitter account | 1 |
| Share on other platforms | 1 |
| Stil sensitize people about it | 1 |
| Tell people to exercise and eat healthy. You’re what you eat | 1 |
| Would use to teach other people on health issues | 1 |
| Write my comment | 1 |
manipulative_fact7
| x1_7_b_other | Freq |
|---|---|
| advice people not to use antibiotics | 1 |
| Discuss with poster | 1 |
| Find out more about it | 1 |
| I will change the wording | 1 |
| Post | 1 |
| Reseatpn the truth of the post | 1 |
| save it | 1 |
| Save it as my profile photo | 1 |
| share | 1 |
| Share information about it | 1 |
| Share information with my friends | 1 |
| Share it to people widely | 1 |
| Talk with friends | 1 |
| Teach people | 1 |
| To make sure i do more research on that | 1 |
| Would share on telegram groups | 1 |
| Would use it to warn people on the dangers of using antibiotics | 1 |
manipulative_fact8
| x1_8_b_other | Freq |
|---|---|
| Advice men and women to use alcohol responsibily | 1 |
| Advise friends on responsible drinking | 1 |
| Contact professional doctor about it | 1 |
| Discuss | 1 |
| I will encourage people | 1 |
| I will sensitize people who are not insocial media about it | 1 |
| I would send to many people | 1 |
| Public awareness | 1 |
| Report the post for misinformation. | 1 |
| Share | 1 |
| Share it with others | 1 |
| Some Doctors would tell you that the drugs will be poisonous when you take alcohol. | 1 |
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2 | 3.0 | 2.783 | 4 | 5 | 1.091 |
| non-manipulative_fact2 | 1 | 2 | 3.0 | 3.040 | 4 | 5 | 1.324 |
| non-manipulative_fact3 | 1 | 2 | 2.0 | 2.738 | 4 | 5 | 1.384 |
| non-manipulative_fact4 | 1 | 2 | 3.0 | 3.035 | 4 | 5 | 1.388 |
| non-manipulative_fact5 | 1 | 2 | 2.0 | 2.906 | 5 | 5 | 1.572 |
| non-manipulative_fact6 | 1 | 2 | 3.0 | 2.803 | 4 | 5 | 1.209 |
| non-manipulative_fact7 | 1 | 2 | 3.0 | 2.815 | 4 | 5 | 1.134 |
| non-manipulative_fact8 | 1 | 2 | 3.0 | 3.100 | 5 | 5 | 1.464 |
| manipulative_fact1 | 1 | 2 | 4.0 | 3.302 | 4 | 5 | 1.227 |
| manipulative_fact2 | 1 | 2 | 3.0 | 3.034 | 4 | 5 | 1.297 |
| manipulative_fact3 | 1 | 2 | 3.0 | 2.736 | 4 | 5 | 1.375 |
| manipulative_fact4 | 1 | 2 | 3.0 | 2.980 | 4 | 5 | 1.304 |
| manipulative_fact5 | 1 | 2 | 3.5 | 3.229 | 4 | 5 | 1.325 |
| manipulative_fact6 | 1 | 2 | 2.0 | 2.393 | 3 | 5 | 1.144 |
| manipulative_fact7 | 1 | 2 | 3.0 | 3.019 | 4 | 5 | 1.321 |
| manipulative_fact8 | 1 | 2 | 4.0 | 3.167 | 4 | 5 | 1.424 |
Section 2
In section 2, we aim to test whether respondents can distinguish whether a post is misinformation or not. We have a pool of 4 misinformation posts with emotion techniques, 4 misinformation posts with tactics, 4 general misinformation posts, 4 factually true posts. We show two posts from each type (randomized), and asked them the following questions for each:
Note that the facts in all of the posts do not overlap (i.e. we have 16 facts / topics)
Would you share this post? - [public] I would share it publicly (e.g. on timeline or feed) [Yes / No] - [private] I would share it privately (e.g. through a private message) [Yes / No]
Would you do anything else with the post? [Yes / No]
[If Yes to previous question] What else would you do? Please tell us in a few words.
Do you agree or disagree with the following statement about the post? The information presented in this post is accurate.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 2
Sample Post Section 2
List of Actions to Take
List of Action People Say
tactics_fact1
| x2_1_t_other | Freq |
|---|---|
| Add to a public blog for more people to know | 1 |
| Carrots don’t prevent diabetes, instead urge to republish | 1 |
| Create awareness | 1 |
| discuss | 1 |
| Discuss with friends | 1 |
| Do a follow up research | 1 |
| Do more research | 1 |
| Educate the public on importance of nutrition | 1 |
| Encourage people | 1 |
| Find out more on the article and other publishers | 1 |
| Get the article and substantiate it with other peer reviewed articles of the same topic | 1 |
| I only disagree | 1 |
| I will definitely start using carrot in my diet | 1 |
| I will sensitize those who are not insocial media | 1 |
| I would inform people by speaking with them | 1 |
| I would practice it myself | 1 |
| I would still encourage people to feed on carrots but after doing a good research | 1 |
| Include some words from the article | 1 |
| indicate other importance of eating carrots to you health | 1 |
| It true | 1 |
| Post | 1 |
| Public awareness | 1 |
| Research more into it | 1 |
| Send this to friends | 1 |
| share | 1 |
| Share information with my friends | 2 |
| Share it among my friends in social media and the friends in real life | 1 |
| Share it to friends | 1 |
| Share on my timeline | 1 |
| Share on other social media. | 1 |
| Share to friends | 1 |
| Shere information | 1 |
| tell people to include carrots in there diet | 1 |
| try the diet | 1 |
| Try to find out whether article gives the right information | 1 |
| use the knowledge in the post to improve health | 1 |
tactics_fact2
| x2_2_t_other | Freq |
|---|---|
| Do more research about the post | 1 |
| Do research | 1 |
| I have to dig in and do research of early years till now | 1 |
| Keep it secret | 1 |
| Nothing | 1 |
| Post | 1 |
| Predict the new effects | 1 |
| Report the post to Facebook administration | 1 |
| Screenshot it | 1 |
| Share information with my friends | 1 |
| Share it publicly | 1 |
| Share it with others | 1 |
| Share on my timeline | 1 |
| Tell people | 1 |
| Uty | 1 |
tactics_fact3
| x2_3_t_other | Freq |
|---|---|
| Campaign towards everyone getting vaccinated | 1 |
| Discuss with friends about it | 1 |
| Encourage members of the public to vaccinate their children | 1 |
| i support kids to be vaccinated | 1 |
| I will educate people on vaccines intake | 1 |
| I would send to many people | 1 |
| I’ll use to ask people some questions | 1 |
| Let’s get vaccinated,Dont mix your health issue with cultures. | 1 |
| parents need to sensitized | 1 |
| Sensitize parents to vaccinate their children | 1 |
| Sharing to others | 1 |
| Tell parents take their children for vaccinations,it prevents diseases | 1 |
| use it to educate people on importance of vaccination | 1 |
| Vaccination is very important to our children | 1 |
| vaccines help children to fight against diseases | 1 |
| We should not be against children’s vaccine | 1 |
tactics_fact4
| x2_4_t_other | Freq |
|---|---|
| Advice fellows to stop taking sugars in excess | 1 |
| Benefit from it by reducing sugar consumption | 1 |
| Comment about it | 1 |
| Comment on the post | 1 |
| Correct the misleading facts | 1 |
| Educate my children | 1 |
| Educate people about obesity | 1 |
| Educate people on taking less sugar. | 1 |
| Educate people on what causes cancer and what triggers cancer cells. | 1 |
| Explain the risks of obesity to those who cannot get access to the internet or social media service | 1 |
| Find out if sugar really causes cancer from medical practitioners. | 1 |
| Give solutions | 1 |
| I would send to many peolpe | 1 |
| I would share it with my friends and family | 1 |
| I would urge people to go sugarless | 1 |
| Post | 1 |
| Provision of health education | 1 |
| Public awareness | 1 |
| share | 1 |
| Share information with my friends | 1 |
| Share on my social media | 1 |
| Share with friends | 1 |
| tell people not to take a lot of sugar | 1 |
| use the knowledge to make life better for me and those around me | 1 |
emotion_fact1
| x2_1_e_other | Freq |
|---|---|
| Advise my colleague | 1 |
| Confirm if it is true | 1 |
| Dismiss it as a myth | 1 |
| I don’t recommend this | 1 |
| I would send to many people | 1 |
| Let your child get vaccinated | 1 |
| Measles is administered through vaccination in health facility. | 1 |
| Post | 1 |
| report | 1 |
| Report | 1 |
| Report it for misinformation | 1 |
| Say no to it | 1 |
| share | 1 |
| Share information | 1 |
| Share it via other social meadia sites | 1 |
| Share it with others | 1 |
| Shere information | 1 |
| Talk on importance of vaccinating children | 1 |
| vaccines are more important to our children | 1 |
emotion_fact2
| x2_2_e_other | Freq |
|---|---|
| Demonstrate | 1 |
| Health educate people on cancer awareness | 1 |
| I would send to many people | 1 |
| I’ll do research on the topic as in the truth about those pharmaceuticals | 1 |
| Report it for being false | 1 |
| Report the post to Facebook administration | 1 |
| Report to my government | 1 |
| Repost | 1 |
| share | 1 |
| Share information with my friends | 1 |
| Share it to others on social media platforms | 1 |
| Tell people | 1 |
emotion_fact3
| x2_3_e_other | Freq |
|---|---|
| Archive it for future reference | 1 |
| Coconut water is very good and healthy | 1 |
| Consult with the birth attendant | 1 |
| Educate my wife | 1 |
| encourage pregnant women to take coconut water regularly | 1 |
| I have to do some research first and get information | 1 |
| I would share the post on social media | 1 |
| I’ll it to confirm if it’s true | 1 |
| Nutrition education to pregnant women | 1 |
| Pin the post | 1 |
| Post | 1 |
| Post it | 1 |
| Post on Facebook | 1 |
| Share information with my friends | 2 |
| Share widely | 1 |
| Share with pregnant women that I know | 1 |
| Share with women empowerment organisations | 1 |
| Show my pregnant friends | 1 |
| Tag a few pregnant friends | 1 |
| Teach people | 1 |
emotion_fact4
| x2_4_e_other | Freq |
|---|---|
| advice women not apply that vaccine | 1 |
| Consult a medical practitioner | 1 |
| creating awareness through posting in different platted | 1 |
| Do more research about the post | 1 |
| Do research about the post | 1 |
| Find out the side effects of HPV vaccine | 1 |
| Give a remedy | 1 |
| I will collect the information and post it everywhere in the social media for the benefit of the people | 1 |
| Inbox my wife | 1 |
| Others | 1 |
| Report | 1 |
| Research | 1 |
| Research more | 1 |
| Research the internet | 1 |
| Save for future use | 1 |
| Sensitize people | 1 |
| sensitize people about it | 1 |
| Share | 1 |
| Share information with my friends | 1 |
| Share with my colleagues | 1 |
| Study more on HPV vaccine | 1 |
| Tell women to get the vaccine | 1 |
| To help protect more life | 1 |
| To make it aware to the people | 1 |
| women should stop using | 1 |
general_misinfo_fact1
| x2_1_g_other | Freq |
|---|---|
| Advice my close friends to use cannabis only for medical use | 1 |
| Advice those in medical fields to do more research | 1 |
| Awareness | 1 |
| Become a consumer of cannabis | 1 |
| creating awareness through posting in different platforms | 1 |
| Do a follow up research to confirm the details | 1 |
| Do some Research to see if its true | 1 |
| I will share | 1 |
| I will share it with my friends | 1 |
| I would send the post to many people | 1 |
| It has been scientifically proven | 1 |
| Seek doctors advice | 1 |
| share | 1 |
| Share | 3 |
| Share information with my friends | 2 |
| Share it with friends | 1 |
| Share with family and friends | 1 |
| Shere information | 1 |
| Start taking it | 1 |
| Tag someone | 1 |
| Teach people the effect of too much | 1 |
| Tweet | 1 |
general_misinfo_fact2
| x2_2_g_other | Freq |
|---|---|
| Ask doctors and specialists first and research | 1 |
| Comment on the post | 1 |
| Consult Labaratory specialists about the same | 1 |
| Discuss with the author | 1 |
| Do a research on the same | 1 |
| Encourage my friends to use approved test kits. | 1 |
| I would report it for being false | 1 |
| I would send to many people | 1 |
| Post | 1 |
| Prove it practically | 1 |
| Research more about rapid pregnancy tests and brands | 1 |
| Share | 1 |
| Share with medical practitioners | 1 |
| Sharing is caring | 1 |
| Shere it with friends | 1 |
| stop this companies | 1 |
| To guide people on how to plan there family | 1 |
general_misinfo_fact3
| x2_3_g_other | Freq |
|---|---|
| Engage in a conversation to agree disagree on the same | 1 |
| Find out more about autism | 1 |
| Let doctors do research so that we can better our coming generation. | 1 |
| Read more about autism | 1 |
| screenshot it | 1 |
| share | 1 |
| Share | 1 |
| Share it in my timeline | 1 |
| Talk about it | 1 |
| Teach and share | 1 |
general_misinfo_fact4
| x2_4_g_other | Freq |
|---|---|
| Awesome | 1 |
| Create awareness | 1 |
| Do a research about it | 1 |
| Get the facts about it | 1 |
| I will share it both through whatsapp and e-mail | 1 |
| Post | 1 |
| Public awareness | 1 |
| Read more on Epsom salt | 1 |
| Report the post to Facebook administration | 1 |
| Research about it more | 1 |
| Share information with my friends | 2 |
| Share it with friends | 1 |
| share the post widely to help more people | 1 |
| Share with friends | 1 |
| Sharing | 1 |
| Try the bath | 1 |
| Will try doing this | 1 |
factually_true_fact1
| x2_1_f_other | Freq |
|---|---|
| Discuss it verbally | 1 |
| Discuss with friends | 1 |
| Do a follow up research to confirm the myth | 1 |
| I would send to many people | 1 |
| is a fact that peole have to know | 1 |
| Myths are hard to believe they have some truth if you believe in them | 1 |
| Post it again | 1 |
| Research | 1 |
| save it | 1 |
| Share information with my friends | 1 |
| Share it in all my timeline and the people in my contact | 1 |
| Share on my timeline | 1 |
| Share the post on social media | 1 |
| Still sensitize those who are not in social media | 1 |
| Teach people | 1 |
| Tell my friends | 1 |
| Try to find out more abiut this information | 1 |
| When heartbeat stops for seconds then it causes death.so it is a myth | 1 |
factually_true_fact2
| x2_2_f_other | Freq |
|---|---|
| Because of tecnology and food we eat | 1 |
| Copypaste the message on my timeline facebook | 1 |
| Fight againist the use of opioids | 1 |
| hi | 1 |
| more information | 1 |
| Nothing | 1 |
| Post it and educating others | 1 |
| Research | 1 |
| Share it with others | 1 |
| Share the message across different platforms | 1 |
| Tag friends | 1 |
| Teenage counselling. And therapy for adults. Rehabilitation | 1 |
| Tweet about it | 1 |
| Tweet the post | 1 |
| Yes | 1 |
factually_true_fact3
| x2_3_f_other | Freq |
|---|---|
| advice people not to sit closer to tvs | 1 |
| Confirm if it is true | 1 |
| I would send to many people | 1 |
| Inform others | 1 |
| move the tv | 1 |
| People should avoid things that may destroy eyes | 1 |
| people should know this | 1 |
| Read more about this. Do some research | 1 |
| Research more about the information | 1 |
| Share | 1 |
| Share information with my friends | 1 |
| Shere information | 1 |
| Sitting close to a TV damages your eyes | 1 |
| Stay further from the tv henceforth | 1 |
| Tag someone | 1 |
| tell more people | 1 |
| Tell my dad | 1 |
| Tell people to stop sitting near | 1 |
| To train my people | 1 |
| use it for better living | 1 |
| Will condemn it is misleading | 1 |
factually_true_fact4
| x2_4_f_other | Freq |
|---|---|
| Create awareness on what is considered healthy food | 1 |
| Debate about it with collegues | 1 |
| Do more research about the post | 1 |
| Do some research to get a facts on obesity | 1 |
| Educate community | 1 |
| Educate people about causes of obesity | 1 |
| Encourage physical fitness | 1 |
| Enlighten my father | 1 |
| I will advise people to ate health food and do some excise | 1 |
| I would have all my friends read this | 1 |
| I would share the information to people and speak to them | 1 |
| I would share with many people | 1 |
| ill sensitize those who are not insocial media | 1 |
| Nothing | 1 |
| Poverty don’t make ones obesity | 1 |
| Retweet the post | 1 |
| Share | 1 |
| Share information with my friends | 2 |
| Share to other social platforms | 1 |
| Tell people to seek medical attention | 1 |
| Telling others | 1 |
| To physically enlighten people about obesity and it’s dangers | 1 |
| Yes | 1 |
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 3.00 | 4 | 3.727 | 5 | 5 | 1.113 |
| tactics_fact2 | 1 | 3.00 | 3 | 3.109 | 4 | 5 | 0.980 |
| tactics_fact3 | 1 | 2.00 | 4 | 3.449 | 4 | 5 | 1.297 |
| tactics_fact4 | 1 | 2.00 | 3 | 3.097 | 4 | 5 | 1.363 |
| emotion_fact1 | 1 | 1.00 | 3 | 2.611 | 4 | 5 | 1.413 |
| emotion_fact2 | 1 | 2.00 | 3 | 2.915 | 4 | 5 | 1.229 |
| emotion_fact3 | 1 | 3.00 | 4 | 3.624 | 4 | 5 | 1.061 |
| emotion_fact4 | 1 | 2.00 | 3 | 3.123 | 4 | 5 | 1.255 |
| general_misinfo_fact1 | 1 | 2.00 | 3 | 3.092 | 4 | 5 | 1.321 |
| general_misinfo_fact2 | 1 | 2.25 | 3 | 3.108 | 4 | 5 | 1.151 |
| general_misinfo_fact3 | 1 | 3.00 | 3 | 3.175 | 4 | 5 | 1.058 |
| general_misinfo_fact4 | 1 | 2.00 | 3 | 3.019 | 4 | 5 | 1.256 |
| factually_true_fact1 | 1 | 3.00 | 4 | 3.748 | 5 | 5 | 1.107 |
| factually_true_fact2 | 1 | 3.00 | 3 | 3.375 | 4 | 5 | 1.006 |
| factually_true_fact3 | 1 | 3.00 | 4 | 3.642 | 5 | 5 | 1.206 |
| factually_true_fact4 | 1 | 3.00 | 4 | 3.577 | 5 | 5 | 1.267 |
Identifying Potential Issues
Check for variation of answers in manipulativeness / accuracy question
We suspect a similar situation can happen as well to the manipulativeness / accuracy question, so we check each respondent to see if they have consecutive answers that are the same, suggesting they are only trying to get through the survey by choosing the same answer.
Note: We did not require responses for manipulativeness / accuracy, so imputing is done by taking the respondents’ previous answer.
“Clean” analysis (after filtering out worrisome respondents)
We filter out respondents with conflict answers in >= 6 posts and respondents giving the same answer with consecutive length >= 6.
(S1) List of Actions to Take
(S1) Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2 | 3 | 2.736 | 4.0 | 5 | 1.077 |
| non-manipulative_fact2 | 1 | 2 | 3 | 2.930 | 4.0 | 5 | 1.298 |
| non-manipulative_fact3 | 1 | 2 | 3 | 2.873 | 4.0 | 5 | 1.362 |
| non-manipulative_fact4 | 1 | 2 | 3 | 3.000 | 4.0 | 5 | 1.354 |
| non-manipulative_fact5 | 1 | 2 | 2 | 2.818 | 4.0 | 5 | 1.529 |
| non-manipulative_fact6 | 1 | 2 | 3 | 2.846 | 4.0 | 5 | 1.227 |
| non-manipulative_fact7 | 1 | 2 | 3 | 2.755 | 4.0 | 5 | 1.128 |
| non-manipulative_fact8 | 1 | 2 | 3 | 3.051 | 4.5 | 5 | 1.450 |
| manipulative_fact1 | 1 | 2 | 4 | 3.268 | 4.0 | 5 | 1.152 |
| manipulative_fact2 | 1 | 2 | 4 | 3.250 | 4.0 | 5 | 1.229 |
| manipulative_fact3 | 1 | 2 | 2 | 2.766 | 4.0 | 5 | 1.402 |
| manipulative_fact4 | 1 | 2 | 3 | 3.000 | 4.0 | 5 | 1.272 |
| manipulative_fact5 | 1 | 2 | 3 | 3.143 | 4.0 | 5 | 1.372 |
| manipulative_fact6 | 1 | 2 | 2 | 2.340 | 3.0 | 5 | 1.091 |
| manipulative_fact7 | 1 | 2 | 3 | 2.837 | 4.0 | 5 | 1.326 |
| manipulative_fact8 | 1 | 2 | 4 | 3.106 | 4.0 | 5 | 1.433 |
(S2) List of Actions to Take
(S2) Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 3 | 4 | 3.615 | 4.00 | 5 | 1.113 |
| tactics_fact2 | 1 | 3 | 3 | 3.083 | 4.00 | 5 | 0.970 |
| tactics_fact3 | 1 | 2 | 4 | 3.364 | 4.00 | 5 | 1.297 |
| tactics_fact4 | 1 | 2 | 3 | 3.049 | 4.00 | 5 | 1.330 |
| emotion_fact1 | 1 | 1 | 2 | 2.424 | 3.00 | 5 | 1.344 |
| emotion_fact2 | 1 | 2 | 3 | 2.837 | 4.00 | 5 | 1.224 |
| emotion_fact3 | 1 | 3 | 4 | 3.615 | 4.00 | 5 | 0.999 |
| emotion_fact4 | 1 | 2 | 3 | 3.076 | 4.00 | 5 | 1.207 |
| general_misinfo_fact1 | 1 | 2 | 3 | 3.067 | 4.00 | 5 | 1.279 |
| general_misinfo_fact2 | 1 | 2 | 3 | 3.000 | 4.00 | 5 | 1.114 |
| general_misinfo_fact3 | 1 | 3 | 3 | 3.139 | 4.00 | 5 | 1.049 |
| general_misinfo_fact4 | 1 | 2 | 3 | 3.000 | 4.00 | 5 | 1.202 |
| factually_true_fact1 | 1 | 3 | 4 | 3.725 | 4.75 | 5 | 1.091 |
| factually_true_fact2 | 1 | 3 | 3 | 3.343 | 4.00 | 5 | 0.949 |
| factually_true_fact3 | 1 | 3 | 4 | 3.582 | 4.00 | 5 | 1.183 |
| factually_true_fact4 | 1 | 3 | 4 | 3.659 | 5.00 | 5 | 1.201 |
Correlation Between Actions and Manipulativeness
This section is used to understand whether actions in the multi-select questions have correlation with how they answer manipulation question and accuracy question.
Section 1
Section 2
Codebook
This section is used to generate demographics codebook.
Consistency Check
Balance of Treatment
We did a treatment randomization using Qualtrics as a test step to see if randomization step can be done on our side (i.e. no actual treatment was assigned to respondents)
Connecting Sharing Behavior with Sharing Intention
## [1] "correlation bewteen intention to share and self-claimed sharing behavior: 0.26"##
## Call:
## lm(formula = share_count ~ share_behavior)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.2779 -1.6303 0.4268 2.4268 4.3697
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.9827 0.5149 13.560 < 2e-16 ***
## share_behavior 0.6476 0.1628 3.979 9.45e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.778 on 216 degrees of freedom
## Multiple R-squared: 0.06829, Adjusted R-squared: 0.06398
## F-statistic: 15.83 on 1 and 216 DF, p-value: 9.452e-05
Looking at Distribution of Sharing from previous lab experiment
Setting: two questions were asked
- Would you like to share privately? (Yes / No)
- Would you like to share publicly (Yes / No)
Ads Analysis
Summary
Note: Performance of individual ad is not of primary purpose at this current stage. This section is only used for basic analytics.
- We used $73.67 for ads in total to recruit n = 257 completed responses (~0.287 USD per survey completion)
- Note that we used the same campaign as pilot V2 to avoid attracting people who have already seen the ads
- this is tripled the cost of pilot V2, where cost per survey completion was 0.092 USD; however, this is expected
- Click through rate is at 1.2% (# clicks = 402) with cost per click being $0.081
- Only dropoff place we can measure is at the consent level, where 257 (63.9%) people the consent questions and all complete till the end of the survey
- Facebook prioritized ads with Image 1 and Image 4 [See more in Ads Analysis Section]
Note Facebook algorithm is not set up to track conversion details with Qualtrics; so we have no idea which ads work the best in terms of conversion. We instead use link clicks as a proxy.
Ads
We created a set of 8 ads (varying across 2 types of headlines and 4 types of images)
- Headline 1 (Airtime + Better): Let’s make social media better together! Take a short survey, earn mobile airtime!:moneybag:
- Headline 2 (Airtime only): Take a short survey, earn mobile airtime!:moneybag:
Funnel Analysis
Individual Ad Analysis