Outcome Measurement Survey V2
March 16, 2022
Project Overview and Research Question
Given the current technologies, misinformation spreads around the globe at a speed faster than ever before. What we see on the internet not only impacts our mindset and changes our beliefs but also could be transformed into offline personal and societal consequences. This could translate to actions that are physically or mentally harmful to people; some could even be deadly (Mu ̈ller and Schwarz, 2019; Chan et al., 2016). Once misinformation starts to propagate, we face the risk of no longer being able to identify what is the truth. More importantly, misinformation sticks in people’s minds. Numerous psychological experiments have shown that erasing misinformation is challenging even in controlled lab settings (Lewandowsky et al., 2012). Thus, it is important to look into actions and methods that prevent misinformation from entering in people’s minds in the first place, where we “vaccinate” people against misinformation through inoculation. Inoculation methods present weakened versions of the misinformation messages beforehand to build resistance and immunity to false narratives (McGuire et al. 1961). Through inoculation, we are able to prepare ourselves to spot and deal with misinformation targeting our communities and mitigate the potential damage misinformation can cause.
In this project, we would like to “vaccinate” people against misinformation through inoculation, where we develop a text-message-based course to deliver effective treatments to participants.
We want to test out the following research questions :
- Can a text message-based course “inoculate” users against misinformation?
- Hypothesis 1: Courses delivered in text message format treat participants effectively in spotting misinformation (i.e. at least one of the three treatments works, which requires a much smaller sample size than all treatments work)
- Hypothesis 2: Participants who receive analytical treatment should do better in spotting misinformation using analytical techniques. Participants who receive emotional treatment should do better in spotting misinformation using emotional techniques. Participants who receive combo treatment (i.e. mixture of analytical and emotional) should do better in both.
- How can we improve the efficacy and scalability of the course?
- Is the course differentially effective for different types of users (based on age, political affiliations, etc.)?
Current Pre-Test Scheme
We are interested in investigating what outcome measures we should use in the actual experiment. Thus, we are pre-testing using Qualtrics survey to find questions (i.e. outcomes) and associated display formats that would generate heterogeneous answers.
In this analysis script version, we used Facebook ads for recruitment of Kenyan respondents who are 18 years or older. The ads directly connect to our Qualtric survey.
Survey Structure
- Consent
- Instruction: “You will see a series of social media posts in the following screens. Please, read each one of them carefully and answer the questions at the end of each screen.”
- Section 1 (4 posts)
- Midway Message
- Section 2 (8 posts)
- Demographics
Note:
- midway message: “you are about halfway through, keep going!”
- respondents do not know a clear cut difference between section 1 and section 2; the only difference they see are the different questions
Summary of Learning
Demographics Description
- n = 278
- Median age is 25 with 1st and 3rd quartiles being 23 and 28 (median age in Kenya is about 20 years old)
- Around 32% respondents are female and 68% are male
- around 62% respondents have some college education
- 36% unemployed and looking for work
- mostly right leaning (median 67 on a scale of 100, where 100 means extremely right)
- 38% live in mostly rural areas; 62% mostly urban
- most describe themselves as having locus of control (median 7.5 on a scale of 10)
- most use social media around 4 (1st quartile) to 7 (3rd quartile) hours on a daily basis, with median being 5 hours
- 75% claim sometimes share social media posts, 15% claim rarely or never share, 10% claim always share
Ads
- We used $25.47 for ads in total to recruit n = 278 completed responses
- However, Facebook algorithm is not set up to track conversion details with Qualtrics; so we have no idea which ads work the best in terms of conversion. We instead use link clicks as a proxy.
- Click through rate is at 2.3% (# clicks = 315) with cost per click being $0.081
- Only dropoff place we can measure is at the consent level, where 278 (88.3%) people the consent questions and all complete till the end of the survey
- Facebook prioritized ads that contain the message “Let’s make social media better together! Take a short survey, earn mobile airtime!:moneybag:”, though the cost per click for these ads are higher, suggesting a mismatch between objective and prioritization
Outcome
Issue
- We see an overwhelming rate of intention to share privately, publicly, and disucss and a pretty high rate of report, which does not suggest normal behaviors.
- On average around 10% of people who choose all of the options (“share private”, “share public”, “report”, “discuss”)
- On average around 15% of people who choose conflicting options (“share” and “report”)
- around 13% of respondents have selected the same answer in manipulativeness / accuracy question on six or more consecutive questions
Learning
- After cleaning out all yes responses (>= 6 posts), conflicting responses (>= 6), consecutively same answer responses (>= 6 posts) observations, we find that the distribution of manipulativeness / accuracy seems to have heterogeneity that we want, suggesting this outcome measure works.
- Formatting / Presentation of survey is essential to avoid people selecting by convenience rather than actual choice.
Proposed Future Changes
- Change formatting of the question might help avoiding respondents selecting the same answers
- Require response for manipulativeness and accuracy questions
Data
Load Packages
Read Data
Data Cleaning
Variable Encoding
Data Analysis
Section 1
In section 1, we aim to test whether respondents can distinguish the level of manipulation of the post. We have a pool of 4 manipulative posts as well as their non-manipulative counterparts (i.e. 4 facts). We show two manipulative posts and two non-manipulative posts (with no overlaps of facts) and asked the following two questions for each post:
If you see a friend / family member sharing this post on social media, would you take any of these actions regarding it?
- [share publicly] share it publicly on social media feeds (options: yes / no)
- [share privately] share it privately with family members / friend (options: yes / no)
- [discuss] discuss the content with the person who posted it (options: yes / no)
- [report] report the post (options: yes / no)
Do you agree or disagree with the following statement about the post? This post is manipulative.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 1
Sample Post Section 1
List of Actions to Take
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2 | 3 | 2.683 | 3.5 | 5 | 1.090 |
| non-manipulative_fact2 | 1 | 2 | 3 | 2.776 | 4.0 | 5 | 1.156 |
| non-manipulative_fact3 | 1 | 2 | 3 | 2.702 | 3.0 | 5 | 1.072 |
| non-manipulative_fact4 | 1 | 2 | 3 | 2.938 | 4.0 | 5 | 1.121 |
| manipulative_fact1 | 1 | 2 | 3 | 3.058 | 4.0 | 5 | 1.164 |
| manipulative_fact2 | 1 | 2 | 3 | 3.033 | 4.0 | 5 | 1.272 |
| manipulative_fact3 | 1 | 2 | 3 | 2.842 | 4.0 | 5 | 1.263 |
| manipulative_fact4 | 1 | 2 | 3 | 2.973 | 4.0 | 5 | 1.145 |
Section 2
In section 2, we aim to test whether respondents can distinguish whether a post is misinformation or not. We have a pool of 4 misinformation posts with emotion techniques, 4 misinformation posts with tactics, 4 general misinformation posts, 4 factually true posts. We show two posts from each type (randomized), and asked them the following questions for each:
Note that the facts in all of the posts do not overlap (i.e. we have 16 facts / topics)
If you see a friend / family member sharing this post on social media, would you take any of these actions regarding it?
- [share publicly] share it publicly on social media feeds (options: yes / no)
- [share privately] share it privately with family members / friend (options: yes / no)
- [discuss] discuss the content with the person who posted it (options: yes / no)
- [report] report the post (options: yes / no)
Do you agree or disagree with the following statement about the post? The information presented in this post is accurate.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 2
Sample Post Section 2
List of Actions to Take
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 3 | 3 | 3.345 | 4 | 5 | 1.088 |
| tactics_fact2 | 1 | 2 | 3 | 2.909 | 4 | 5 | 1.074 |
| tactics_fact3 | 1 | 2 | 3 | 3.234 | 4 | 5 | 1.174 |
| tactics_fact4 | 1 | 2 | 3 | 3.000 | 4 | 5 | 1.230 |
| emotion_fact1 | 1 | 1 | 2 | 2.353 | 3 | 5 | 1.143 |
| emotion_fact2 | 1 | 2 | 3 | 2.904 | 4 | 5 | 1.067 |
| emotion_fact3 | 1 | 2 | 3 | 3.193 | 4 | 5 | 1.143 |
| emotion_fact4 | 1 | 2 | 3 | 2.891 | 4 | 5 | 1.159 |
| general_misinfo_fact1 | 1 | 2 | 3 | 2.863 | 4 | 5 | 1.112 |
| general_misinfo_fact2 | 1 | 2 | 3 | 2.964 | 4 | 5 | 1.138 |
| general_misinfo_fact3 | 1 | 2 | 3 | 2.914 | 4 | 5 | 1.083 |
| general_misinfo_fact4 | 1 | 2 | 3 | 2.961 | 4 | 5 | 1.011 |
| factually_true_fact1 | 1 | 2 | 3 | 3.232 | 4 | 5 | 1.122 |
| factually_true_fact2 | 1 | 2 | 3 | 3.073 | 4 | 5 | 1.075 |
| factually_true_fact3 | 1 | 2 | 3 | 3.164 | 4 | 5 | 1.196 |
| factually_true_fact4 | 1 | 3 | 3 | 3.300 | 4 | 5 | 1.083 |
Identifying Potential Issues
Percentage of people selecting same answers on questions
Since the following question is displayed in a matrix format, we want to check percentage of people who just click “all yes” or “all no” as this pattern of response might suggest people just want to get through the survey quickly.
Note: this analysis is done by aggregating on the order of the post respondents see level instead by individual post
If you see a friend / family member sharing this post on social media, would you take any of these actions regarding it?
- [share publicly] share it publicly on social media feeds (options: yes / no)
- [share privately] share it privately with family members / friend (options: yes / no)
- [discuss] discuss the content with the person who posted it (options: yes / no)
- [report] report the post (options: yes / no)
Percentage of seemingly conflicting answer
Here we define conflicting answer when respondents both choose share (either privately or publicly) and report for the same question as above.
Check for variation of answers in manipulativeness / accuracy question
We suspect a similar situation can happen as well to the manipulativeness / accuracy question, so we check each respondent to see if they have consecutive answers that are the same, suggesting they are only trying to get through the survey by choosing the same answer.
Note: We did not require responses for manipulativeness / accuracy, so imputing is done by taking the respondents’ previous answer.
“Clean” analysis (after filtering out worrisome respondents)
We filter out respondents with all yes answers in >= 6 posts, respondents with conflict answers in >= 6 posts, and respondents with consecutive length >= 6.
(S1) List of Actions to Take
(S1) Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2 | 3 | 2.645 | 3 | 5 | 1.057 |
| non-manipulative_fact2 | 1 | 2 | 3 | 2.795 | 4 | 5 | 1.193 |
| non-manipulative_fact3 | 1 | 2 | 3 | 2.657 | 3 | 5 | 1.080 |
| non-manipulative_fact4 | 1 | 2 | 3 | 2.865 | 4 | 5 | 1.062 |
| manipulative_fact1 | 1 | 2 | 3 | 3.075 | 4 | 5 | 1.119 |
| manipulative_fact2 | 1 | 2 | 3 | 2.948 | 4 | 5 | 1.228 |
| manipulative_fact3 | 1 | 2 | 3 | 2.789 | 4 | 5 | 1.215 |
| manipulative_fact4 | 1 | 2 | 3 | 3.000 | 4 | 5 | 1.133 |
(S2) List of Actions to Take
(S2) Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 3 | 3 | 3.290 | 4 | 5 | 1.116 |
| tactics_fact2 | 1 | 2 | 3 | 2.807 | 3 | 5 | 1.076 |
| tactics_fact3 | 1 | 2 | 3 | 3.170 | 4 | 5 | 1.181 |
| tactics_fact4 | 1 | 2 | 3 | 2.919 | 4 | 5 | 1.266 |
| emotion_fact1 | 1 | 1 | 2 | 2.316 | 3 | 5 | 1.123 |
| emotion_fact2 | 1 | 2 | 3 | 2.764 | 3 | 5 | 1.029 |
| emotion_fact3 | 1 | 2 | 3 | 3.156 | 4 | 5 | 1.124 |
| emotion_fact4 | 1 | 2 | 3 | 2.817 | 4 | 5 | 1.182 |
| general_misinfo_fact1 | 1 | 2 | 3 | 2.766 | 3 | 5 | 1.070 |
| general_misinfo_fact2 | 1 | 2 | 3 | 2.952 | 4 | 5 | 1.155 |
| general_misinfo_fact3 | 1 | 2 | 3 | 2.808 | 3 | 5 | 1.025 |
| general_misinfo_fact4 | 1 | 2 | 3 | 2.925 | 4 | 5 | 0.963 |
| factually_true_fact1 | 1 | 2 | 3 | 3.167 | 4 | 5 | 1.106 |
| factually_true_fact2 | 1 | 2 | 3 | 3.053 | 4 | 5 | 1.029 |
| factually_true_fact3 | 1 | 2 | 3 | 3.097 | 4 | 5 | 1.209 |
| factually_true_fact4 | 1 | 3 | 3 | 3.253 | 4 | 5 | 1.072 |
Correlation Between Actions and Manipulativeness
This section is used to understand whether actions in the multi-select questions have correlation with how they answer manipulation question and accuracy question.
Section 1
Section 2
Codebook
This section is used to generate demographics codebook.
Consistency Check
Balance of Treatment
We did a treatment randomization using Qualtrics as a test step to see if randomization step can be done on our side (i.e. no actual treatment was assigned to respondents)
Ads Analysis
Funnel Analysis
Individual Ad Analysis
We created a set of 8 ads (varying across 2 types of headlines and 4 types of images)
- Headline 1 (Airtime + Better): Let’s make social media better together! Take a short survey, earn mobile airtime!:moneybag:
- Headline 2 (Airtime only): Take a short survey, earn mobile airtime!:moneybag:
