Outcome Measurement Survey V3 (with Pledge)
February 14, 2022
Project Overview and Research Question
Given the current technologies, misinformation spreads around the globe at a speed faster than ever before. What we see on the internet not only impacts our mindset and changes our beliefs but also could be transformed into offline personal and societal consequences. This could translate to actions that are physically or mentally harmful to people; some could even be deadly (Mu ̈ller and Schwarz, 2019; Chan et al., 2016). Once misinformation starts to propagate, we face the risk of no longer being able to identify what is the truth. More importantly, misinformation sticks in people’s minds. Numerous psychological experiments have shown that erasing misinformation is challenging even in controlled lab settings (Lewandowsky et al., 2012). Thus, it is important to look into actions and methods that prevent misinformation from entering in people’s minds in the first place, where we “vaccinate” people against misinformation through inoculation. Inoculation methods present weakened versions of the misinformation messages beforehand to build resistance and immunity to false narratives (McGuire et al. 1961). Through inoculation, we are able to prepare ourselves to spot and deal with misinformation targeting our communities and mitigate the potential damage misinformation can cause.
In this project, we would like to “vaccinate” people against misinformation through inoculation, where we develop a text-message-based course to deliver effective treatments to participants.
We want to test out the following research questions :
- Can a text message-based course “inoculate” users against misinformation?
- Hypothesis 1: Courses delivered in text message format treat participants effectively in spotting misinformation (i.e. at least one of the three treatments works, which requires a much smaller sample size than all treatments work)
- Hypothesis 2: Participants who receive analytical treatment should do better in spotting misinformation using analytical techniques. Participants who receive emotional treatment should do better in spotting misinformation using emotional techniques. Participants who receive combo treatment (i.e. mixture of analytical and emotional) should do better in both.
- How can we improve the efficacy and scalability of the course?
- Is the course differentially effective for different types of users (based on age, political affiliations, etc.)?
Current Pre-Test Scheme
We are interested in investigating what outcome measures we should use in the actual experiment. Thus, we are pre-testing using Qualtrics survey to find questions (i.e. outcomes) and associated display formats that would generate heterogeneous answers.
In this analysis script version, we used Facebook ads for recruitment of Kenyan respondents who are 18 years or older. The ads directly connect to our Qualtric survey. Note: The performance of individual ad is not of interest to this project at this current stage. We are mostly interested in getting as many participants as possible during the pilot stage
Survey Structure
- Consent
- Pledge: “While there are no right or wrong answers, we will evaluate your participation and discard low quality responses, so think about each question carefully and answer truthfully. Please, confirm you will do so by choosing the right option below.”
- I will pay attention throughout the survey and answer the questions after thinking through them carefully.
- I will NOT pay attention throughout the survey NOR answer the questions after thinking through them carefully.
- Instruction: “You will see a series of social media posts in the following screens. Please, read each one of them carefully and answer the questions at the end of each screen.”
- Section 1 (4 posts)
- Midway Message: “you are about halfway through, keep going!”
- Section 2 (8 posts)
- Demographics
Note:
- respondents do not know a clear cut difference between section 1 and section 2; the only difference they see are the different questions
Summary of Learning
Demographics Description
- n = 257
- Median age is 27 with 1st and 3rd quartiles being 24 and 32 (median age in Kenya is about 20 years old)
- Around 30% respondents are female and 68% are male (2% chose other) [~ same as pilot v2]
- around 69% respondents have some college education
- 36% unemployed and looking for work [~ same as pilot v2]
- mostly right leaning (median 64 on a scale of 100, where 100 means extremely right)
- 38% live in mostly rural areas; 62% mostly urban [~ same as pilot v2]
- most describe themselves as having locus of control (median 7 on a scale of 10)
- most use social media around 3 (1st quartile) to 7 (3rd quartile) hours on a daily basis, with median being 5 hours [~ same as pilot v2]
- 68% claim sometimes share social media posts, 12% claim rarely or never share, 20% claim always share
Ads
- We used $73.67 for ads in total to recruit n = 257 completed responses (~0.287 USD per survey completion)
- Note that we used the same campaign as pilot V2 to avoid attracting people who have already seen the ads
- this is tripled the cost of pilot V2, where cost per survey completion was 0.092 USD; however, this is expected
- Click through rate is at 1.2% (# clicks = 402) with cost per click being $0.081
- Only dropoff place we can measure is at the consent level, where 257 (63.9%) people the consent questions and all complete till the end of the survey
- Facebook prioritized ads with Image 1 and Image 4 [See more in Ads Analysis Section]
Note Facebook algorithm is not set up to track conversion details with Qualtrics; so we have no idea which ads work the best in terms of conversion. We instead use link clicks as a proxy.
Outcome
Interesting Yet Unsure Finding
- We see a similar overwhelming rate of intention to share privately, publicly, or both.
- We see an overwhelming rate of willingness to dicuss publicly about the post (~60%), followed by discussing privately (~25%)
Issue - around 21% of respondents have selected the same answer in manipulativeness / accuracy question on six or more consecutive questions
Learning
- After cleaning out conflicting responses (>= 6) and consecutively same answer responses (>= 6 posts) observations, we find that the distribution of manipulativeness / accuracy seems to have heterogeneity that we want, suggesting this outcome measure works.
- Formatting / Presentation of survey is essential to avoid people selecting by convenience rather than actual choice.
- Some individual posts are showing a skewed distribution. We should look at these distribution and posts carefully and discuss whether we need to swap them out or not.
Yet to be Done
- Find out correlation between intention to share and sharing behavior
- Investigate skewed distribution posts
Data
Load Packages
Read Data
Data Cleaning
Variable Encoding
Data Analysis
Section 1
In section 1, we aim to test whether respondents can distinguish the level of manipulation of the post. We have a pool of 4 manipulative posts as well as their non-manipulative counterparts (i.e. 4 facts). We show two manipulative posts and two non-manipulative posts (with no overlaps of facts) and asked the following two questions for each post:
How do you feel about sharing this post on social media?
- [share publicly] I would share it publicly on timeline or feeds
- [share privately] I would share it privately with other family member / friend
- [share both] I would share both publicly and privately
- [no share]I would not share the post.
Would you do any of the following? Please select what you would be most likely to do.
- [discuss privately] I would discuss it privately with the poster
- [discuss publicly] I would discuss it publicly (for example: commenting on the post)
- [report] I would report the post
- [other] I would do something else
- [ignore] I would not do any of the above
Do you agree or disagree with the following statement about the post? This post is manipulative.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 1
Sample Post Section 1
List of Actions to Take
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2 | 3 | 2.620 | 3 | 5 | 1.257 |
| non-manipulative_fact2 | 1 | 2 | 3 | 2.861 | 4 | 5 | 1.365 |
| non-manipulative_fact3 | 1 | 2 | 3 | 2.899 | 4 | 5 | 1.318 |
| non-manipulative_fact4 | 1 | 2 | 3 | 2.841 | 4 | 5 | 1.219 |
| manipulative_fact1 | 1 | 2 | 3 | 3.129 | 4 | 5 | 1.280 |
| manipulative_fact2 | 1 | 2 | 3 | 2.829 | 4 | 5 | 1.299 |
| manipulative_fact3 | 1 | 2 | 3 | 2.939 | 4 | 5 | 1.416 |
| manipulative_fact4 | 1 | 2 | 3 | 2.919 | 4 | 5 | 1.349 |
Section 2
In section 2, we aim to test whether respondents can distinguish whether a post is misinformation or not. We have a pool of 4 misinformation posts with emotion techniques, 4 misinformation posts with tactics, 4 general misinformation posts, 4 factually true posts. We show two posts from each type (randomized), and asked them the following questions for each:
Note that the facts in all of the posts do not overlap (i.e. we have 16 facts / topics)
How do you feel about sharing this post on social media?
- [share publicly] I would share it publicly on timeline or feeds
- [share privately] I would share it privately with other family member / friend
- [share both] I would share both publicly and privately
- [no share]I would not share the post.
Would you do any of the following? Please select what you would be most likely to do.
- [discuss privately] I would discuss it privately with the poster
- [discuss publicly] I would discuss it publicly (for example: commenting on the post)
- [report] I would report the post
- [other] I would do something else
- [ignore] I would not do any of the above
Do you agree or disagree with the following statement about the post? The information presented in this post is accurate.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 2
Sample Post Section 2
List of Actions to Take
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 3 | 4 | 3.601 | 4 | 5 | 1.148 |
| tactics_fact2 | 1 | 3 | 3 | 3.250 | 4 | 5 | 1.213 |
| tactics_fact3 | 1 | 3 | 4 | 3.405 | 4 | 5 | 1.276 |
| tactics_fact4 | 1 | 2 | 4 | 3.346 | 4 | 5 | 1.259 |
| emotion_fact1 | 1 | 1 | 3 | 2.750 | 4 | 5 | 1.410 |
| emotion_fact2 | 1 | 2 | 3 | 3.142 | 4 | 5 | 1.298 |
| emotion_fact3 | 1 | 3 | 4 | 3.458 | 4 | 5 | 1.111 |
| emotion_fact4 | 1 | 2 | 3 | 3.208 | 4 | 5 | 1.223 |
| general_misinfo_fact1 | 1 | 3 | 3 | 3.323 | 4 | 5 | 1.215 |
| general_misinfo_fact2 | 1 | 2 | 3 | 3.228 | 4 | 5 | 1.214 |
| general_misinfo_fact3 | 1 | 3 | 3 | 3.244 | 4 | 5 | 1.166 |
| general_misinfo_fact4 | 1 | 3 | 3 | 3.239 | 4 | 5 | 1.178 |
| factually_true_fact1 | 1 | 3 | 4 | 3.620 | 4 | 5 | 1.105 |
| factually_true_fact2 | 1 | 3 | 4 | 3.467 | 4 | 5 | 1.136 |
| factually_true_fact3 | 1 | 3 | 4 | 3.756 | 5 | 5 | 1.185 |
| factually_true_fact4 | 1 | 3 | 4 | 3.521 | 4 | 5 | 1.180 |
Identifying Potential Issues
Percentage of seemingly conflicting answer
Here we define conflicting answer when respondents both choose share (either privately or publicly) and report for the same question as above.
Check for variation of answers in manipulativeness / accuracy question
We suspect a similar situation can happen as well to the manipulativeness / accuracy question, so we check each respondent to see if they have consecutive answers that are the same, suggesting they are only trying to get through the survey by choosing the same answer.
Note: We did not require responses for manipulativeness / accuracy, so imputing is done by taking the respondents’ previous answer.
“Clean” analysis (after filtering out worrisome respondents)
We filter out respondents with conflict answers in >= 6 posts and respondents giving the same answer with consecutive length >= 6.
(S1) List of Actions to Take
## $y
## [1] "count"
##
## attr(,"class")
## [1] "labels"
## $y
## [1] "count"
##
## attr(,"class")
## [1] "labels"(S1) Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 1 | 3 | 2.534 | 3 | 5 | 1.203 |
| non-manipulative_fact2 | 1 | 2 | 3 | 2.890 | 4 | 5 | 1.355 |
| non-manipulative_fact3 | 1 | 2 | 3 | 2.845 | 4 | 5 | 1.295 |
| non-manipulative_fact4 | 1 | 2 | 3 | 2.825 | 4 | 5 | 1.146 |
| manipulative_fact1 | 1 | 2 | 3 | 3.201 | 4 | 5 | 1.194 |
| manipulative_fact2 | 1 | 2 | 3 | 2.781 | 4 | 5 | 1.285 |
| manipulative_fact3 | 1 | 2 | 3 | 2.929 | 4 | 5 | 1.428 |
| manipulative_fact4 | 1 | 2 | 3 | 2.987 | 4 | 5 | 1.343 |
(S2) List of Actions to Take
## $y
## [1] "count"
##
## attr(,"class")
## [1] "labels"
## $y
## [1] "count"
##
## attr(,"class")
## [1] "labels"(S2) Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 3.0 | 4 | 3.543 | 4 | 5 | 1.130 |
| tactics_fact2 | 1 | 3.0 | 3 | 3.194 | 4 | 5 | 1.190 |
| tactics_fact3 | 1 | 3.0 | 3 | 3.395 | 4 | 5 | 1.231 |
| tactics_fact4 | 1 | 2.0 | 3 | 3.333 | 4 | 5 | 1.234 |
| emotion_fact1 | 1 | 1.0 | 2 | 2.550 | 4 | 5 | 1.384 |
| emotion_fact2 | 1 | 2.0 | 3 | 2.962 | 4 | 5 | 1.272 |
| emotion_fact3 | 1 | 3.0 | 3 | 3.378 | 4 | 5 | 1.106 |
| emotion_fact4 | 1 | 2.0 | 3 | 3.043 | 4 | 5 | 1.218 |
| general_misinfo_fact1 | 1 | 2.5 | 3 | 3.259 | 4 | 5 | 1.155 |
| general_misinfo_fact2 | 1 | 2.0 | 3 | 3.150 | 4 | 5 | 1.202 |
| general_misinfo_fact3 | 1 | 2.0 | 3 | 3.156 | 4 | 5 | 1.133 |
| general_misinfo_fact4 | 1 | 3.0 | 3 | 3.113 | 4 | 5 | 1.123 |
| factually_true_fact1 | 1 | 3.0 | 4 | 3.703 | 4 | 5 | 0.987 |
| factually_true_fact2 | 1 | 3.0 | 3 | 3.423 | 4 | 5 | 1.104 |
| factually_true_fact3 | 1 | 3.0 | 4 | 3.776 | 5 | 5 | 1.169 |
| factually_true_fact4 | 1 | 3.0 | 4 | 3.477 | 4 | 5 | 1.193 |
Correlation Between Actions and Manipulativeness
This section is used to understand whether actions in the multi-select questions have correlation with how they answer manipulation question and accuracy question.
Section 1
Section 2
Codebook
This section is used to generate demographics codebook.
Consistency Check
Balance of Treatment
We did a treatment randomization using Qualtrics as a test step to see if randomization step can be done on our side (i.e. no actual treatment was assigned to respondents)
Connecting Sharing Behavior with Sharing Intention
## [1] "correlation bewteen intention to share and self-claimed sharing behavior: 0.274317108209975"##
## Call:
## lm(formula = share_count ~ share_behavior)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.7961 -1.4997 0.9075 2.2039 4.7966
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.2034 0.5129 14.045 <2e-16 ***
## share_behavior 1.2964 0.2356 5.502 7e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.583 on 372 degrees of freedom
## Multiple R-squared: 0.07525, Adjusted R-squared: 0.07276
## F-statistic: 30.27 on 1 and 372 DF, p-value: 7.005e-08
Looking at Distribution of Sharing from previous lab experiment
Note that this is different from our setting
Our setting: respondents choose from one of four options: no share, share privately, share publicly, share both
Previous lab setting: two questions were asked
- Would you like to share privately? (Yes / No)
- Would you like to share publicly (Yes / No)
This analysis mainly help us understand the percentage of people who are willing to share
Ads Analysis
Summary
Note: Performance of individual ad is not of primary purpose at this current stage. This section is only used for basic analytics.
- We used $73.67 for ads in total to recruit n = 257 completed responses (~0.287 USD per survey completion)
- Note that we used the same campaign as pilot V2 to avoid attracting people who have already seen the ads
- this is tripled the cost of pilot V2, where cost per survey completion was 0.092 USD; however, this is expected
- Click through rate is at 1.2% (# clicks = 402) with cost per click being $0.081
- Only dropoff place we can measure is at the consent level, where 257 (63.9%) people the consent questions and all complete till the end of the survey
- Facebook prioritized ads with Image 1 and Image 4 [See more in Ads Analysis Section]
Note Facebook algorithm is not set up to track conversion details with Qualtrics; so we have no idea which ads work the best in terms of conversion. We instead use link clicks as a proxy.
Ads
We created a set of 8 ads (varying across 2 types of headlines and 4 types of images)
- Headline 1 (Airtime + Better): Let’s make social media better together! Take a short survey, earn mobile airtime!:moneybag:
- Headline 2 (Airtime only): Take a short survey, earn mobile airtime!:moneybag:
Funnel Analysis
Individual Ad Analysis