Outcome Measurement Survey V5 (Cornell Experiment)
March 16, 2022
Project Overview and Research Question
Given the current technologies, misinformation spreads around the globe at a speed faster than ever before. What we see on the internet not only impacts our mindset and changes our beliefs but also could be transformed into offline personal and societal consequences. This could translate to actions that are physically or mentally harmful to people; some could even be deadly (Mu ̈ller and Schwarz, 2019; Chan et al., 2016). Once misinformation starts to propagate, we face the risk of no longer being able to identify what is the truth. More importantly, misinformation sticks in people’s minds. Numerous psychological experiments have shown that erasing misinformation is challenging even in controlled lab settings (Lewandowsky et al., 2012). Thus, it is important to look into actions and methods that prevent misinformation from entering in people’s minds in the first place, where we “vaccinate” people against misinformation through inoculation. Inoculation methods present weakened versions of the misinformation messages beforehand to build resistance and immunity to false narratives (McGuire et al. 1961). Through inoculation, we are able to prepare ourselves to spot and deal with misinformation targeting our communities and mitigate the potential damage misinformation can cause.
In this project, we would like to “vaccinate” people against misinformation through inoculation, where we develop a text-message-based course to deliver effective treatments to participants.
We want to test out the following research questions :
- Can a text message-based course “inoculate” users against misinformation?
- Hypothesis 1: Courses delivered in text message format treat participants effectively in spotting misinformation (i.e. at least one of the three treatments works, which requires a much smaller sample size than all treatments work)
- Hypothesis 2: Participants who receive analytical treatment should do better in spotting misinformation using analytical techniques. Participants who receive emotional treatment should do better in spotting misinformation using emotional techniques. Participants who receive combo treatment (i.e. mixture of analytical and emotional) should do better in both.
- How can we improve the efficacy and scalability of the course?
- Is the course differentially effective for different types of users (based on age, political affiliations, etc.)?
Current Pre-Test Scheme
This pilot is mainly used as a way to gain some feedback on the course content from actual people. Claire from First Draft is currently teaching a course in Cornell and has offered the opportunity to have her students take the entire flow as extra credits.
We enrolled the students by distributing the link to the pre-course survey, where students will automatically enroll into one of the four treatment courses once they finish the pre-course survey. They will then get 8 days of content (or wait 8 days if they are in control) before they receive the link to the post-course survey where they do a similar set of questions as the pre-course survey and fill out some basic feedback questions.
Survey Structure
- Consent
- Pledge: “While there are no right or wrong answers, we will evaluate your participation and discard low quality responses, so think about each question carefully and answer truthfully. Please, confirm you will do so by choosing the right option below.”
- I will pay attention throughout the survey and answer the questions after thinking through them carefully.
- I will NOT pay attention throughout the survey NOR answer the questions after thinking through them carefully.
- Instruction: “You will see a series of social media posts in the following screens. Please, read each one of them carefully and answer the questions at the end of each screen.”
- Section 1 (4 posts)
- Midway Message: “you are about halfway through, keep going!”
- Section 2 (8 posts)
- Demographics (if pre-course survey) / Feedback (if post-course survey)
Note:
- respondents do not know a clear cut difference between section 1 and section 2; the only difference they see are the different questions
Technical Issues
We have experienced tremendous amount of technical issues in this iteration, mainly with the following:
- Phone number format
- Text message platform, Arist, did not send activation message that contains the post-course survey link
- Post-course survey link not visible enough for people to click on
Potential Analyses Issue Due to Above Technical Issues
- Control course people might have already started the course content before doing the post-course survey
- Missing large chunks of people not completing the post-survey
- There might be duplicates as students want to earn extra credits
Summary of Learning
Survey Respondents
- n (pre_course_survey) = 98, n (post_course_survey) = 60
- after filtering and joining the responses, we get n = 34 valid respondents
Concerns
- roo many technical issues undermine the results
- Cornell students are way too good at spotting the manipulativeness / accuracy of the posts and do not want to share in almost all cases. This might be because they already know what this survey is about given their teacher Claire is the head of First Draft.
Recommendation
- DO NOT use this data for any treatment effect estimation as this is a hugely biased subset of pouplation
Data
Load Packages
Read Data
Data Cleaning
Variable Encoding
Treatment Distribution
Data Analysis
Section 1
In section 1, we aim to test whether respondents can distinguish the level of manipulation of the post. We have a pool of 4 manipulative posts as well as their non-manipulative counterparts (i.e. 4 facts). We show two manipulative posts and two non-manipulative posts (with no overlaps of facts) and asked the following questions for each post:
Would you share this post? - [public] I would share it publicly (e.g. on timeline or feed) [Yes / No] - [private] I would share it privately (e.g. through a private message) [Yes / No]
Would you do anything else with the post? [Yes / No]
[If Yes to previous question] What else would you do? Please tell us in a few words.
Do you agree or disagree with the following statement about the post? This post is manipulative.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 1
Sample Post Section 1
List of Actions to Take
Pre Course (Sharing Intention)
Post Course (Sharing Intention)
Pre Course (Whether Take Actions or Not)
Post Course (Whether Take Actions or Not)
List of Action People Say (Pre)
non-manipulative_fact1
| x1_1_a_other | Freq |
|---|---|
| Do more research on the topic | 1 |
non-manipulative_fact3
| x1_3_a_other | Freq |
|---|---|
| Look at the comments and see if it is true | 1 |
non-manipulative_fact4
| x1_4_a_other | Freq |
|---|---|
| Take it into consideration next time I am sick | 1 |
non-manipulative_fact5
| x1_5_a_other | Freq |
|---|---|
| Apply to my life if met in a sunburn crisis | 1 |
| I would save this post. | 1 |
non-manipulative_fact6
| x1_6_a_other | Freq |
|---|---|
| I would possibly share this privately with close friends who often skip a meal | 1 |
| stop skipping meals | 1 |
| Take it into consideration. It displays the word “evidence” | 1 |
| Try it myself | 1 |
non-manipulative_fact8
| x1_8_a_other | Freq |
|---|---|
| read it myself | 1 |
manipulative_fact1
| x1_1_b_other | Freq |
|---|---|
| I’d put it in my discord meme server | 1 |
manipulative_fact2
| x1_2_b_other | Freq |
|---|---|
| Ask to get rid of it | 1 |
manipulative_fact4
| x1_4_b_other | Freq |
|---|---|
| Send it to my mom because we have a running joke about this | 1 |
manipulative_fact6
| x1_6_b_other | Freq |
|---|---|
| I would save the post. | 1 |
| Save it to remind myself to not skip meals | 1 |
manipulative_fact8
| x1_8_b_other | Freq |
|---|---|
| Comment under the post of how it is wrong. | 1 |
| look up which antibiotics specifically will react poorly with alcohol just to be safe | 1 |
| Report it | 1 |
| Send it to a friend as a joke | 1 |
List of Action People Say (Post)
non-manipulative_fact1
| x1_1_a_other | Freq |
|---|---|
| Gender stereotype presented | 1 |
non-manipulative_fact2
| x1_2_a_other | Freq |
|---|---|
| I would downvote it. | 1 |
manipulative_fact3
| x1_3_b_other | Freq |
|---|---|
| Meme it | 1 |
| Report it because it’s opinionated and it can be hurtful towards a specific group of people | 1 |
manipulative_fact5
| x1_5_b_other | Freq |
|---|---|
| Report it maybe | 1 |
manipulative_fact8
| x1_8_b_other | Freq |
|---|---|
| I would downvote it. | 1 |
Rating on manipulativeness
Pre Course
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2.00 | 2 | 2.500 | 4.00 | 4 | 1.168 |
| non-manipulative_fact2 | 1 | 1.25 | 2 | 2.000 | 2.00 | 4 | 0.943 |
| non-manipulative_fact3 | 2 | 2.00 | 2 | 2.727 | 3.50 | 4 | 0.905 |
| non-manipulative_fact4 | 1 | 2.00 | 2 | 2.538 | 3.00 | 5 | 1.198 |
| non-manipulative_fact5 | 1 | 1.00 | 2 | 1.833 | 2.00 | 3 | 0.707 |
| non-manipulative_fact6 | 2 | 2.00 | 3 | 2.727 | 3.00 | 4 | 0.631 |
| non-manipulative_fact7 | 1 | 2.00 | 3 | 2.708 | 3.00 | 5 | 0.955 |
| non-manipulative_fact8 | 1 | 2.00 | 2 | 2.250 | 3.00 | 3 | 0.754 |
| manipulative_fact1 | 2 | 3.00 | 4 | 3.667 | 4.00 | 5 | 0.970 |
| manipulative_fact2 | 1 | 2.75 | 4 | 3.500 | 4.00 | 5 | 1.155 |
| manipulative_fact3 | 1 | 3.75 | 4 | 3.812 | 5.00 | 5 | 1.223 |
| manipulative_fact4 | 2 | 4.00 | 4 | 4.000 | 4.00 | 5 | 0.791 |
| manipulative_fact5 | 1 | 3.00 | 4 | 3.471 | 4.00 | 5 | 1.328 |
| manipulative_fact6 | 1 | 2.00 | 2 | 2.692 | 4.00 | 4 | 1.182 |
| manipulative_fact7 | 1 | 4.00 | 4 | 3.875 | 4.25 | 5 | 1.246 |
| manipulative_fact8 | 2 | 4.00 | 4 | 4.000 | 5.00 | 5 | 0.866 |
Post Course
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2.00 | 2.0 | 2.583 | 3.00 | 5 | 1.084 |
| non-manipulative_fact2 | 1 | 1.75 | 3.5 | 3.000 | 4.00 | 5 | 1.512 |
| non-manipulative_fact3 | 2 | 2.25 | 3.0 | 2.833 | 3.00 | 4 | 0.753 |
| non-manipulative_fact4 | 2 | 2.00 | 2.5 | 2.600 | 3.00 | 4 | 0.699 |
| non-manipulative_fact5 | 1 | 2.00 | 2.0 | 2.333 | 2.75 | 4 | 1.033 |
| non-manipulative_fact6 | 3 | 3.00 | 3.5 | 3.625 | 4.00 | 5 | 0.744 |
| non-manipulative_fact7 | 2 | 2.00 | 3.0 | 2.889 | 3.00 | 4 | 0.782 |
| non-manipulative_fact8 | 1 | 1.00 | 2.0 | 2.111 | 3.00 | 4 | 1.054 |
| manipulative_fact1 | 1 | 1.75 | 2.0 | 2.000 | 2.25 | 3 | 0.816 |
| manipulative_fact2 | 1 | 2.00 | 3.0 | 3.000 | 4.00 | 5 | 1.279 |
| manipulative_fact3 | 2 | 3.00 | 4.0 | 3.643 | 4.00 | 5 | 0.745 |
| manipulative_fact4 | 3 | 4.00 | 4.0 | 3.833 | 4.00 | 4 | 0.408 |
| manipulative_fact5 | 1 | 1.75 | 3.0 | 2.750 | 4.00 | 4 | 1.282 |
| manipulative_fact6 | 2 | 3.00 | 3.0 | 3.111 | 4.00 | 4 | 0.782 |
| manipulative_fact7 | 2 | 4.00 | 4.0 | 4.000 | 4.50 | 5 | 1.000 |
| manipulative_fact8 | 4 | 4.00 | 4.0 | 4.250 | 4.25 | 5 | 0.463 |
Section 2
In section 2, we aim to test whether respondents can distinguish whether a post is misinformation or not. We have a pool of 4 misinformation posts with emotion techniques, 4 misinformation posts with tactics, 4 general misinformation posts, 4 factually true posts. We show two posts from each type (randomized), and asked them the following questions for each:
Note that the facts in all of the posts do not overlap (i.e. we have 16 facts / topics)
Would you share this post? - [public] I would share it publicly (e.g. on timeline or feed) [Yes / No] - [private] I would share it privately (e.g. through a private message) [Yes / No]
Would you do anything else with the post? [Yes / No]
[If Yes to previous question] What else would you do? Please tell us in a few words.
Do you agree or disagree with the following statement about the post? The information presented in this post is accurate.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
Sample Post Section 2
Sample Post Section 2
List of Actions to Take
Pre Course (Sharing Intention)
Post Course (Sharing Intention)
Pre Course (Take Actions)
Post Course (Take Actions)
List of Action People Say (Pre)
tactics_fact1
| x2_1_t_other | Freq |
|---|---|
| search for the article | 1 |
tactics_fact2
| x2_2_t_other | Freq |
|---|---|
| Address the apparent issue in data visualization (scale) | 1 |
| I would probably reply and ask what the metrics and units are for as well as how this data was collected | 1 |
| That its not true | 1 |
| Yes, I would report the post as its misleading. | 1 |
tactics_fact3
| x2_3_t_other | Freq |
|---|---|
| I would likely report it for spreading misleading information. | 1 |
| Report it. | 1 |
| Write a comment - look at the number, not the size of the circle | 1 |
tactics_fact4
| x2_4_t_other | Freq |
|---|---|
| Comment on the error of logical reasoning and say that there are other factors to obesity. | 1 |
| probably make fun of it with my friends and report it to see if it is misleading or not | 1 |
| Unfollow | 1 |
emotion_fact1
| x2_1_e_other | Freq |
|---|---|
| I might ask Trushar where they got this information so they could maybe dig themselves out of this hole | 1 |
| I would share it to my friends as a sort of ‘joke’ since it’s very obviously a non-scientific post. | 1 |
| If I were to share it publicly or privately, I would be using this as counterexample of what the right thing to do is. | 1 |
| Put on discord meme server | 1 |
| Signal it to moderators, if possible. | 1 |
emotion_fact2
| x2_2_e_other | Freq |
|---|---|
| Ask for evidence | 1 |
emotion_fact3
| x2_3_e_other | Freq |
|---|---|
| google to verify the information | 1 |
| I would probably share this as something funny. | 1 |
| Look at the comments and see if it is true | 1 |
| share it with my friends | 1 |
emotion_fact4
| x2_4_e_other | Freq |
|---|---|
| Dislike/downvote | 1 |
| probably send it to a few of my friends and discuss whether or not it is true/fact check it. | 1 |
| report it | 1 |
| Report it for disinformation | 1 |
| Signal it to moderators | 1 |
| Slide away as soon as possible because the picture looks terrifying | 1 |
| Unfollow/block Avni | 1 |
general_misinfo_fact1
| x2_1_g_other | Freq |
|---|---|
| Look at the comments and see if it is true | 1 |
| Screenshot it for myself | 1 |
| Share it even out of my social media circles. Legalize marijuana | 1 |
| Share it with my mother | 1 |
general_misinfo_fact2
| x2_2_g_other | Freq |
|---|---|
| Although I wouldn’t personally share it with friends, I’m there are other people (especially to those this topic is more relevant) that would share this with friends | 1 |
| Google to see if it’s true | 1 |
| I might comment and ask how this effects its efficacy in indicating pregnancy from urine | 1 |
| look up which brand pregnancy tests we need to look out for | 1 |
| Remember the brand mentioned in the article | 1 |
| search on the internet for more information | 1 |
| Would refer back to it if I or a friend received a positive pregnancy test | 1 |
general_misinfo_fact3
| x2_3_g_other | Freq |
|---|---|
| I would dislike it if I could | 1 |
| I’d laugh about it with my other autistic friends | 1 |
| Ignite a conversation in my circles | 1 |
| Look at the comments and see if it is true | 1 |
| Report | 1 |
| Unfollow/block Yogesh | 1 |
general_misinfo_fact4
| x2_4_g_other | Freq |
|---|---|
| Ask for evidence | 1 |
| send it to my friends and discuss if we think it is real or worth trying or if it is potentially dangerous | 1 |
factually_true_fact2
| x2_2_f_other | Freq |
|---|---|
| Look at the comments and see if it is true | 1 |
| Save for a later post | 1 |
factually_true_fact3
| x2_3_f_other | Freq |
|---|---|
| Comment about my agreement towards the statement | 1 |
| I would save it for myself | 1 |
| Look at the comments and see if it is true | 1 |
| watch the video | 1 |
factually_true_fact4
| x2_4_f_other | Freq |
|---|---|
| Like the post | 1 |
List of Action People Say (Post)
tactics_fact1
| x2_1_t_other | Freq |
|---|---|
| Joke about it with a friend | 1 |
tactics_fact2
| x2_2_t_other | Freq |
|---|---|
| Why this is false and how we should be careful of these posts | 1 |
tactics_fact3
| x2_3_t_other | Freq |
|---|---|
| Dislike it, misrepresenting the circles in diagram | 1 |
| I would downvote it and report it for spreading misinformation. | 1 |
| Report it for misinformation. | 1 |
| Search up its validity on Google | 1 |
tactics_fact4
| x2_4_t_other | Freq |
|---|---|
| I would downvote it and report it. | 1 |
emotion_fact1
| x2_1_e_other | Freq |
|---|---|
| I would downvote and report it for misinformation. | 1 |
| report it | 1 |
| Report it and maybe google the validity | 1 |
emotion_fact4
| x2_4_e_other | Freq |
|---|---|
| I would downvote it and report for misinformation. | 1 |
general_misinfo_fact1
| x2_1_g_other | Freq |
|---|---|
| Make jokes about it | 1 |
| Report it for misinformation. | 1 |
| Send it to my friends | 1 |
general_misinfo_fact2
| x2_2_g_other | Freq |
|---|---|
| Report it for misinformation. | 1 |
general_misinfo_fact3
| x2_3_g_other | Freq |
|---|---|
| Ask for more evidence | 1 |
factually_true_fact1
| x2_1_f_other | Freq |
|---|---|
| Look up this information to confirm | 1 |
factually_true_fact4
| x2_4_f_other | Freq |
|---|---|
| Like it | 1 |
Rating on manipulativeness
Pre Course
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 2.0 | 3.0 | 2.560 | 3 | 4 | 0.870 |
| tactics_fact2 | 1 | 1.5 | 2.0 | 2.171 | 3 | 4 | 0.923 |
| tactics_fact3 | 1 | 1.0 | 2.0 | 2.206 | 3 | 5 | 1.175 |
| tactics_fact4 | 1 | 1.0 | 2.0 | 2.143 | 3 | 5 | 1.208 |
| emotion_fact1 | 1 | 1.0 | 1.5 | 1.857 | 2 | 5 | 1.239 |
| emotion_fact2 | 1 | 1.0 | 2.0 | 2.351 | 3 | 5 | 1.086 |
| emotion_fact3 | 1 | 2.0 | 2.0 | 2.484 | 3 | 5 | 0.962 |
| emotion_fact4 | 1 | 2.0 | 2.5 | 2.577 | 3 | 5 | 1.270 |
| general_misinfo_fact1 | 1 | 2.0 | 3.0 | 2.719 | 3 | 5 | 0.991 |
| general_misinfo_fact2 | 1 | 2.0 | 3.0 | 2.690 | 3 | 4 | 0.660 |
| general_misinfo_fact3 | 1 | 1.0 | 2.0 | 2.115 | 2 | 5 | 1.395 |
| general_misinfo_fact4 | 1 | 1.0 | 2.0 | 2.000 | 3 | 4 | 0.970 |
| factually_true_fact1 | 2 | 3.0 | 3.0 | 3.200 | 4 | 5 | 0.764 |
| factually_true_fact2 | 1 | 3.0 | 4.0 | 3.455 | 4 | 5 | 0.938 |
| factually_true_fact3 | 1 | 2.0 | 3.0 | 3.032 | 4 | 5 | 1.048 |
| factually_true_fact4 | 2 | 2.0 | 4.0 | 3.424 | 4 | 5 | 1.032 |
Pre Course
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 2.00 | 2.0 | 2.294 | 3.00 | 4 | 0.849 |
| tactics_fact2 | 1 | 2.00 | 2.0 | 2.231 | 3.00 | 4 | 0.927 |
| tactics_fact3 | 1 | 1.00 | 1.5 | 1.750 | 2.00 | 5 | 1.065 |
| tactics_fact4 | 1 | 1.00 | 2.0 | 2.136 | 3.00 | 4 | 1.082 |
| emotion_fact1 | 1 | 1.00 | 1.0 | 1.600 | 2.00 | 5 | 1.095 |
| emotion_fact2 | 1 | 2.00 | 2.5 | 2.833 | 3.25 | 5 | 1.267 |
| emotion_fact3 | 1 | 2.00 | 2.0 | 2.421 | 3.00 | 4 | 0.902 |
| emotion_fact4 | 1 | 2.00 | 2.0 | 2.294 | 3.00 | 4 | 1.047 |
| general_misinfo_fact1 | 1 | 2.00 | 2.0 | 2.214 | 2.75 | 4 | 0.802 |
| general_misinfo_fact2 | 1 | 2.00 | 3.0 | 2.588 | 3.00 | 5 | 1.121 |
| general_misinfo_fact3 | 1 | 1.00 | 2.0 | 1.864 | 2.00 | 5 | 1.037 |
| general_misinfo_fact4 | 1 | 1.50 | 2.0 | 2.200 | 3.00 | 4 | 1.014 |
| factually_true_fact1 | 2 | 2.25 | 3.0 | 2.864 | 3.00 | 4 | 0.640 |
| factually_true_fact2 | 2 | 3.00 | 4.0 | 3.533 | 4.00 | 5 | 0.743 |
| factually_true_fact3 | 2 | 2.50 | 3.0 | 3.133 | 4.00 | 4 | 0.834 |
| factually_true_fact4 | 2 | 3.00 | 3.0 | 3.375 | 4.00 | 5 | 0.806 |
