Outcome Measurement Survey V1
January 31, 2022
Summary
Overall Survey Structure
Consent -> Instruction -> Section 1 (4 posts) -> Midway Message -> Section 2 (8 posts) -> Demographics
Note:
- midway message: “you are about halfway through, keep going!”
- respondents do not know a clear cut difference between section 1 and section 2; the only difference they see are the different questions
Demographics Description
- n = 99
- Deployed on Lucid (US population)
- Population ~80% white
- age lean old (1st quartile: 32, median: 47)
- in general educated (most have some college education, scientific index largely between 1 and 2 with 2 being the highest)
- have control over their life (locus of control median 7 on a scale of 10)
- gender roughly balanced
- mostly urban (70%)
- political leaning mostly right-wing (1st quartile 48, median 59, on a scale of 100, where 100 denotes right wing)
Future Changes
add indicator if a post is shown and store the order of appearance
- Question to address: Do people select the same answers for their questions?
- measure time spent on posts
Data
Load Packages
Read Data
Data Cleaning
Variable Encoding
Data Analysis
Section 1
In section 1, we aim to test whether respondents can distinguish the level of manipulation of the post. We have a pool of 4 manipulative posts as well as their non-manipulative counterparts (i.e. 4 facts). We show two manipulative posts and two non-manipulative posts (with no overlaps of facts) and asked the following two questions for each post:
If you see a friend / family member sharing this post on social media, what action(s) would you take? Please select all that apply:
- [share] Share the post
- [report] Report the post
- [delete] Privately prompt my friend / family member to delete the post
- [ignore] Ignore the post / do nothing
- [warn] Warn others about the post
- [ask] Ask my friend / family member why they shared the post
- [discuss] Discuss the post with my friend / family member
- [others] Other reason(s)
Do you agree or disagree with the following statement about the post? This post is manipulative.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
List of Actions to Take
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| non-manipulative_fact1 | 1 | 2 | 3 | 2.511 | 3 | 5 | 0.997 |
| non-manipulative_fact2 | 1 | 2 | 3 | 2.714 | 3 | 5 | 1.041 |
| non-manipulative_fact3 | 1 | 2 | 3 | 2.707 | 3 | 5 | 0.918 |
| non-manipulative_fact4 | 2 | 3 | 3 | 3.070 | 4 | 5 | 0.768 |
| manipulative_fact1 | 1 | 2 | 3 | 3.096 | 4 | 5 | 1.209 |
| manipulative_fact2 | 1 | 2 | 3 | 3.163 | 4 | 5 | 1.106 |
| manipulative_fact3 | 1 | 3 | 3 | 3.293 | 4 | 5 | 0.929 |
| manipulative_fact4 | 1 | 2 | 3 | 2.946 | 4 | 5 | 1.102 |
Section 2
In section 2, we aim to test whether respondents can distinguish whether a post is misinformation or not. We have a pool of 4 misinformation posts with emotion techniques, 4 misinformation posts with tactics, 4 general misinformation posts, 4 factually true posts. We show two posts from each type (randomized), and asked them the following questions for each:
Note that the facts in all of the posts do not overlap (i.e. we have 16 facts / topics)
If you see a friend / family member sharing this post on social media, what action(s) would you take? Please select all that apply:
- [share] Share the post
- [report] Report the post
- [delete] Privately prompt my friend / family member to delete the post
- [ignore] Ignore the post / do nothing
- [warn] Warn others about the post
- [ask] Ask my friend / family member why they shared the post
- [discuss] Discuss the post with my friend / family member
- [others] Other reason(s)
Do you agree or disagree with the following statement about the post? The information presented in this post is accurate.
- Completely disagree (1)
- Disagree (2)
- Neither agree nor disagree (3)
- Agree (4)
- Completely agree (5)
List of Actions to Take
Rating on manipulativeness
| Min. | 1st Qu. | Median | Mean | 3rd Qu. | Max. | SD | |
|---|---|---|---|---|---|---|---|
| tactics_fact1 | 1 | 2.0 | 3 | 2.810 | 3.00 | 5 | 1.100 |
| tactics_fact2 | 1 | 2.5 | 3 | 2.886 | 3.00 | 5 | 0.932 |
| tactics_fact3 | 1 | 2.0 | 3 | 2.973 | 4.00 | 5 | 1.213 |
| tactics_fact4 | 1 | 2.0 | 2 | 2.529 | 3.00 | 5 | 1.261 |
| emotion_fact1 | 1 | 1.0 | 2 | 2.404 | 3.00 | 5 | 1.192 |
| emotion_fact2 | 1 | 2.0 | 3 | 3.021 | 4.00 | 5 | 1.053 |
| emotion_fact3 | 1 | 2.0 | 3 | 2.788 | 3.25 | 5 | 1.073 |
| emotion_fact4 | 1 | 2.0 | 3 | 2.735 | 3.00 | 5 | 1.076 |
| general_misinfo_fact1 | 1 | 3.0 | 3 | 3.113 | 4.00 | 5 | 1.068 |
| general_misinfo_fact2 | 1 | 2.0 | 3 | 2.822 | 3.00 | 5 | 0.860 |
| general_misinfo_fact3 | 1 | 2.0 | 3 | 2.826 | 3.00 | 5 | 1.018 |
| general_misinfo_fact4 | 1 | 2.0 | 2 | 2.472 | 3.00 | 5 | 1.170 |
| factually_true_fact1 | 1 | 3.0 | 3 | 3.070 | 4.00 | 5 | 0.842 |
| factually_true_fact2 | 1 | 3.0 | 3 | 3.075 | 4.00 | 5 | 1.141 |
| factually_true_fact3 | 1 | 3.0 | 3 | 3.404 | 4.00 | 5 | 0.934 |
| factually_true_fact4 | 2 | 3.0 | 4 | 3.638 | 4.00 | 5 | 0.845 |
Responses in Other
Here we list out the free text response respondents report when they choose “Other Reason(s)” as one of the option for the following question (note that this include both section 1 and section 2)
If you see a friend / family member sharing this post on social media, what action(s) would you take? Please select all that apply:
- [share] Share the post
- [report] Report the post
- [delete] Privately prompt my friend / family member to delete the post
- [ignore] Ignore the post / do nothing
- [warn] Warn others about the post
- [ask] Ask my friend / family member why they shared the post
- [discuss] Discuss the post with my friend / family member
- [others] Other reason(s)
| reasons | |
|---|---|
| non-manipulative_fact1 | Just read it |
| non-manipulative_fact1 | None |
| non-manipulative_fact2 | None |
| non-manipulative_fact3 | Look into it |
| manipulative_fact2 | Lose weight |
| manipulative_fact3 | None |
| manipulative_fact4 | None |
| tactics_fact2 | None |
| tactics_fact2 | Research the topic |
| tactics_fact3 | None |
| emotion_fact1 | from start to finish |
| emotion_fact2 | None |
| emotion_fact3 | None |
| general_misinfo_fact1 | look at the article and search google to see if it is actual research |
| general_misinfo_fact1 | None |
| general_misinfo_fact1 | Take the meds |
| general_misinfo_fact2 | Research the topic |
| general_misinfo_fact3 | None |
| general_misinfo_fact3 | Research the topic |
| general_misinfo_fact4 | None |
| factually_true_fact1 | None |
| factually_true_fact3 | N/A |
| factually_true_fact4 | None |
Correlation Between Actions and Manipulation / Accuracy
This section is used to understand whether actions in the multi-select questions have correlation with how they answer manipulation question and accuracy question.
Section 1
Section 2
Codebook
This section is used to generate demographics codebook.
Consistency Check
Balance of Treatment
We did a treatment randomization using Qualtrics as a test step to see if randomization step can be done on our side (i.e. no actual treatment was assigned to respondents)
Multi-select Choice Pattern
This section is used to check whether people are less likely to select more choices in multi-select questions in section 2 than section 1 due to fatigue of posts.
