Problem
COVID-19 vaccine hesitancy has been recognized as a problem across nations. A resistance to getting vaccinated is emerging as a major hurdle, especially in the developing world, where vaccine access issues are still being gradually resolved. Persistent pools of unvaccinated people around the world could present a greater risk for the emergence of new variants of concern. Addressing people’s vaccine hesitancy is hence crucial to curb the spread of COVID-19, and to consequently avert hospitalizations and death.
Objectives
We intend to understand why people are hesitant about getting the COVID-19 vaccine. Hesitancy could not only occur within the unvaccinated population but also in a subset of people who already got vaccinated. Therefore, phase 1 of the project has the following objectives:
Approach
We intend to use chatbot as a medium (on Facebook) to conduct conversations with people to understand how we can best achieve the above three objectives. We have run five pilots as of June 23, 2022, – 2 in the United States using Qualtrics on Lucid, and 3 in South Africa on Facebook. The eventual goal will be running this using multiple chatbots that enable the conversation to flow more naturally than in a survey format. We hypothesize that respondents are more likely to respond to our sensitive questions around vaccine hesitancy if the questions are asked more casually in an open stress-free setting. Therefore, in all version of the pilots, we make our tone as causal as possible (using emojis, GIFs, emphatic prompts) and include delays in the appearance of questions (and empathetic responses after each question) to make the conversation more authentic.
Analysis script used is linked here. Relevant GitHub issue is linked here.
This analysis is based on 399 respondents who completed the current pilot survey wave.
We have split the detailed analyses into 5 sections (preceded by a set of key takeaways):
Summarizing our key takeaways from the current pilot below:
Goals for this section:
List of analyses in this section:
We start off with a plot demonstrating the split among vaccinated and unvaccinated participants in this pilot:
We asked participants about whether they have the motivation to get the COVID-19 vaccine and whether they have the ability to get the vaccine for both vaccinated and unvaccinated people. We then fork them into 8 different segments based on the vaccination status, motivation to get the vaccine, and ability to get the vaccine. We obtain the distribution below:
| Vaccination status | Able to get vaccine | Have motivation to get vaccine | Count | Percentage of total participants |
|---|---|---|---|---|
| unvax | yes | no | 195 | 49% |
| unvax | yes | yes | 27 | 7% |
| unvax | no | no | 15 | 4% |
| unvax | no | yes | 11 | 3% |
| vax | yes | yes | 80 | 20% |
| vax | yes | no | 64 | 16% |
| vax | no | yes | 6 | 2% |
| vax | no | no | 1 | 0% |
Takeaways
62% of participants are unvaccinated. Among the unvaccinated:
Let’s investigate each impediment (motivation & ability) in detail, and see the distributions of the reasons why they have such impediments.
Takeaways
We asked: What’s the main reason you don’t want to be vaccinated?
Provided options:
The distribution of the answers demonstrates below:
We asked: is there a main reason why you think there isn’t a benefit?
Provided options:
The distribution of the answers demonstrates below:
We asked: is there a main reason why you think there is risk?
Provided options:
The distribution of the answers demonstrates below:
We asked: is there a main reason why against your belief?
Provided options:
The distribution of the answers demonstrates below:
We asked: What’s the main difficulty of getting vaccinated?
Provided options:
The distribution of the answers demonstrates below:
We asked: is there a main reason why there isn’t availability?
Provided options:
We asked: is there a main reason why there isn’t time?
Provided options:
We asked: is there a main reason why there isn’t money?
Provided options:
The correlation plots here aim to understand what demographic variables and ability impediments are directly correlated with each other. We want to filter out the demographic variables that are highly correlated with other demographic variables and the demographic variables that are not related to the motivation/ability impediment to minimize the number of questions respondents have to answer.
Since the correlation matrixes provided by ggcorrplot()
shows the correlation coefficients between continuous variables, we
mapped binary and ordinal variables to continuous variables.
Details on the mapping:
female: 1 if female, 0 if malecountry: 1 if live in South Africa, 0 if notincome: 0 if the participant is unemployed, 1 if income
< R5,000, 2 if income in R5,000 – R9,999, …, 6 if income >
R100,000education: 1 if the participant’s education < high
school, 2 if education is high school, …, 6 if education is a graduate
degreereligiosity: 1 if the participant is not very
religious, 2 if somewhat religious, 3 if very religiouspolitics: 1 if the participant is conservative, 2 if
moderate, 3 if liberallocation: 1 if the participant lives in rural, 2 if
suburban, 3 if urban,white: 1 if the participant is a white or caucasian, 0
if notability: 1 if the participant has the ability to get
vax, 0 if notmotivation: 1 if the participant has the ability to get
vax, 0 if not = treat_nchar: Character count of response
for best treatment (free text) explanationSince we are comparing categorical variables (because some of the variables aren’t linear so we should not assign numerical values to them), we also measure association using Cramer’s V coefficient. The Cramer’s V is a correlation statistic developed to measure the strength of association between two nominal variables. We calculate the Cramer’s V coefficients between motivation/ability and demographic variables as below to find which demographic variables have a high association with motivation/ability impediments, which can be used as a reference in participants segmentation.
Correlation plot:
Takeaways: Low demographic correlation overall
Takeaways: Low demographic correlation overall
Takeaways: Low demographic association overall
Takeaways: Low demographic correlation overall
Goals for this section:
List of analyses in this section:
We now consider the funnel dropoff and cost comparison across the last two pilot waves.
## Error in file(file, "rt"): cannot open the connection
## Error in file(file, "rt"): cannot open the connection
## Error in eval(expr, envir, enclos): object 'ads' not found
## Error in eval(expr, envir, enclos): object 'ads' not found
## Error in eval(expr, envir, enclos): object 'ads' not found
## Error in eval(expr, envir, enclos): object 'ads' not found
## Error in data.frame(metric = c("Impressions", "Link Clicks", "Messages Sent", : object 'impressions' not found
## Error in eval(expr, envir, enclos): object 'ads_old' not found
## Error in eval(expr, envir, enclos): object 'ads_old' not found
## Error in eval(expr, envir, enclos): object 'ads_old' not found
## Error in eval(expr, envir, enclos): object 'ads_old' not found
## Error in data.frame(metric = c("Impressions", "Link Clicks", "Messages Sent", : object 'impressions' not found
## Error in left_join(dropoff_old, dropoff_current, suffix = c("_v4a", "_v4b"), : object 'dropoff_old' not found
Takeaway: Cost-per-action has fallen considerably since the previous pilot version. Variation of cost-per-action with pilot version can be seen below.
## Error in list2(...): object 'dropoff_old' not found
Takeaway: There is strong improvement in funnel dropoff rates before respondents complete the treatment questions. Variation of funnel dropoff (% of previous bucket) with pilot version can be seen below:
## Error in list2(...): object 'dropoff_old' not found
Dropoff by Fork Path (Motivation/Ability/Both/None) for current pilot
This table looks at how much respondents dropoff from their conversation based on what impediments they have grouped by vaccination status. Note that if the respondents have both motivation and ability impediments, they will need to fill out both the ability and the motivation section (i.e. twice the length). Note:Percentages are calculated within the group.
| Impediment | Dropoff count (unvax) | Dropoff % (unvax) | Dropoff count (vax) | Dropoff % (vax) |
|---|---|---|---|---|
| Both motivation and ability | 8 | 34.78 | 6 | 85.71 |
| None | 9 | 25.00 | 47 | 37.01 |
| Motivation | 42 | 17.72 | 26 | 28.89 |
| Ability | 1 | 8.33 | 2 | 25.00 |
Takeaway: For unvaccinated respondents, motivation questions (as compared to ability questions) lead to considerable more dropoff. The high dropoff for motivation and ability questions could be on account of serving more questions.
Enjoyment levels for current pilot (n = 396)
Note that we filter out responses that do not belong to one of the three pre-defined choices.
Takeaway: Most participants enjoyed interacting with the chatbot.
Comfort levels for current pilot (n = 326):
Takeaway: Most participants were comfortable interacting with the chatbot.
Parts to improve:
| enjoyable_not_parts | n |
|---|---|
| All | 3 |
| Everything | 2 |
| All of it. | 1 |
| All.of.it as these questions are trying to convince people to get the vaccine and I will never submit to control by WHO or any government | 1 |
| Asking personal information | 1 |
| Nothing | 1 |
| Political | 1 |
| Talking about vaccine | 1 |
| The parts of household total yearly income | 1 |
| The question about the vaccine | 1 |
| Those parts that seemed like a bribery into taking it, I mean what’s the “which of this are most likely to get you jabbed?” | 1 |
| Vaccine | 1 |
| comfortable_not_parts | n |
|---|---|
| All | 2 |
| All of it | 1 |
| All parts | 1 |
| Just chart board. I feel I am wasting my time | 1 |
| Mostly part | 1 |
| Politics | 1 |
| Questions about my gender my race | 1 |
| Some personal details like gender and residency info | 1 |
| The fact that this bot kept trying to “humanize” with me by agreeing to me | 1 |
| The part when I already answer your question that I don’t want your vaccine than you keep asking me but if were likely to take it but seriously why don’t all people who says vaccines are Safe let them and they families take it and respect our choices | 1 |
| When you asked about my household’s total yearly income | 1 |
Goal for this section:
Analyses for this section:
Pre-Treatment: How likely are you to get vaxxed on a 1-5 scale? (n = 397)
Takeaway: More than 50% respondents are very unlikely to get a vaccine before treatment.
Which would be most likely to get you vaxxed? (n = 399)
Takeaways:
If you get your favored treatment, how likely are you to get vaxxed on a 1-5 scale? (n = 249)
Takeaway: Around 10% respondents move from 1 to [2-5]. Majority still favors not getting vaxxed.
The table below highlights improvement in vaccination likelihood for each treatment.
Change in vax chances by treatment (n = 247)
| best_treatment | n | vax_chance_pre | vax_chance_post | vax_chance_change |
|---|---|---|---|---|
| Rewards for vaxxing | 41 | 2.59 | 4.00 | 1.41 |
| Trusted ppl support | 44 | 2.23 | 3.05 | 0.82 |
| Planning reminders | 4 | 2.75 | 3.50 | 0.75 |
| Others getting it | 7 | 2.43 | 3.14 | 0.71 |
| More information | 52 | 2.44 | 3.10 | 0.66 |
| None of the above | 98 | 1.56 | 1.97 | 0.41 |
Takeaway: Rewards for vaccination stand out as the most helpful treatment. Followed by support from trusted connections.
The following graph shows a (crude) proxy for the average treatment effect for every treatment:
The following graph shows treatment effect proxy versus participant counts for each treatment:
Takeaway: No odd outliers - most treatments generate a positive change in vaccination likelihood.
Free text explanations provided for each treatment (sorted from most to least responses) can be seen below:
| more information | n |
|---|---|
| side effects | 2 |
| The after effects | 2 |
| 1. What’s the ingredients or whatsoever the vaccine contain | 1 |
| A clearer indication as to what happens to my body when I take the vaccine | 1 |
| About the effects of vax | 1 |
| About the vaccine | 1 |
| About the Vax | 1 |
| Actual statistics of success rate | 1 |
| Actually not at all | 1 |
| Any info | 1 |
| Assurance that if i don’t vax wjat would happen in future | 1 |
| Assurance that the vaccine is not harmful | 1 |
| Benefits | 1 |
| Benefits of getting vaccinated | 1 |
| certainty of side effects and safety of the vaxx in the future | 1 |
| Collectives research and development of what caused the Covid in the first place | 1 |
| Even you can get covid but after vaccine you will not get very sick | 1 |
| From doctors,scientists and medical professionals | 1 |
| From the doctors and scientists that developed the vaccine | 1 |
| Getting to apply for a better job with vac certificate | 1 |
| Google Information | 1 |
| How it helps | 1 |
| How it really works | 1 |
| How vaccines helps if you ever contact with covid-19, if you’re vaccinated you won’t be as sick as someone who is not vaccinated | 1 |
| I asked I googled and I even listened to the media even if some other information was scary. I thought it is better to live the few years than to be at risk right away | 1 |
| I did a research about the vaccine in order to be able to clear the doubts planted in my head by other people | 1 |
| I dont want it | 1 |
| I felt more safer | 1 |
| I made my own research about the vaccine | 1 |
| I need to know more about the vaccine, last time i checked any vaccine has to go through trials for at least 3 years, not just a couple of months | 1 |
| I started doing my own research on family and friends and people I knew who had been vaccinated and who hadn’t and I lost loved ones | 1 |
| if it is really safe | 1 |
| If only this vax is African produced | 1 |
| If vaccinated even if you are exposed to the virus it won’t be as strong as when you are not vaccinated | 1 |
| If you get vaxxed you don’t get sick easy | 1 |
| Information from doctor and nurses And research on the internet | 1 |
| Information from independent sources | 1 |
| It boosts the immune system | 1 |
| It helps decrese the spread of the virus | 1 |
| It was safe and secure | 1 |
| It was said that once you get vaccinated its not going to be easy to get the covid 19. Or once you get it. It doesnt become deadly | 1 |
| k | 1 |
| know how did they come up with a vax and where did they got that little corona to do vax | 1 |
| Knowing the after effects of the vaccine | 1 |
| Learning that I won’t be able to access somethings ad places if I’m not vaccinated | 1 |
| Long and short term effects | 1 |
| More info about the vax | 1 |
| More information about covid and vaccines | 1 |
| My aunt is a volunteer in a clinic, and she had already got her vaccine so she told me there’s nothing wrong with being vaccinated and also highlighted the side effect more specifically. | 1 |
| On how it works, how they inject it. | 1 |
| People really don’t wanna vaccine the government forces them | 1 |
| Pressed wrong answer | 1 |
| Prooving the vac works | 1 |
| Research findings, any side effects..previous use of similar vacs etc | 1 |
| Safety of it first | 1 |
| Safety, and insurance that will help should I experience server side effect or death. | 1 |
| Saftey regarding it | 1 |
| Save myself and people around me | 1 |
| Seeing people who got vaccinated surviving than those who refuse to take the jab | 1 |
| Side effects | 1 |
| Since i am looking for work other jobs want vaccinated people | 1 |
| Statistics | 1 |
| Statistics, majority of those who lost their lives were not vaccinated. | 1 |
| Tell people everything about the vaccine tell them what to expect in a long term after they got vaccinated | 1 |
| That if you get you are not at risk for your family not to get sick | 1 |
| That i can get vaccinated with my cardiac problem and that i can still live a long life | 1 |
| That is qritten on bronchures | 1 |
| That it safe and learning about the sides effect and also to protect the ones near me | 1 |
| That it will boost my immune system to fight the virus | 1 |
| That it works | 1 |
| That you won’t die of COVID if vaxxed | 1 |
| The advantages & disadvantages | 1 |
| The benefit of vax… the importance of it | 1 |
| The love i have for my family. | 1 |
| The positive side of the vaccine shot | 1 |
| The pros and cons,how it would affect my health in future and stuff like that | 1 |
| The side effects of getting the vaccine and how they vaccine benefit me | 1 |
| The vax will protect you from the covid …not that it will cure it no ….but it will help the immune system to fight the virus but you must still wear a mask stay indoors etc | 1 |
| There more info in the public domain even fake news but what gave me the push was the number of deaths due to covid | 1 |
| There was more good from getting the vaccine than not getting it at all. | 1 |
| There were rumours that if you get vaccinated you will die so information about the safety of vaccine made me want to vaccinated | 1 |
| To be Educated more about Vaccine | 1 |
| Utube videos and social media ads explaining how the vacine affects the virus | 1 |
| Well i got some information from my friend who has already got the vax and online | 1 |
| What is it the vax | 1 |
| What is the vax | 1 |
| What it does | 1 |
| What more its made of | 1 |
| What’s in the vaccine | 1 |
| Who are at risk & where do they get the information | 1 |
| Why do people that got the vaxx still get infected | 1 |
| Why should it be mandatory to be vaxed | 1 |
| Why some people die immediately after vaccination, | 1 |
| Why some ppl get very sick | 1 |
| trusted ppl support | n |
|---|---|
| My family | 9 |
| Family | 8 |
| My dad | 2 |
| My doctor | 2 |
| My family and friends | 2 |
| My friends and family | 2 |
| My parents | 2 |
| All the efforts put inbthe research and preparation of the vac, the staff administering the procedurez and ultimately standing the chance of never being infected due to the vac | 1 |
| As am an HIV patient we have our own clinic,so it would be best if our doctors would do that for us | 1 |
| Can we skip this question | 1 |
| Clicks | 1 |
| Clinic members | 1 |
| Cousin | 1 |
| Doctor’s order | 1 |
| 𝐃𝐨𝐜𝐭𝐨𝐫𝐬 .𝐬𝐜𝐢𝐞𝐧𝐭𝐢𝐬𝐭 | 1 |
| Doctors and health care workers | 1 |
| doctors medical professionals | 1 |
| Don’t know | 1 |
| Dr | 1 |
| Fam | 1 |
| FAMILY | 1 |
| Family and friends | 1 |
| Family doctor | 1 |
| Family members | 1 |
| Family or friends | 1 |
| Idk tbh | 1 |
| Jesus | 1 |
| Johnson and johnso | 1 |
| Knowing that I have done what I can to protect myself, my family and collegues | 1 |
| Like friends and family | 1 |
| Many peoplr supported the vacation | 1 |
| My family member’s | 1 |
| My airtime | 1 |
| My children and neighbours | 1 |
| My community | 1 |
| My daughter whois a fifth year medical student - and her professors | 1 |
| My family 😊 | 1 |
| My family ofcause | 1 |
| My family off course | 1 |
| My family. Doctors who does everything to ensure we’re safe and healthy. | 1 |
| My friends | 1 |
| My friends and Family | 1 |
| My gut | 1 |
| My hole family got the vax | 1 |
| My mom | 1 |
| My mom , sister and my son | 1 |
| My mom & brothers | 1 |
| My mom and dad | 1 |
| my mother | 1 |
| My mother | 1 |
| My neighbor | 1 |
| My parents and husband | 1 |
| My partner i guess | 1 |
| Nurses, Teachers and family | 1 |
| Ok | 1 |
| Our home based.care.nurses.doctors.our parents.& our president | 1 |
| Our local clinic | 1 |
| Parents | 1 |
| Pharmacy | 1 |
| Ppl around me | 1 |
| President | 1 |
| Protecting my self and people around me | 1 |
| Stop asking stupid questions | 1 |
| The community | 1 |
| The doctors | 1 |
| The health workers who explained everything to me | 1 |
| The information l was getting from the Tv on a daily basis | 1 |
| The truth about the vaccine, just like they did with other vaccines | 1 |
| The truth behind covid | 1 |
| Thebgov shld tell us the truth about what really hapoend | 1 |
| they doctors | 1 |
| Those around me who still have to get it | 1 |
| Trusted doctors | 1 |
| Unicef and W.H.O | 1 |
| What | 1 |
| Whole fam | 1 |
| rewards for vaxxing | n |
|---|---|
| Money | 6 |
| Airtime | 2 |
| Vouchers | 2 |
| 💰 | 1 |
| A job maybe or learnership programme | 1 |
| Airtim🙂 | 1 |
| Any | 1 |
| Any thing from airtime to food voutchers | 1 |
| Food and cloths | 1 |
| Food voucher | 1 |
| Food vouchers or cash | 1 |
| Food vouchers or food parcels. That would be rewarding as people are dying of hunger | 1 |
| Freedom to move and the greater responsibility to keep safe | 1 |
| Gifts | 1 |
| Grocery vouchers | 1 |
| Have some food parcels or anything that might attract young and old people. | 1 |
| https://scontent.xx.fbcdn.net/v/t39.1997-6/cp0/39178562_1505197616293642_5411344281094848512_n.png?_nc_cat=1&ccb=1-5&_nc_sid=ac3552&_nc_ohc=nZrVu3nW41EAX_SH7p_&_nc_ad=z-m&_nc_cid=0&_nc_ht=scontent.xx&oh=00_AT_s9tX417qJeOqeSMxSh4NQyVKdu8oJetHeLD-w_Zk75A&oe=61FAE849 | 1 |
| i don’t know | 1 |
| I don’t Know | 1 |
| I found a job | 1 |
| I need cash | 1 |
| I will be able to go to university | 1 |
| Job | 1 |
| like life after death | 1 |
| like money for the things i dont have home | 1 |
| Money maybe 😅 | 1 |
| Money or airtime or data | 1 |
| Money or food vouchers | 1 |
| Money or toys | 1 |
| money, R1 million | 1 |
| money,food,vouchers, | 1 |
| Other people getting it | 1 |
| Shopping vouchers | 1 |
| Shoprite voucher | 1 |
| Since it is a trial bases then monitary rewards are best since we don’t know the long term effects of these vaccines | 1 |
| Store vouchers | 1 |
| superpowers | 1 |
| They should reward people who gets rewarded | 1 |
| Voucher rewards | 1 |
| We’ll we come from a very disadvantaged area, so maybe grocery vouchers | 1 |
| others getting it | n |
|---|---|
| My family | 3 |
| Family | 2 |
| All the people i wish to do it | 1 |
| Close family members | 1 |
| Friends and family | 1 |
| I’m telling everyone I meet to go and take vaccine. Everywhere I go I talk about it | 1 |
| Im not sure | 1 |
| My family and friends | 1 |
| My husband | 1 |
| My husband got it first so when I saw he was ok I agreed to get vaccinated as well | 1 |
| My parents | 1 |
| My teachers firstly, my friends and family | 1 |
| Not to be infected since people are taking it lightly | 1 |
| Ok | 1 |
| People I trust | 1 |
| People in my community | 1 |
| Study mates,Family and and most of the people within my community the | 1 |
| The president, Bishop Lekganyane and other top influences getting it make me believe that it is safe to take it | 1 |
| planning reminders | n |
|---|---|
| Any way | 1 |
| Getting my family safe | 1 |
| Giving me data | 1 |
| https://scontent.xx.fbcdn.net/v/t39.1997-6/cp0/39178562_1505197616293642_5411344281094848512_n.png?_nc_cat=1&ccb=1-5&_nc_sid=ac3552&_nc_ohc=nZrVu3nW41EAX_Ulc-j&_nc_ad=z-m&_nc_cid=0&_nc_ht=scontent.xx&oh=00_AT_idLzBb5sx1xDB0kafBCit9HD7QOI6WvUcCexDEB9sNQ&oe=61FCE289 | 1 |
| I’d plan how my days go,so that I can be prepared on the day of the second vax | 1 |
| Pick out a day where I’ll go vaccinate | 1 |
| Yes | 1 |
General analyses of all pilot variables are being generated through a separate script here. List of analyses generated: