Project Overview

Problem

COVID-19 vaccine hesitancy has been recognized as a problem across nations. In some of the world’s poorest nations, where vaccine access issues are being gradually resolved, a resistance to getting vaccinated is emerging as a major hurdle. Scientists fear that persistent pools of unvaccinated people around the world will present a greater risk for the emergence of new variants of concern, like Omicron. Addressing people’s hesitancy is therefore crucial to curb the spread of COVID-19, and to consequently avert hospitalizations and death.

Objectives

We aim to understand why people are hesitant about getting the COVID-19 vaccine. Hesitancy could not only occur within the unvaccinated population but also in a subset of people who already got vaccinated. Therefore, phase 1 of the project has the following objectives:

  1. Understand why people are hesitant to get the COVID-19 vaccines
  2. Understand ways to best elicit vaccine impediments from respondents
  3. Pinpoint what interventions will help people get vaccinated

Approach

We intend to use chatbot as a medium (on Facebook) to conduct conversations with people to understand how we can best achieve the above three objectives. We have run five pilots as of March 07 2022, – 2 in the United States using Qualtrics on Lucid, and 3 in South Africa on Facebook. The eventual goal will be running this using multiple chatbots that enable the conversation to flow more naturally than in a survey format.

We hypothesize that respondents are more likely to respond to our sensitive questions around vaccine hesitancy if the questions are asked more casually in an open stress-free setting. Therefore, in all version of the pilots, we make our tone as causal as possible (using emojis, GIFs, emphatic prompts) and include delays in the appearance of questions (and empathetic responses after each question) to make the conversation more authentic.


Current Pilot

This analysis is based on 1689 respondents who completed the current pilot survey wave. We aim to compare the performance across three different themes, built upon hypothesized drivers of hesitancy, as well as different versions of creative within those themes.

  1. Learning Goal: To come away with descriptive segments and generalizable findings about the effectiveness of treatments on those segments, we’d ideally randomly the same population for the duration of our study.

  2. Viability Goal: We will need to drive for the lowest possible cost per participants in order to collect as many responses as possible within our budget. Our past pilots have relied on Facebook’s optimization to make progress towards this goal.

Overview of the Ads Strategy

Setting Principle:

  • One campaign covering all pilots in South Africa
  • 5 ad sets (different combinations of impediment, text, image)
  • Ad set optimization is on, so it will select the best ad within the ad set.
  • If ad sets work well and we want to scale, we duplicate the ad set within the campaign and set a new budget
  • We don’t duplicate campaigns (only ad sets), and we don’t change budgeting once it is initially set (if we want to scale, we duplicate more ad sets).


Ad Performance will be measured using the following indicators:

  • Click-through rate
  • Recruitment of non-vaccinated but potentially treatable participants
    • Total quantity
    • Percent of total participants recruited
    • Retention
  • Average participant elicitation
  • Cost
    • per impression
    • per Link Click
    • per Survey Complete

Setup

Detailed setting can be found here.

  • 1 campaign
  • 5 ad sets, 3 ads in each ad set
  • 15 ads split into:
    • 3 impediment themes (3 inaccessible, 6 risky, 6 unnecessary)
    • 2 different prompts (6 control and 9 airtime)
    • 9 different images (Images 1-6 used twice, images 7-9 used once)

Analysis

The ad analysis contains 3 tables using different combinations of 15 distinct ads:

  • split by 3 ad impediment themes
  • split by 2 ad body text approaches
  • split by 9 ad images

Metrics explanation:

  • Impressions (Total Count) = the total number of times our ad has been viewed
  • Clickthrough (%) = #clicks / #impressions
  • Messages Sent (%) = #conversations / #clicks
  • Consent Obtained (%) = #consents / #conversations
  • Core Survey Complete (%) = #forking section completed / #consents
  • Treatment Complete (%) = #treatment section completed / #forking section completed
  • Demo Questions Complete (%) = #demog section completed / #treatment section completed
  • Full Survey Complete (%) = #full chat completed / #demog section completed
  • Total characters elicited per completed survey (treatment) = average #character in best treatment explanation per full chat completed
  • Avg characters elicited per completed survey (impediment explanations) = average #character in impediment explanations per full chat completed
  • Cost per Impression = amount spent / #impressions (in USD)
  • Cost per Link Click = amount spent / #clicks (in USD)
  • Cost per Survey Complete (All participants) = amount spent / #full chat completed (in USD)
  • Cost per Survey Complete (Unvax) = amount spent / #full chat completed with unvaccinated participants (in USD)
  • Cost per Survey Complete (Unvax, Open to Treatment) = amount spent / #full chat completed with unvaccinated and open to treatment participants (in USD)

Table 1: Unnecessary vs Risky vs Inaccessible


This table compared three Ad impediment sources (vaccine is unnecessary vs vaccine is risky vs vaccine is inaccessible) in terms of the metrics described above.

Table 2: Control vs Airtime


This table compared two Ad body text approaches - control (share your opinion) vs (take a short survey and earn airtime) - in terms of the metrics described above.

Table 3: Image-wise Split


This table compared nine images (provided below the table) in terms of the metrics described above.

Images used
Image 1

Image 2

Image 3

Image 4

Image 5

Image 6

Image 7

Image 8

Image 9