Project Overview

Problem

COVID-19 vaccine hesitancy has been recognized as a problem across nations. In some of the world’s poorest nations, where vaccine access issues are being gradually resolved, a resistance to getting vaccinated is emerging as a major hurdle. Scientists fear that persistent pools of unvaccinated people around the world will present a greater risk for the emergence of new variants of concern, like Omicron. Addressing people’s hesitancy is therefore crucial to curb the spread of COVID-19, and to consequently avert hospitalizations and death.

Objectives

We aim to understand why people are hesitant about getting the COVID-19 vaccine. Hesitancy could not only occur within the unvaccinated population but also in a subset of people who already got vaccinated. Therefore, phase 1 of the project has the following objectives:

  1. Understand why people are hesitant to get the COVID-19 vaccines
  2. Understand ways to best elicit vaccine impediments from respondents
  3. Pinpoint what interventions will help people get vaccinated

Approach

We intend to use chatbot as a medium (on Facebook) to conduct conversations with people to understand how we can best achieve the above three objectives. We have run five pilots as of August 30 2022, – 2 in the United States using Qualtrics on Lucid, and 3 in South Africa on Facebook. The eventual goal will be running this using multiple chatbots that enable the conversation to flow more naturally than in a survey format.

We hypothesize that respondents are more likely to respond to our sensitive questions around vaccine hesitancy if the questions are asked more casually in an open stress-free setting. Therefore, in all version of the pilots, we make our tone as causal as possible (using emojis, GIFs, emphatic prompts) and include delays in the appearance of questions (and empathetic responses after each question) to make the conversation more authentic.


Current Pilot

This analysis is based on 4351 respondents who completed the current pilot survey wave and are randomized. We aim to compare the performance across four different themes (fear, unnecessary, risky, inaccessible) for each target audience (Ghana, Kenya, Nigeria, South Africa, South Africa - lookalike), built upon hypothesized drivers of hesitancy, as well as different versions of creative within those themes.

  1. Goal that specific to this pilot: In this pilot, we will test out the lookalike audiences functionality (recommended by our Facebook expert) to test how useful it is. Given we only have data in South Africa, we will be only testing this with the South Africa ad sets. If this strategy proves successful in the pilot, we will implement for all countries in the main experiment using the pilot data for the other three countries.
    • Given we are mainly interested in the unvaccinated population, we created a segment on Chatfuel that only contains unvaccinated participants from our previous pilot data. The resulting segment is 3284 participants.
    • According to Facebook’s estimate, a 1% lookalike audience based on the segment we created is approximately 384,000 people, which is about 1% of the entire South Africa Facebook population (23.9M - 28.1M by Facebook estimate). Thus, we will use this 1% lookalike audience as our targeting audience for this pilot for the South Africa ad sets. We will also run an ad set that uses the standard targeting (18+ English-speaking).

  2. Learning Goal: To come away with descriptive segments and generalizable findings about the effectiveness of treatments on those segments, we’d ideally randomly the same population for the duration of our study.

  3. Viability Goal: We will need to drive for the lowest possible cost per participants in order to collect as many responses as possible within our budget. Our past pilots have relied on Facebook’s optimization to make progress towards this goal.

Overview of the Ads Strategy

Setting Principle:

  • One campaign covering all ad sets in Ghana, Kenya, Nigeria, South Africa, and South Africa-lookalike
  • 20 ad sets (different combinations of theme, country), but end up with 35 ad sets in pilot v8 due to the uploading and technical issue
  • Ad set optimization is on, so it will select the best ad within the ad set.
  • If ad sets work well and we want to scale, we duplicate the ad set within the campaign and set a new budget
  • We don’t duplicate campaigns (only ad sets), and we don’t change budgeting once it is initially set (if we want to scale, we duplicate more ad sets).


Ad Performance will be measured using the following indicators:

  • Click-through rate
  • Recruitment of non-vaccinated but potentially treatable participants
    • Total quantity
    • Percent of total participants recruited
    • Retention
  • Average participant elicitation
  • Cost
    • per impression
    • per Link Click
    • per Survey Complete

Setup

Detailed setting for Pilot v8 can be found here.

  • 1 campaign
  • 35 ad sets (was planned to be 20, but uploading and technical issue happens), 3 ads in each ad set
  • 105 ads split into:
    • 4 impediment themes (15 fear, 30 inaccessible, 30 risky, 30 unnecessary)
    • 5 different targeting audience (21 Ghana, 21 Kenya, 21 Nigeria, 21 South Africa, and 21 South Africa-lookalike)
    • 9 different images (Images 1,2,8 used 25 times, images 3-7,9 used 5 times)

Analysis

The ad analysis contains 3 tables using different combinations of 15 distinct ads:

  • split by 4 ad impediment themes (fear, risky, unnecessary, inaccessible)
  • split by 5 ad target audience (South Africa-lookalike, South Africa, Kenya, Nigeria, Ghana)
  • split by 9 ad images

Metrics explanation:

  • Impressions (Total Count) = the total number of times our ad has been viewed
  • Clickthrough (%) = #clicks / #impressions
  • Messages Sent (%) = #conversations / #clicks
  • Consent Obtained (%) = #consents / #conversations
  • Core Survey Complete (%) = #forking section completed / #consents
  • Treatment Complete (%) = #treatment section completed / #forking section completed
  • Demo Questions Complete (%) = #demog section completed / #treatment section completed
  • Full Survey Complete (%) = #full chat completed / #demog section completed
  • Total characters elicited per completed survey (treatment) = average #character in best treatment explanation per full chat completed
  • Avg characters elicited per completed survey (impediment explanations) = average #character in impediment explanations per full chat completed
  • Cost per Impression = amount spent / #impressions (in USD)
  • Cost per Link Click = amount spent / #clicks (in USD)
  • Cost per Survey Complete (All participants) = amount spent / #full chat completed (in USD)
  • Cost per Survey Complete (Unvax) = amount spent / #full chat completed with unvaccinated participants (in USD)
  • Country Accuracy(only for country-wise split analysis) = #users that entered survey from correct country/ #users that entered survey through the country-specific ads

Table 1: Fear vs Unnecessary vs Risky vs Inaccessible


This table compared four Ad impediment sources (vaccine is fearful vs vaccine is risky vs vaccine is unnecessary vs vaccine is inaccessible) in terms of the metrics described above.

Table 2: Country-wise Split


This table compared 5 targeting audience - South Africa-lookalike vs South Africa vs Kenya vs Nigeria vs Ghana - in terms of the metrics described above.

Table 3: Image-wise Split


This table compared nine images (provided below the table) in terms of the metrics described above. Note: 2 Image5 Ads failed to run; 1 Image9 Ad failed to run, 5 Image2 Ads failed to run, 24 Image1 Ads been rejected by FB in the second wave.

Images used
Image 1

Image 2

Image 3

Image 4

Image 5

Image 6

Image 7

Image 8

Image 9

Takeaways and Next Steps

1. Almost all ads with image 1 have been rejected by Facebook in the second wave of this campaign. It may be because it has a “thumbs up” button on the image

\(~\)Next Steps:

  • Edit out the “thumbs up” from image 1 OR
  • Drop this image and find a new one

2. South Africa-lookalike Ads have a higher cost in making impressions from Fb Ads Manager, but it has better funnel performance and lower cost in recruiting unvaccinated people than normal South Africa ads ($3.373 vs $3.52).

\(~\)Next Steps:

  • Reach out to an FB expert to discuss the high cost of impressions. (Like will the cost per impression be lower if we have a larger segment?) Discuss the tradeoff between high recruiting costs and better funnel performance.

3. The new theme, fear, has the highest cost per survey complete (unvax) with average funnel performance

\(~\)Next Steps:

  • Discuss whether will we retain this theme or not.

4. Ads targeting Ghana have the lowest clickthrough rate, worse funnel performance, and higher cost than the other two new countries.

\(~\)Next Steps:

  • Find and test the best-performed Ads configuration for Ghana (maybe talk with local ads expert). Look into the funnel analysis of Ghana.