Treatment Analysis

Here we estimate we are the treatment effect for each treatment while comparing between Pilot 4B and Pilot 5.

Note that the treatments here are proxy treatments, where respondents select what is the best treatment to convince them to get vaccinated.

Main Takeaway

Rewards for vaxxing have the highest treatment effect in pre-defined treatments followed by trust source / people / transparency in both versions. Many people have selected other treatment(s) in pilot 5, which generate the highest treatment effect.

By Version

Funnel & Engagement Analysis

Goals for this section:

  • Understand variation in survey completion rates across pilots/vaccination status
  • Understand cost effectiveness across pilots
  • Assess engagement and comfort across pilots

List of analyses in this section:

  • Funnel dropoff analysis
    • Summary of the variation of funnel dropoff with pilot version
    • Funnel Dropoff by Fork Path with pilot version analysis (separated into Vax and Unvax group)
    • Variation of cost-per-action with pilot version analysis
  • Engagement Analysis
    • Variation of enjoyment with pilot version analysis
    • Variation of comfort with pilot version analysis

Funnel Analysis

Funnel Dropoff Summary

Funnel Dropoff by Fork Path Analysis

Variation of Cost-per-Action Analysis

Engagement Analysis

Variation in enjoyment level

Variation in comfort level

Covariate Analysis

Goals for this section:

  • Understanding the variation in impediments to vaccination faced by our respondents
  • Investigate if these impediments have strong associations with demographic characteristics

Main Takeaway

We see a shift in percentage of vaccinated population (37.94% -> 57.3%) and a shift in impediments, reflecting South Africa’s current situation after Omicron and people gradually growing into a state that cares more about the benefits of vaccines.

+ Top Motivational Impediment: risk (45.42%) -> time (49/06%)
+ Top Ability Impediment: time (44.12%) -> time (66.19%)

Vaccination status

We start off with a plot demonstrating the distribution of vaccinated and unvaccinated participants in this pilot:

Impediments

Overview

We asked participants about whether they have the motivation to get the COVID-19 vaccine and whether they have the ability to get the vaccine for both vaccinated and unvaccinated people. We then fork them into 8 different segments based on the vaccination status, motivation to get the vaccine, and ability to get the vaccine. We obtain the distribution below:

Pilot 4B
Distribution of forking segments of participants’ impediments
Vaccination status Able to get vaccine Have motivation to get vaccine Count Percentage of total participants
unvax yes no 194 49%
unvax yes yes 27 7%
unvax no no 15 4%
unvax no yes 11 3%
vax yes yes 80 20%
vax yes no 64 16%
vax no yes 6 2%
vax no no 1 0%
Pilot 5
Distribution of forking segments of participants’ impediments
Vaccination status Able to get vaccine Have motivation to get vaccine Count Percentage of total participants
unvax yes no 438 26%
unvax yes yes 151 9%
unvax no no 66 4%
unvax no yes 32 2%
vax yes yes 610 36%
vax yes no 341 20%
vax no no 31 2%
vax no yes 14 1%

Analysis for each impediment

Let’s investigate each impediment (motivation & ability) in detail, and see the distributions of the reasons why they have such impediments.

  • Main Level: demonstrates the distribution of the reasons that chosen from the options we provided to them to explain why participants have motivation/ability impediments
  • Sub-level: analyze the explanations why participants choose the specific reason in the Main level

Takeaways

South Africa’s current situation after Omicron is reflected here as people gradually growing into a state that cares more about the benefits of vaccines.

  • Top Motivational Impediment: risk (45.42%) -> time (49/06%)
    • Top Ability Impediment: time (44.12%) -> time (66.19%)
Motivational (Main Level)

We asked: What’s the main reason you don’t want to be vaccinated?

Provided options:

  • there’s no clear benefit (benefit)
  • it’s too risky (risk)
  • against my beliefs (belief)

The distribution of the answers demonstrates below:

Motivational (Sub-Level)
Benefit

We asked: is there a main reason why you think there isn’t a benefit?

Provided options:

  • covid not dangerous
  • unlikely to get sick
  • had covid already
  • i can recover
  • other reason(s)

The distribution of the answers demonstrates below:

Risk

We asked: is there a main reason why you think there is risk?

Provided options:

  • don’t trust pharma
  • vaccines don’t work
  • bad side effects
  • needles/pain
  • other reason(s)

The distribution of the answers demonstrates below:

Belief

We asked: is there a main reason why against your belief?

Provided options:

  • don’t trust gov
  • religious reasons
  • freedom to choose
  • other reason(s)

The distribution of the answers demonstrates below:

Ability (Main Level)

We asked: What’s the main difficulty of getting vaccinated?

Provided options:

  • no vax available (availability)
  • lack of time (time)
  • lack of money (money)

The distribution of the answers demonstrates below:

Ability (Sub-Level)
Availability

We asked: is there a main reason why there isn’t availability?

Provided options:

  • too far away
  • no vaccines left
  • other reason(s)

Time

We asked: is there a main reason why there isn’t time?

Provided options:

  • no time to research
  • work
  • family
  • keep forgetting
  • other reason(s)

Money

We asked: is there a main reason why there isn’t money?

Provided options:

  • travel costs
  • no insurance
  • no cost

Demographics

We mapped binary and ordinal demographic variables to continuous variables (with value 0, 1, 2,…).

How we did the mapping:

  • ability: 1 if the participant has the ability to get vax, 0 if not
  • female: 1 if female, 0 if male
  • country: 1 if live in South Africa, 0 if not
  • income: 0 if the participant is unemployed, 1 if income < R5,000, 2 if income in R5,000 – R9,999, …, 6 if income > R100,000
  • education: 1 if the participant’s education < high school, 2 if education is high school, …, 6 if education is a graduate degree
  • religiosity: 1 if the participant is not very religious, 2 if somewhat religious, 3 if very religious
  • politics: 1 if the participant is conservative, 2 if moderate, 3 if liberal
  • location: 1 if the participant lives in rural, 2 if suburban, 3 if urban,
  • white: 1 if the participant is a white or caucasian, 0 if not

Here we run a regression on covariate balance with the goal of understanding whether the demographics are different in Pilot 4B and Pilot 5. Difference in estimate confidence interval covering 0 means that there is no significant difference in the demographic variable of interest between the two versions; on the other hand, if the confidence interval does not cover 0, this means that there is some evidence supporting that the demographics being collected in the two versions are different.

Takeaway

We do not see significant differences in demographic variables between two versions

General Analysis

Main Takeaways

Pilot Version 5 is doing significantly better for free text response elicitation in terms of number of characters being elicited.

  • All free text response questions: pilot 4 (38.19) -> pilot 5 (42.25) p = 0.001
  • Treatment related free text response questions: pilot 4 (34.85) -> pilot 5 (48.44) p < 0.001
  • Impediments related free text response questions: pilot 4 (57.12) -> pilot 5 (64.12) p = 0.015

Free text elaboration Distribution

Density Plot (Overall)

Density Plot (By Vaccination Group)

Statistics (Overall)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Here the statistics are used to analyze and compare the free text response distribution in different pilot versions, which is a supplement to the regression analysis.

Statistics (By Vaccination Group)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Regression Analysis (Overall)

  nchar
Predictors Estimates CI p
(Intercept) 38.19 36.22 – 40.16 <0.001
version Pilot 5 3.52 1.26 – 5.79 0.002
Observations 10314
R2 / R2 adjusted 0.001 / 0.001

Regresssion Analysis (By Vacccintaion Group)

  nchar
Predictors Estimates CI p
(Intercept) 36.14 33.66 – 38.61 <0.001
version Pilot 5 4.47 1.41 – 7.54 0.004
vax status [Vaccinated] 5.59 1.50 – 9.68 0.007
version Pilot 5 * vax
status [Vaccinated] 		-3.80 -8.49 – 0.88 0.112
Observations 10314
R2 / R2 adjusted 0.002 / 0.002

Free text analyses comparison on best treatment

Density Plot (Overall)

Density Plot (By Vaccination Group)

Statistics (Overall)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Here the statistics are used to analyze and compare the free text response distribution in different pilot versions, which is a supplement to the regression analysis.

Statistics (By Vaccination Group)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Regression Analysis (Overall)

  nchar
Predictors Estimates CI p
(Intercept) 34.85 28.83 – 40.86 <0.001
version Pilot 5 12.67 6.04 – 19.30 <0.001
Observations 1508
R2 / R2 adjusted 0.009 / 0.009

Regresssion Analysis (By Vacccintaion Group)

  nchar
Predictors Estimates CI p
(Intercept) 28.45 20.39 – 36.50 <0.001
version Pilot 5 20.60 11.38 – 29.82 <0.001
vax status [Vaccinated] 14.43 2.34 – 26.52 0.019
version Pilot 5 * vax
status [Vaccinated] 		-16.90 -30.28 – -3.53 0.013
Observations 1508
R2 / R2 adjusted 0.013 / 0.011

Free text analyses comparison on Impediment

Density Plot (Overall)

Density Plot (By Vaccination Group)

Statistics (Overall)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Statistics (By Vaccination Group)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Regression Analysis (Overall)

  nchar
Predictors Estimates CI p
(Intercept) 57.12 52.71 – 61.54 <0.001
version Pilot 5 6.32 0.85 – 11.80 0.024
Observations 1930
R2 / R2 adjusted 0.003 / 0.002

Regresssion Analysis (By Vacccintaion Group)

  nchar
Predictors Estimates CI p
(Intercept) 58.94 53.97 – 63.91 <0.001
version Pilot 5 7.41 0.96 – 13.85 0.024
vax status [Vaccinated] -8.53 -19.29 – 2.23 0.120
version Pilot 5 * vax
status [Vaccinated] 		0.89 -11.77 – 13.55 0.890
Observations 1930
R2 / R2 adjusted 0.007 / 0.005

Ads Analysis

Ad-related A|B analyses are being generated through a separate script here.