Treatment Analysis

Takeaway

By Version

Funnel & Engagement Analysis

Goals for this section:

  • Understand variation in survey completion rates across pilots/vaccination status
  • Understand cost effectiveness across pilots
  • Assess engagement and comfort across pilots

List of analyses in this section:

  • Funnel dropoff analysis
    • Summary of the variation of funnel dropoff with pilot version
    • Funnel Dropoff by Fork Path with pilot version analysis (separated into Vax and Unvax group)
    • Variation of cost-per-action with pilot version analysis
  • Engagement Analysis
    • Variation of enjoyment with pilot version analysis
    • Variation of comfort with pilot version analysis

Funnel Analysis

Funnel Dropoff Summary

## Error: Problem with `filter()` input `..1`.
## ℹ Input `..1` is `ad_set %in% c("pilot_v4_1", "pilot_v4_2", "pilot_v4_3")`.
## x object 'ad_set' not found
## Error in eval(expr, envir, enclos): object 'list_C' not found
## Error in eval(expr, envir, enclos): object 'list_C' not found
## Error in is.data.frame(x): object 'df_full_C' not found
## Error in list2(...): object 'dropoff_old' not found

Funnel Dropoff by Fork Path Analysis

Variation of Cost-per-Action Analysis

Engagement Analysis

## Error in filter(., full_complete == "yes"): object 'df_full_C' not found

Variation in enjoyment level

## Error: Can't subset columns that don't exist.
## x Column `enjoyable` doesn't exist.

Variation in comfort level

Covariate Analysis

Goals for this section:

  • Understanding the variation in impediments to vaccination faced by our respondents
  • Investigate if these impediments have strong associations with demographic characteristics

Vaccination status

We start off with a plot demonstrating the distribution of vaccinated and unvaccinated participants in this pilot:

Impediments

Overview

We asked participants about whether they have the motivation to get the COVID-19 vaccine and whether they have the ability to get the vaccine for both vaccinated and unvaccinated people. We then fork them into 8 different segments based on the vaccination status, motivation to get the vaccine, and ability to get the vaccine. We obtain the distribution below:

Pilot 4B
Distribution of forking segments of participants’ impediments
Vaccination status Able to get vaccine Have motivation to get vaccine Count Percentage of total participants
unvax yes no 193 49%
unvax yes yes 27 7%
unvax no no 15 4%
unvax no yes 11 3%
vax yes yes 80 20%
vax yes no 63 16%
vax no yes 6 2%
vax no no 1 0%
Pilot 5
Distribution of forking segments of participants’ impediments
Vaccination status Able to get vaccine Have motivation to get vaccine Count Percentage of total participants
unvax yes no 403 %
unvax yes yes 126 %
unvax no no 62 %
unvax no yes 26 %
vax yes yes 500 %
vax yes no 291 %
vax no no 25 %
vax no yes 11 %

Takeaways

Analysis for each impediment

Let’s investigate each impediment (motivation & ability) in detail, and see the distributions of the reasons why they have such impediments.

  • Main Level: demonstrates the distribution of the reasons that chosen from the options we provided to them to explain why participants have motivation/ability impediments
  • Sub-level: analyze the explanations why participants choose the specific reason in the Main level

Takeaways

Motivational (Main Level)

We asked: What’s the main reason you don’t want to be vaccinated?

Provided options:

  • there’s no clear benefit (benefit)
  • it’s too risky (risk)
  • against my beliefs (belief)

The distribution of the answers demonstrates below:

## $x
## [1] "Motivational Impediment"
## 
## $y
## [1] "count"
## 
## attr(,"class")
## [1] "labels"
Motivational (Sub-Level)
Benefit

We asked: is there a main reason why you think there isn’t a benefit?

Provided options:

  • covid not dangerous
  • unlikely to get sick
  • had covid already
  • i can recover
  • other reason(s)

The distribution of the answers demonstrates below:

Risk

We asked: is there a main reason why you think there is risk?

Provided options:

  • don’t trust pharma
  • vaccines don’t work
  • bad side effects
  • needles/pain
  • other reason(s)

The distribution of the answers demonstrates below:

Belief

We asked: is there a main reason why against your belief?

Provided options:

  • don’t trust gov
  • religious reasons
  • freedom to choose
  • other reason(s)

The distribution of the answers demonstrates below:

Ability (Main Level)

We asked: What’s the main difficulty of getting vaccinated?

Provided options:

  • no vax available (availability)
  • lack of time (time)
  • lack of money (money)

The distribution of the answers demonstrates below:

Ability (Sub-Level)
Availability

We asked: is there a main reason why there isn’t availability?

Provided options:

  • too far away
  • no vaccines left
  • other reason(s)

Time

We asked: is there a main reason why there isn’t time?

Provided options:

  • no time to research
  • work
  • family
  • keep forgetting
  • other reason(s)

Money

We asked: is there a main reason why there isn’t money?

Provided options:

  • travel costs
  • no insurance
  • no cost

Demographics

We mapped binary and ordinal demographic variables to continuous variables (with value 0, 1, 2,…).

How we did the mapping:

  • ability: 1 if the participant has the ability to get vax, 0 if not
  • female: 1 if female, 0 if male
  • country: 1 if live in South Africa, 0 if not
  • income: 0 if the participant is unemployed, 1 if income < R5,000, 2 if income in R5,000 – R9,999, …, 6 if income > R100,000
  • education: 1 if the participant’s education < high school, 2 if education is high school, …, 6 if education is a graduate degree
  • religiosity: 1 if the participant is not very religious, 2 if somewhat religious, 3 if very religious
  • politics: 1 if the participant is conservative, 2 if moderate, 3 if liberal
  • location: 1 if the participant lives in rural, 2 if suburban, 3 if urban,
  • white: 1 if the participant is a white or caucasian, 0 if not

General Analysis

Free text elaboration Distribution

Density Plot (Overall)

Density Plot (By Vaccination Group)

Statistics (Overall)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Statistics (By Vaccination Group)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

##    version   vax_status    N Missing  Mean    SD Min Q1 Median Q3 Max
## 1 Pilot 4B   Vaccinated  910    3890 41.73 44.81   2 12     28 55 333
## 2 Pilot 4B Unvaccinated 1566    6306 35.96 47.81   1  9     20 43 430
## 3  Pilot 5   Vaccinated 3978   17550 43.11 44.87   1 13     30 59 458
## 4  Pilot 5 Unvaccinated 2663   13379 40.96 60.37   1  4     22 54 784

Regression Analysis (Overall)

## 
## Call:
## lm(formula = nchar ~ version, data = free_text_combined)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -41.25 -31.08 -16.08  13.92 741.75 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      38.080      1.013  37.609  < 2e-16 ***
## versionPilot 5    4.167      1.186   3.512 0.000446 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 50.38 on 9115 degrees of freedom
##   (41125 observations deleted due to missingness)
## Multiple R-squared:  0.001352,   Adjusted R-squared:  0.001242 
## F-statistic: 12.34 on 1 and 9115 DF,  p-value: 0.0004464

Regresssion Analysis (By Vacccintaion Group)

## 
## Call:
## lm(formula = nchar ~ version + vax_status + version * vax_status, 
##     data = free_text_combined)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -42.11 -31.11 -15.96  13.89 743.04 
## 
## Coefficients:
##                                     Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                           35.961      1.273  28.259  < 2e-16 ***
## versionPilot 5                         4.996      1.604   3.115  0.00184 ** 
## vax_statusVaccinated                   5.764      2.099   2.746  0.00604 ** 
## versionPilot 5:vax_statusVaccinated   -3.612      2.449  -1.475  0.14022    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 50.36 on 9113 degrees of freedom
##   (41125 observations deleted due to missingness)
## Multiple R-squared:  0.002496,   Adjusted R-squared:  0.002168 
## F-statistic: 7.601 on 3 and 9113 DF,  p-value: 4.499e-05

Free text analyses comparison on best treatment

Density Plot (Overall)

Density Plot (By Vaccination Group)

Statistics (Overall)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Statistics (By Vaccination Group)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

##    version   vax_status   N Missing  Mean    SD Min Q1 Median Q3 Max
## 1 Pilot 4B   Vaccinated 117      33 42.90 41.26   2 12     26 67 186
## 2 Pilot 4B Unvaccinated 148      98 28.51 37.55   1  9     17 35 264
## 3  Pilot 5   Vaccinated 635     193 47.21 45.16   1 16     34 63 314
## 4  Pilot 5 Unvaccinated 418     199 50.30 58.41   1 14     34 62 512

Regression Analysis (Overall)

## 
## Call:
## lm(formula = nchar ~ version, data = free_text_combined)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -47.44 -29.44 -14.44  13.99 463.56 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      34.864      2.999  11.626  < 2e-16 ***
## versionPilot 5   13.572      3.355   4.045 5.53e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 48.82 on 1316 degrees of freedom
##   (523 observations deleted due to missingness)
## Multiple R-squared:  0.01228,    Adjusted R-squared:  0.01153 
## F-statistic: 16.36 on 1 and 1316 DF,  p-value: 5.532e-05

Regresssion Analysis (By Vacccintaion Group)

## 
## Call:
## lm(formula = nchar ~ version + vax_status + version * vax_status, 
##     data = free_text_combined)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -49.30 -30.30 -14.30  13.79 461.70 
## 
## Coefficients:
##                                     Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                           28.514      4.006   7.118 1.79e-12 ***
## versionPilot 5                        21.783      4.661   4.673 3.27e-06 ***
## vax_statusVaccinated                  14.384      6.028   2.386  0.01717 *  
## versionPilot 5:vax_statusVaccinated  -17.470      6.765  -2.582  0.00992 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 48.73 on 1314 degrees of freedom
##   (523 observations deleted due to missingness)
## Multiple R-squared:  0.0173, Adjusted R-squared:  0.01505 
## F-statistic: 7.709 on 3 and 1314 DF,  p-value: 4.17e-05

Free text analyses comparison on Impediment

Density Plot (Overall)

Density Plot (By Vaccination Group)

Statistics (Overall)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

Statistics (By Vaccination Group)

Note that here we aggregate free text responses from all respondents.Therefore, N + missing should equal to number of respondents * number of free text response questions, and missing means that respondent did not encounter one of the free text questions (either they chose another option that did not need free text response or they chose another path that would not encounter some free text questions).

##    version   vax_status   N Missing  Mean    SD Min Q1 Median Q3 Max
## 1 Pilot 4B   Vaccinated 142    1508 50.27 44.02   2 23   37.5 65 264
## 2 Pilot 4B Unvaccinated 530    2176 58.83 57.58   1 21   40.5 76 430
## 3  Pilot 5   Vaccinated 395    6229 59.44 46.53   2 26   50.0 81 297
## 4  Pilot 5 Unvaccinated 697    4239 66.78 67.87   1 25   50.0 85 713

Regression Analysis (Overall)

## 
## Call:
## lm(formula = nchar ~ version, data = free_text_combined)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -63.12 -38.02 -15.12  18.20 648.88 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      57.022      2.271   25.11   <2e-16 ***
## versionPilot 5    7.102      2.887    2.46    0.014 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 58.87 on 1762 degrees of freedom
##   (14152 observations deleted due to missingness)
## Multiple R-squared:  0.003424,   Adjusted R-squared:  0.002858 
## F-statistic: 6.054 on 1 and 1762 DF,  p-value: 0.01397

Regresssion Analysis (By Vacccintaion Group)

## 
## Call:
## lm(formula = nchar ~ version + vax_status + version * vax_status, 
##     data = free_text_combined)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -65.78 -37.53 -14.83  18.22 646.22 
## 
## Coefficients:
##                                     Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                           58.832      2.554  23.033   <2e-16 ***
## versionPilot 5                         7.947      3.389   2.345   0.0191 *  
## vax_statusVaccinated                  -8.564      5.556  -1.541   0.1234    
## versionPilot 5:vax_statusVaccinated    1.226      6.677   0.184   0.8544    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 58.8 on 1760 degrees of freedom
##   (14152 observations deleted due to missingness)
## Multiple R-squared:  0.00698,    Adjusted R-squared:  0.005287 
## F-statistic: 4.124 on 3 and 1760 DF,  p-value: 0.00633

Ads Analysis

Goals for this section:

  • Understand the variation in demographics distribution across pilots
  • Provide an overview on the performance of current Ads based on various metrics
  • Compare the Ads performance across three different themes, built upon hypothesized drivers of hesitancy, as well as different versions of creative within those themes.

Demographics and Ads

Age

Age and Impression count

Age and Conversation Started count

Gender

Gender and Impression count

Gender and Conversation Started count

Region

Region and Impression count

Region and Conversation Started count

Ad Performance Summary

Metrics explanation:

Impressions (Total Count)

the total number of times our ad has been viewed

Impressions (Total Count)

the total number of times our ad has been viewed

Clickthrough (%)

number of clicks / number of impressions

Messages Sent (%)

number of conversations / number of clicks

Core Survey Complete (%)

number of forking section completed / number of consents

Treatment Complete (%)

number of treatment section completed / number of forking section completed

Demo Questions Complete (%)

number of demog section completed / number of treatment section completed

Full Survey Complete (%)

number of full chat completed / number of demog section completed

Total characters elicited per completed survey (treatment)

average number of character in best treatment explanation per full chat completed

Avg characters elicited per completed survey (impediment explanations)

average number of character in impediment explanations per full chat completed

Cost per Impression

amount spent / number of impressions (in USD)

Cost per Survey Complete (All participants)

amount spent / number of full chat completed (in USD)

Cost per Survey Complete (Unvax)

amount spent / number of full chat completed with unvaccinated participants (in USD)

Cost per Survey Complete (Unvax, Open to Treatment)

amount spent / number of full chat completed with unvaccinated and open to treatment participants (in USD)

Side-by-Side Chart on Key Metrics

Extreme VS Moderate

Compare by Image

This table compared nine images (provided below the table) in terms of the metrics described above.

Unnecessary vs Unsafe vs Inaccessible

This table compared three Ad impediment sources (vaccine is unnecessary vs vaccine is risky vs vaccine is inaccessible) in terms of the metrics described above.

Survey vs Control vs Airtime

This table compared three Ad body text approaches - control (share your opinion) vs airtime (take a short survey and earn airtime) vs survey (take this short survey)- in terms of the metrics described above.