Current Pilot

This analysis is based on 2450 respondents who completed the current pilot survey wave (version 5), and 551 respondents who completed the last pilot survey wave (version 4b).

Setup

Detailed setting can be found here.

  • 1 campaign
  • 5 ad sets, 3 ads in each ad set
  • 15 ads split into:
    • 3 impediment themes (3 inaccessible, 6 risky, 6 unnecessary)
    • 2 different prompts (6 control and 9 airtime)
    • 9 different images (Images 1-6 used twice, images 7-9 used once)

Overview of the Ads Strategy

Ad Performance will be measured using the following indicators:

  • Click-through rate
  • Recruitment of non-vaccinated but potentially treatable participants
    • Total quantity
    • Percent of total participants recruited
    • Retention
  • Average participant elicitation
  • Cost
    • per impression
    • per Link Click
    • per Survey Complete

Goals for this analysis script (version 5):

  • See whether Facebook A|B testing is a “true” randomized A|B test, and look into whether the best Ad set selected by Facebook aligns with the Ad set selected by our key metrics
  • Provide an overview on the performance of current Ads based on various metrics and compare the Ads performance between version 4b and version 5 in terms of the key cost metrics
  • Evaluate the strategy of using airtime text to recruit by estimating the advantages and disadvantages of mentioning airtime in our Ads
  • Analysis on Ad targeting: find demographic covariates that would predict unvax, open to treatment participants
  • Understand the variation in demographics distribution across pilots
  • Compare the Ads performance across three different themes, built upon hypothesized drivers of hesitancy, as well as different versions of creativity within those themes.

Goals of this pilot with ads:

  • Recruit full survey completes for unvaxxed, open to treatment participants at a low cost while maintaining participant response quality
  • Take the learnings here and apply them to our next large pilot

Key Takeaways:

  • FB’s A|B testing is NOT a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets)
  • FB’s A|B testing can pick the best Ad set, which is the best-performed Ad set based on our key cost metric when we look at the bottom of the funnel
  • When we adjust for the quality of responses (e.g., elicitation, time spent in survey, nonsense answers), airtime ads still outperform our control when using our key cost metric
  • Airtime participants will not skew responses
  • The proportion of females in Ads impressions is keep increasing, from 53% to 62%
  • The proportion of people with age > 45 in the conversation started is keep decreasing, from 40% to 20%
  • Most of the impressions/conversations started are made by young people (age: 18-34)
  • In Side-by-Side comparisons, unnecessary outperformed risk and inaccessible; airtime outperformed control; image 1 outperformed all other images

FB A|B testing algorithm analysis

Takeaways:

  • FB’s A|B testing can pick the best Ad set, which is the best-performed Ad set based on our key cost metric when we look at the bottom of the funnel
  • FB’s A|B testing is NOT a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets)

Winning Ad set comparison

To understand Facebook’s A|B test capabilities to draw conclusions of which ad performed best, we are going to see whether the winning Ad set selected by Facebook matches the winning Ad set selected by our bottom of the funnel key metrics.

  • Facebook A|B test selected pilot_v5_unnecessary_airtime
  • The metric table below suggests us pilot_v5_unnecessary_airtime is the best based on our funnel analysis

Conclusion:

  • FB’s winning Ad set matches the winning Ad set selected by our metrics. We can make a preliminary determination that Facebook’s A|B test can draw conclusions of which ad performed best

  • Interpretation: FB A|B test divides our budget to equally and randomly divide exposure between each version of our Ad sets and chooses the most cost-efficient Ad set as the winner, which aligns with our cost metrics. i.e. Given a budget, the more Ad audiences started the conversion, the more participants (that are unvaccinated and open to treatment) we can obtain, hence more cost-efficient when we look at the key cost metrics at the bottom of the funnel.

Demographic variables distribution

We are interested in:

  • is FB A|B testing a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets), OR
  • is it a different comparative test (FB divides up the total population into five equal samples, then runs an algorithm to identify attractive target populations to maximize cost efficiency on each sample)?

Metric:

  • if it is a true A|B test, we should see roughly the same demographic distributions in each of the five ad sets.
  • if it isn’t, we should see very different demographic distributions in each of the five ad sets.

Findings:

  • Using the Facebook ads data on demographics, if we combine the 18-24 and 25 - 34 population group, the age distribution is roughly balanced across five ad sets
  • Similarly, region (as identified by FB) is also roughly balanced
  • However, gender is not balanced across five ad sets, with two ad sets having about 10% lower female ratio (~57.5%) than the other three ads sets (~67.5%), which might suggest we are hitting different populations.

Age

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Gender

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Region

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Age | Gender

Since we found imbalanced distribution of gender, we are interested in will the distribution of Age is conditional on gender. Based on the plots below, the answer is No.

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Ad Performance Summary Comparison

In this section, we are interested in:

  • Has our overall key cost metric (survey complete, unvaxxed, open to treatment) decreased for this pilot?
  • What were the drivers of this increase/decrease (and at which stages of the funnel)?

Takeaways:

  • The overall key cost metric has increased for this pilot (worse than version 4b)
  • The increasing cost could be caused by the A|B testing that we ran on FB Ads Manager. In the FB A|B test, all 5 ad sets have been controlled the same spending, instead of cost-benefit optimized, resulting in a lower clickthrough rate
  • So, we think we cannot compare the campaigns by simply looking at the key cost metric
  • Based on other metrics, we found the new pilot has better performance when we look at the drop-off rates of each section in the chatbot.

Metrics explanation:

  • Impressions (Total Count) = the total number of times our ad has been viewed
  • Clickthrough (%) = #clicks / #impressions
  • Messages Sent (%) = #conversations / #clicks
  • Consent Obtained (%) = #consents / #conversations
  • Core Survey Complete (%) = #forking section completed / #consents
  • Treatment Complete (%) = #treatment section completed / #forking section completed
  • Demo Questions Complete (%) = #demog section completed / #treatment section completed
  • Full Survey Complete (%) = #full chat completed / #demog section completed
  • Total characters elicited per completed survey (treatment) = average #character in best treatment explanation per full chat completed
  • Avg characters elicited per completed survey (impediment explanations) = average #character in impediment explanations per full chat completed
  • Cost per Impression = amount spent / #impressions (in USD)
  • Cost per Link Click = amount spent / #clicks (in USD)
  • Cost per Survey Complete (All participants) = amount spent / #full chat completed (in USD)
  • Cost per Survey Complete (Unvax) = amount spent / #full chat completed with unvaccinated participants (in USD)
  • Cost per Survey Complete (Unvax, Open to Treatment) = amount spent / #full chat completed with unvaccinated and open to treatment participants (in USD)

Airtime analysis

Our last pilot ads that mention airtime were a lot more cost-efficient in recruiting a large number of participants.

In this subsection, we are interested in:

  • When we adjust for the quality of responses (e.g., elicitation, time spent in survey, nonsense answers), do airtime ads still outperform our control when using our key cost metric (survey complete, unvaxxed, open to treatment)?

  • How does recruit airtime participants also skew responses

  • Should we continue to use airtime ads to recruit participants?

Takeaways:

  • After adjusting for the number of characters elicited or time spent in the survey, ads mentioning airtime are still a lot more cost-efficient.
  • When looking at the best treatment respondents select, the treatment selection distribution is roughly the same as the control group, suggesting financial incentives in ads do not translate to people selecting “rewards for vaxxing” in treatment.
  • We should continue to use airtime ads to recruit participants.

Adjusted cost analysis

  • We compared the cost per ten characters elicitation in treatment and impediment explanations between the airtime and control group.
  • We also compared the cost per minute spent in Survey Complete between the airtime and control groups.

In both comparisons, we found airtime is still a lot more cost-efficient.

Adjusted by character elicitation

Adjusted by time spent in survey

Skew responses analysis

We estimated the distribution of best treatment selected by participants recruited by airtime and participants recruited by control. We would like to see is there a skewed distribution of airtime’s. Based on the table below, the answer is NO.

Treatments

Ad Theme analysis

In our last and this pilot ads with an “unnecessary” theme were promoted by Facebook’s algorithm since the cost per messaging conversation is cheaper.

What we are interested in:

  • When we look at the bottom of the funnel using our key cost metric (survey complete, unvaxxed, open to treatment), do “unnecessary” ads still outperform “risky” and “inaccessible” ads?
  • If so, by how much?
  • How should we shift our ad creative using this new information?

Takeaway:

  • In version 5, we found “unnecessary” ads still outperform “risky” and “inaccessible” ads in all three metrics. “unnecessary” ads is 5%/15% more cost-efficient than “risky”/“inaccessible” ads in Cost per Survey Complete; 30%/45% more cost-efficient than “risky”/“inaccessible” ads in Cost per Unvax Survey Complete; 30%/33% more cost-efficient than “risky”/“inaccessible” ads in Cost per Unvax and Open to treatment Survey Complete
  • While in version 4b, we found “unnecessary” ads have the worst performance. “risky” ads are the best based on the cost metrics. Surprisingly, the (unvax, open to treatment survey completed) in “unnecessary” ads have a much higher cost (1.46) than that of in “unsafe” (0.662) and “inaccessible” (0.711).

Version 5

Takeaway: “unnecessary” theme is the best according to the key cost metrics

Version 4b

Takeaway: “unnecessary” theme is NOT the best

Ad Targeting analysis

In this sub-section, we are interested in:

  • What demographic configurations predict vaxxed participants (with a low chance of false positives)?
  • What demographic configurations predict participants who are not open to treatment (with a low chance of false positives)?

Takeaway:

  • Using data provided by Ads Manager, demographic configurations in Ads Manager include age, gender, and region. Based on the regression result, we found there are NO significant demographic variables (all with high pvalue).
  • Using data provided by Chatfuel, we found some significant demographic covariates and listed them in each tab.
  • We need more data and should consider more techniques to explore the targeting dimension

Hypotheses:

  • ethnicity [white or caucasian] -> not open to treatment

Predict vaxxed participants(from Ad configurations)

There is no significant covariate based on regression output.

Age

  vax status
Predictors Estimates CI p
(Intercept) 0.58 0.56 – 0.61 <0.001
Age25-34 0.00 -0.03 – 0.03 1.000
Age35-44 -0.00 -0.03 – 0.03 1.000
Age45-54 0.00 -0.03 – 0.03 1.000
Age55-64 0.00 -0.03 – 0.03 1.000
Age [65+] 0.00 -0.03 – 0.03 1.000
Observations 10308
R2 / R2 adjusted 0.000 / -0.000

Gender

  vax status
Predictors Estimates CI p
(Intercept) 0.58 0.56 – 0.61 <0.001
Gender [male] 0.00 -0.03 – 0.03 1.000
Observations 3436
R2 / R2 adjusted 0.000 / -0.000

Region

  vax status
Predictors Estimates CI p
(Intercept) 0.58 0.56 – 0.61 <0.001
Region [Free State] 0.00 -0.03 – 0.03 1.000
Region [Gauteng] 0.00 -0.03 – 0.03 1.000
Region [KwaZulu-Natal] -0.00 -0.03 – 0.03 1.000
Region [Limpopo] 0.00 -0.03 – 0.03 1.000
Region [Mpumalanga] 0.00 -0.03 – 0.03 1.000
Region [North West] -0.00 -0.03 – 0.03 1.000
Region [Northern Cape] -0.00 -0.03 – 0.03 1.000
Region [Unknown] 0.01 -0.04 – 0.05 0.734
Region [Western Cape] 0.00 -0.03 – 0.03 1.000
Observations 16173
R2 / R2 adjusted 0.000 / -0.001

Age * Gender

  vax status
Predictors Estimates CI p
(Intercept) 0.58 0.56 – 0.61 <0.001
Age25-34 -0.00 -0.03 – 0.03 1.000
Age35-44 -0.00 -0.03 – 0.03 1.000
Age45-54 -0.00 -0.03 – 0.03 1.000
Age55-64 -0.00 -0.03 – 0.03 1.000
Age [65+] -0.00 -0.03 – 0.03 1.000
Gender [male] -0.00 -0.03 – 0.03 1.000
Gender [unknown] -0.00 -0.03 – 0.03 1.000
Age25-34:Gendermale 0.00 -0.05 – 0.05 1.000
Age35-44:Gendermale 0.00 -0.05 – 0.05 1.000
Age45-54:Gendermale -0.00 -0.05 – 0.05 1.000
Age55-64:Gendermale 0.00 -0.05 – 0.05 1.000
Age [65+] * Gender [male] 0.00 -0.05 – 0.05 1.000
Age25-34:Genderunknown 0.00 -0.05 – 0.05 1.000
Age35-44:Genderunknown 0.00 -0.05 – 0.05 1.000
Age45-54:Genderunknown 0.01 -0.04 – 0.05 0.789
Age55-64:Genderunknown 0.01 -0.04 – 0.07 0.680
Age [65+] * Gender
[unknown] 		-0.00 -0.05 – 0.05 0.955
Observations 29734
R2 / R2 adjusted 0.000 / -0.001

Predict not open to treatment(from Ad configurations)

There is no significant covariate based on regression output.

Age

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Age25-34 0.00 -0.02 – 0.02 1.000
Age35-44 -0.00 -0.02 – 0.02 1.000
Age45-54 0.00 -0.02 – 0.02 1.000
Age55-64 -0.00 -0.02 – 0.02 1.000
Age [65+] -0.00 -0.02 – 0.02 1.000
Observations 10308
R2 / R2 adjusted 0.000 / -0.000

Gender

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Gender [male] -0.00 -0.02 – 0.02 1.000
Gender [unknown] 0.00 -0.02 – 0.02 1.000
Observations 5154
R2 / R2 adjusted 0.000 / -0.000

Region

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Region [Free State] 0.00 -0.02 – 0.02 1.000
Region [Gauteng] 0.00 -0.02 – 0.02 1.000
Region [KwaZulu-Natal] 0.00 -0.02 – 0.02 1.000
Region [Limpopo] -0.00 -0.02 – 0.02 1.000
Region [Mpumalanga] 0.00 -0.02 – 0.02 1.000
Region [North West] -0.00 -0.02 – 0.02 1.000
Region [Northern Cape] -0.00 -0.02 – 0.02 1.000
Region [Unknown] 0.01 -0.02 – 0.04 0.418
Region [Western Cape] -0.00 -0.02 – 0.02 1.000
Observations 16173
R2 / R2 adjusted 0.000 / -0.001

Age * Gender

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Age25-34 -0.00 -0.02 – 0.02 1.000
Age35-44 -0.00 -0.02 – 0.02 1.000
Age45-54 0.00 -0.02 – 0.02 1.000
Age55-64 -0.00 -0.02 – 0.02 1.000
Age [65+] -0.00 -0.02 – 0.02 1.000
Gender [male] -0.00 -0.02 – 0.02 1.000
Gender [unknown] -0.00 -0.02 – 0.02 1.000
Age25-34:Gendermale 0.00 -0.03 – 0.03 1.000
Age35-44:Gendermale 0.00 -0.03 – 0.03 1.000
Age45-54:Gendermale -0.00 -0.03 – 0.03 1.000
Age55-64:Gendermale 0.00 -0.03 – 0.03 1.000
Age [65+] * Gender [male] 0.00 -0.03 – 0.03 1.000
Age25-34:Genderunknown 0.00 -0.03 – 0.03 1.000
Age35-44:Genderunknown 0.00 -0.03 – 0.03 1.000
Age45-54:Genderunknown -0.00 -0.03 – 0.03 0.798
Age55-64:Genderunknown -0.00 -0.04 – 0.03 0.903
Age [65+] * Gender
[unknown] 		0.00 -0.03 – 0.03 0.926
Observations 29734
R2 / R2 adjusted 0.000 / -0.001

Predict vaxxed participants(chatfuel data)

We found:

  • income != [prefer not to say] * ethnicity != [prefer not to say] -> vaxxed
  • religion [other] * politics [moderate] * location [rural] -> vaxxed

Tree model

Sum model

  vax status
Predictors Estimates CI p
(Intercept) 0.71 0.36 – 1.06 <0.001
cv age -0.00 -0.00 – 0.00 0.531
ethnicity [black or
african] 		-0.12 -0.43 – 0.20 0.466
ethnicity [coloured] -0.13 -0.46 – 0.20 0.431
ethnicity [other] 0.41 -0.35 – 1.17 0.292
ethnicity [prefer not to
say] 		-0.25 -0.61 – 0.12 0.186
ethnicity [white or
caucasian] 		-0.15 -0.48 – 0.18 0.362
income [> R100,000] 0.02 -0.12 – 0.15 0.798
income [prefer not to
say] 		-0.12 -0.19 – -0.04 0.002
education [2-year degree] -0.18 -0.36 – -0.00 0.050
education [4-year degree] 0.04 -0.17 – 0.25 0.710
education [graduate
degree] 		-0.11 -0.30 – 0.08 0.262
education [high school] -0.07 -0.16 – 0.01 0.074
education [prefer not to
say] 		-0.14 -0.31 – 0.02 0.079
education [some college] -0.12 -0.22 – -0.02 0.014
religion [hinduism] 0.20 -0.19 – 0.60 0.317
religion [islam] -0.04 -0.27 – 0.19 0.727
religion [no religion] -0.06 -0.18 – 0.06 0.309
religion [other] -0.12 -0.31 – 0.06 0.179
religion [prefer not to
say] 		-0.05 -0.21 – 0.11 0.562
politics [liberal] -0.04 -0.18 – 0.09 0.518
politics [moderate] 0.01 -0.10 – 0.12 0.833
politics [prefer not to
say] 		-0.01 -0.11 – 0.09 0.904
location [rural] 0.10 -0.04 – 0.25 0.165
location [suburban] 0.10 -0.05 – 0.25 0.208
location [urban] 0.09 -0.05 – 0.24 0.213
Observations 928
R2 / R2 adjusted 0.045 / 0.019

Interaction model 1

  vax status
Predictors Estimates CI p
(Intercept) 0.80 0.37 – 1.23 <0.001
income [> R100,000] -0.03 -1.74 – 1.68 0.974
income [prefer not to
say] 		-0.80 -1.86 – 0.26 0.138
ethnicity [black or
african] 		-0.13 -0.57 – 0.31 0.559
ethnicity [coloured] -0.18 -0.69 – 0.32 0.475
ethnicity [other] 0.10 -1.36 – 1.56 0.893
ethnicity [prefer not to
say] 		-0.77 -2.16 – 0.63 0.280
ethnicity [white or
caucasian] 		0.10 -0.43 – 0.63 0.710
education [2-year degree] 0.77 -1.24 – 2.79 0.451
education [4-year degree] 0.10 -0.91 – 1.11 0.846
education [graduate
degree] 		0.10 -0.91 – 1.11 0.846
education [high school] 0.20 -0.61 – 1.01 0.627
education [prefer not to
say] 		0.10 -0.91 – 1.11 0.846
education [some college] 0.97 -0.67 – 2.60 0.247
income [> R100,000] *
ethnicity [black or
african] 		0.03 -1.64 – 1.69 0.975
income [prefer not to
say] * ethnicity [black
or african] 		0.66 -0.41 – 1.73 0.224
income [> R100,000] *
ethnicity [coloured] 		-0.11 -1.92 – 1.70 0.904
income [prefer not to
say] * ethnicity
[coloured] 		0.74 -0.40 – 1.88 0.201
income [> R100,000] *
ethnicity [prefer not to
say] 		0.10 -2.11 – 2.31 0.929
income [prefer not to
say] * ethnicity [prefer
not to say] 		1.43 -0.17 – 3.03 0.079
income [> R100,000] *
ethnicity [white or
caucasian] 		-0.10 -1.56 – 1.36 0.893
income [prefer not to
say] * ethnicity [white
or caucasian] 		0.10 -1.08 – 1.28 0.868
income [> R100,000] *
education [2-year degree] 		-0.54 -2.02 – 0.93 0.469
income [prefer not to
say] * education [2-year
degree] 		-0.14 -0.70 – 0.41 0.620
income [> R100,000] *
education [4-year degree] 		0.13 -1.50 – 1.76 0.877
income [prefer not to
say] * education [4-year
degree] 		0.70 -0.76 – 2.16 0.348
income [> R100,000] *
education [graduate
degree] 		-0.20 -1.58 – 1.17 0.770
income [prefer not to
say] * education
[graduate degree] 		-0.30 -1.76 – 1.16 0.687
income [> R100,000] *
education [high school] 		-0.54 -1.66 – 0.59 0.348
income [prefer not to
say] * education [high
school] 		0.30 -1.13 – 1.73 0.681
income [> R100,000] *
education [prefer not to
say] 		-0.87 -2.50 – 0.76 0.295
income [prefer not to
say] * education [prefer
not to say] 		-0.01 -1.17 – 1.14 0.981
income [> R100,000] *
education [some college] 		0.21 -0.29 – 0.72 0.409
income [prefer not to
say] * education [some
college] 		0.03 -0.87 – 0.94 0.942
ethnicity [black or
african] * education
[2-year degree] 		-0.91 -2.91 – 1.09 0.370
ethnicity [prefer not to
say] * education [2-year
degree] 		-0.30 -2.53 – 1.93 0.792
ethnicity [white or
caucasian] * education
[2-year degree] 		-0.50 -2.30 – 1.30 0.587
ethnicity [black or
african] * education
[4-year degree] 		-0.02 -1.07 – 1.03 0.971
ethnicity [prefer not to
say] * education [4-year
degree] 		-1.47 -3.00 – 0.07 0.061
ethnicity [black or
african] * education
[graduate degree] 		-0.21 -1.28 – 0.85 0.694
ethnicity [coloured] *
education [graduate
degree] 		-0.36 -1.82 – 1.11 0.634
ethnicity [black or
african] * education
[high school] 		-0.26 -1.07 – 0.56 0.533
ethnicity [coloured] *
education [high school] 		-0.20 -1.08 – 0.68 0.649
ethnicity [other] *
education [high school] 		-0.10 -1.98 – 1.78 0.917
ethnicity [prefer not to
say] * education [high
school] 		-0.23 -1.86 – 1.39 0.778
ethnicity [white or
caucasian] * education
[high school] 		-0.43 -1.38 – 0.51 0.370
ethnicity [black or
african] * education
[prefer not to say] 		-0.50 -1.55 – 0.56 0.356
ethnicity [coloured] *
education [prefer not to
say] 		-0.38 -1.57 – 0.80 0.527
ethnicity [prefer not to
say] * education [prefer
not to say] 		-0.13 -2.01 – 1.74 0.889
ethnicity [black or
african] * education
[some college] 		-1.12 -2.76 – 0.52 0.182
ethnicity [coloured] *
education [some college] 		-1.15 -2.85 – 0.55 0.183
ethnicity [prefer not to
say] * education [some
college] 		-1.00 -2.58 – 0.58 0.213
ethnicity [white or
caucasian] * education
[some college] 		-1.20 -2.71 – 0.31 0.119
(income [> R100,000]
ethnicity [black or
african]) education
[2-year degree] 		0.30 -1.14 – 1.74 0.679
(income [> R100,000]
ethnicity [black or
african]) education
[4-year degree] 		-0.28 -1.94 – 1.38 0.744
(income [prefer not to
say] * ethnicity [black
or african]) * education
[4-year degree] 		-0.98 -2.57 – 0.62 0.230
(income [> R100,000]
ethnicity [black or
african]) education
[graduate degree] 		0.25 -1.17 – 1.67 0.728
(income [prefer not to
say] * ethnicity [black
or african]) * education
[graduate degree] 		0.05 -1.51 – 1.61 0.950
(income [> R100,000]
ethnicity [coloured])
education [graduate
degree] 		0.74 -1.22 – 2.70 0.462
(income [> R100,000]
ethnicity [black or
african]) education
[high school] 		0.49 -0.61 – 1.58 0.386
(income [prefer not to
say] * ethnicity [black
or african]) * education
[high school] 		-0.28 -1.72 – 1.17 0.705
(income [prefer not to
say] * ethnicity
[coloured]) * education
[high school] 		-0.41 -1.95 – 1.13 0.605
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[high school] 		-0.60 -2.57 – 1.37 0.550
(income [prefer not to
say] * ethnicity [white
or caucasian]) *
education [high school] 		-0.08 -1.69 – 1.52 0.917
(income [prefer not to
say] * ethnicity [black
or african]) * education
[prefer not to say] 		0.41 -0.82 – 1.64 0.512
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[prefer not to say] 		-0.12 -1.96 – 1.72 0.899
(income [prefer not to
say] * ethnicity [black
or african]) * education
[some college] 		0.10 -0.83 – 1.04 0.828
(income [prefer not to
say] * ethnicity
[coloured]) * education
[some college] 		0.03 -1.10 – 1.15 0.963
Observations 933
R2 / R2 adjusted 0.093 / 0.023

Interaction model 2

  vax status
Predictors Estimates CI p
(Intercept) 0.67 0.10 – 1.23 0.021
religion [hinduism] 0.40 -0.59 – 1.39 0.427
religion [islam] -1.45 -3.16 – 0.27 0.098
religion [no religion] -0.33 -1.13 – 0.47 0.422
religion [other] -0.18 -1.42 – 1.06 0.775
religion [prefer not to
say] 		-0.51 -1.64 – 0.61 0.372
politics [liberal] -0.00 -0.80 – 0.80 1.000
politics [moderate] -0.27 -0.98 – 0.45 0.464
politics [prefer not to
say] 		-0.25 -0.84 – 0.34 0.412
location [rural] -0.07 -0.66 – 0.52 0.824
location [suburban] -0.19 -0.79 – 0.41 0.536
location [urban] -0.07 -0.65 – 0.52 0.823
religion [hinduism] *
politics [liberal] 		0.04 -1.36 – 1.44 0.958
religion [islam] *
politics [liberal] 		0.54 -0.68 – 1.76 0.387
religion [no religion] *
politics [liberal] 		0.20 -0.72 – 1.13 0.664
religion [other] *
politics [liberal] 		-0.13 -1.05 – 0.79 0.783
religion [prefer not to
say] * politics [liberal] 		1.04 -0.36 – 2.44 0.147
religion [islam] *
politics [moderate] 		2.05 0.03 – 4.07 0.047
religion [no religion] *
politics [moderate] 		0.93 -0.41 – 2.27 0.174
religion [other] *
politics [moderate] 		-0.67 -1.49 – 0.15 0.111
religion [prefer not to
say] * politics
[moderate] 		-0.00 -1.15 – 1.15 1.000
religion [hinduism] *
politics [prefer not to
say] 		-0.14 -1.36 – 1.08 0.820
religion [islam] *
politics [prefer not to
say] 		1.03 -0.37 – 2.42 0.148
religion [no religion] *
politics [prefer not to
say] 		0.31 -0.34 – 0.96 0.350
religion [other] *
politics [prefer not to
say] 		-0.24 -0.97 – 0.50 0.523
religion [prefer not to
say] * politics [prefer
not to say] 		0.43 -0.61 – 1.47 0.420
religion [hinduism] *
location [rural] 		-0.14 -1.81 – 1.53 0.871
religion [islam] *
location [rural] 		0.37 -0.84 – 1.59 0.546
religion [no religion] *
location [rural] 		0.73 -0.34 – 1.80 0.183
religion [other] *
location [rural] 		-0.13 -1.24 – 0.98 0.823
religion [prefer not to
say] * location [rural] 		0.91 -0.59 – 2.42 0.233
religion [hinduism] *
location [suburban] 		0.12 -1.10 – 1.35 0.843
religion [islam] *
location [suburban] 		0.97 -1.01 – 2.95 0.337
religion [no religion] *
location [suburban] 		0.35 -0.73 – 1.43 0.522
religion [other] *
location [suburban] 		0.20 -1.23 – 1.64 0.780
religion [prefer not to
say] * location
[suburban] 		-0.20 -0.92 – 0.53 0.591
religion [islam] *
location [urban] 		0.85 -0.55 – 2.24 0.235
religion [no religion] *
location [urban] 		0.06 -0.49 – 0.61 0.825
religion [other] *
location [urban] 		0.25 -0.84 – 1.34 0.657
religion [prefer not to
say] * location [urban] 		-0.09 -0.63 – 0.46 0.754
politics [liberal] *
location [rural] 		0.02 -0.82 – 0.86 0.965
politics [moderate] *
location [rural] 		0.31 -0.43 – 1.05 0.414
politics [prefer not to
say] * location [rural] 		0.19 -0.43 – 0.81 0.540
politics [liberal] *
location [suburban] 		0.05 -0.81 – 0.90 0.913
politics [moderate] *
location [suburban] 		0.34 -0.42 – 1.10 0.382
politics [prefer not to
say] * location
[suburban] 		0.39 -0.25 – 1.03 0.233
politics [liberal] *
location [urban] 		-0.04 -0.87 – 0.79 0.929
politics [moderate] *
location [urban] 		0.27 -0.47 – 1.01 0.479
politics [prefer not to
say] * location [urban] 		0.22 -0.40 – 0.84 0.483
(religion [islam]
politics [liberal])
location [rural] 		-0.08 -1.80 – 1.64 0.924
(religion [no religion]
politics [liberal])
location [rural] 		-1.10 -2.33 – 0.13 0.081
(religion [other]
politics [liberal])
location [rural] 		0.82 -0.58 – 2.22 0.252
(religion [prefer not to
say] * politics
[liberal]) * location
[rural] 		-1.06 -2.92 – 0.81 0.266
(religion [islam]
politics [moderate])
location [rural] 		-0.62 -2.38 – 1.15 0.494
(religion [no religion]
politics [moderate])
location [rural] 		-1.30 -2.93 – 0.32 0.115
(religion [other]
politics [moderate])
location [rural] 		1.33 0.01 – 2.65 0.048
(religion [prefer not to
say] * politics
[moderate]) * location
[rural] 		-0.04 -1.85 – 1.76 0.964
(religion [no religion]
politics [prefer not to
say]) location [rural] 		-0.76 -1.75 – 0.24 0.136
(religion [prefer not to
say] * politics [prefer
not to say]) * location
[rural] 		-0.71 -2.18 – 0.77 0.348
(religion [islam]
politics [liberal])
location [suburban] 		0.41 -1.45 – 2.28 0.663
(religion [no religion]
politics [liberal])
location [suburban] 		-0.75 -2.29 – 0.79 0.339
(religion [other]
politics [liberal])
location [suburban] 		-0.42 -1.96 – 1.12 0.594
(religion [prefer not to
say] * politics
[liberal]) * location
[suburban] 		-0.52 -1.85 – 0.82 0.449
(religion [islam]
politics [moderate])
location [suburban] 		-2.12 -4.48 – 0.24 0.079
(religion [no religion]
politics [moderate])
location [suburban] 		-1.07 -2.64 – 0.50 0.181
(religion [other]
politics [moderate])
location [suburban] 		0.76 -0.48 – 2.00 0.227
(religion [prefer not to
say] * politics
[moderate]) * location
[suburban] 		0.16 -1.16 – 1.49 0.807
(religion [islam]
politics [prefer not to
say]) location
[suburban] 		-1.17 -3.15 – 0.81 0.247
(religion [no religion]
politics [prefer not to
say]) location
[suburban] 		-0.35 -1.38 – 0.68 0.503
(religion [other]
politics [prefer not to
say]) location
[suburban] 		0.20 -0.93 – 1.32 0.730
(religion [islam]
politics [moderate])
location [urban] 		-1.30 -3.13 – 0.53 0.165
(religion [no religion]
politics [moderate])
location [urban] 		-0.86 -2.15 – 0.43 0.189
Observations 933
R2 / R2 adjusted 0.072 / -0.005

Predict not open to treatment(chatfuel data)

We found:

  • ethnicity != [asian or india, black or africa, coloured, other] -> not open to treatment
  • ethnicity [prefer not to say] -> not open to treatment
  • ethnicity [white or caucasian] -> not open to treatment
  • income [prefer not to say] -> not open to treatment
  • education [graduate degree] -> not open to treatment
  • education [prefer not to say] -> not open to treatment
  • education [some college] -> not open to treatment
  • religion [no religion] -> not open to treatment
  • religion [other] -> not open to treatment
  • ethnicity [prefer not to say] * education [4-year degree] -> not open to treatment
  • income [> R100,000] * education [prefer not to say] -> not open to treatment
  • religion [islam] -> not open to treatment
  • religion [prefer not to say] * politics[moderate] -> not open to treatment
  • religion [islam] * politics [moderate] *location [suburban] -> not open to treatment

Tree model

Sum model

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.05 -0.18 – 0.28 0.668
cv age -0.00 -0.00 – 0.00 0.539
ethnicity [black or
african] 		0.16 -0.04 – 0.37 0.124
ethnicity [coloured] 0.18 -0.03 – 0.40 0.094
ethnicity [other] -0.13 -0.62 – 0.37 0.616
ethnicity [prefer not to
say] 		0.38 0.14 – 0.62 0.002
ethnicity [white or
caucasian] 		0.36 0.15 – 0.58 0.001
income [> R100,000] -0.06 -0.15 – 0.02 0.152
income [prefer not to
say] 		0.06 0.01 – 0.11 0.020
education [2-year degree] 0.12 -0.00 – 0.23 0.051
education [4-year degree] 0.02 -0.11 – 0.16 0.756
education [graduate
degree] 		0.16 0.04 – 0.28 0.011
education [high school] 0.04 -0.01 – 0.09 0.128
education [prefer not to
say] 		0.11 0.01 – 0.22 0.040
education [some college] 0.08 0.02 – 0.14 0.014
religion [hinduism] 0.12 -0.14 – 0.38 0.356
religion [islam] 0.01 -0.14 – 0.15 0.934
religion [no religion] 0.09 0.01 – 0.17 0.020
religion [other] 0.14 0.02 – 0.25 0.025
religion [prefer not to
say] 		-0.04 -0.14 – 0.07 0.460
politics [liberal] -0.02 -0.11 – 0.06 0.576
politics [moderate] -0.02 -0.10 – 0.05 0.498
politics [prefer not to
say] 		0.00 -0.06 – 0.07 0.904
location [rural] -0.21 -0.31 – -0.12 <0.001
location [suburban] -0.15 -0.25 – -0.05 0.004
location [urban] -0.15 -0.24 – -0.05 0.003
Observations 928
R2 / R2 adjusted 0.122 / 0.098

Interaction model 1

  not open to treatment
Predictors Estimates CI p
(Intercept) -0.00 -0.28 – 0.28 1.000
income [> R100,000] -0.11 -1.22 – 1.01 0.853
income [prefer not to
say] 		0.00 -0.69 – 0.69 1.000
ethnicity [black or
african] 		0.05 -0.23 – 0.34 0.711
ethnicity [coloured] 0.23 -0.10 – 0.56 0.170
ethnicity [other] 0.00 -0.95 – 0.95 1.000
ethnicity [prefer not to
say] 		0.57 -0.34 – 1.47 0.219
ethnicity [white or
caucasian] 		0.00 -0.34 – 0.34 1.000
education [2-year degree] -0.47 -1.78 – 0.84 0.481
education [4-year degree] 0.00 -0.66 – 0.66 1.000
education [graduate
degree] 		0.00 -0.66 – 0.66 1.000
education [high school] 0.00 -0.52 – 0.52 1.000
education [prefer not to
say] 		-0.00 -0.66 – 0.66 1.000
education [some college] 0.43 -0.63 – 1.50 0.424
income [> R100,000] *
ethnicity [black or
african] 		0.05 -1.03 – 1.13 0.926
income [prefer not to
say] * ethnicity [black
or african] 		0.04 -0.66 – 0.73 0.916
income [> R100,000] *
ethnicity [coloured] 		-0.06 -1.24 – 1.12 0.921
income [prefer not to
say] * ethnicity
[coloured] 		-0.01 -0.75 – 0.73 0.982
income [> R100,000] *
ethnicity [prefer not to
say] 		0.27 -1.17 – 1.70 0.715
income [prefer not to
say] * ethnicity [prefer
not to say] 		-0.23 -1.27 – 0.81 0.659
income [> R100,000] *
ethnicity [white or
caucasian] 		-0.00 -0.95 – 0.95 1.000
income [prefer not to
say] * ethnicity [white
or caucasian] 		0.60 -0.17 – 1.37 0.125
income [> R100,000] *
education [2-year degree] 		0.58 -0.38 – 1.53 0.239
income [prefer not to
say] * education [2-year
degree] 		0.04 -0.32 – 0.40 0.843
income [> R100,000] *
education [4-year degree] 		0.11 -0.95 – 1.16 0.845
income [prefer not to
say] * education [4-year
degree] 		-0.60 -1.55 – 0.35 0.215
income [> R100,000] *
education [graduate
degree] 		0.27 -0.62 – 1.16 0.549
income [prefer not to
say] * education
[graduate degree] 		0.40 -0.55 – 1.35 0.409
income [> R100,000] *
education [high school] 		0.44 -0.29 – 1.17 0.238
income [prefer not to
say] * education [high
school] 		-0.00 -0.93 – 0.93 1.000
income [> R100,000] *
education [prefer not to
say] 		1.11 0.05 – 2.16 0.041
income [prefer not to
say] * education [prefer
not to say] 		-0.03 -0.78 – 0.72 0.941
income [> R100,000] *
education [some college] 		0.02 -0.31 – 0.35 0.895
income [prefer not to
say] * education [some
college] 		-0.43 -1.02 – 0.16 0.149
ethnicity [black or
african] * education
[2-year degree] 		0.59 -0.71 – 1.89 0.371
ethnicity [prefer not to
say] * education [2-year
degree] 		0.10 -1.35 – 1.55 0.892
ethnicity [white or
caucasian] * education
[2-year degree] 		0.50 -0.67 – 1.67 0.402
ethnicity [black or
african] * education
[4-year degree] 		0.03 -0.65 – 0.71 0.933
ethnicity [prefer not to
say] * education [4-year
degree] 		1.27 0.27 – 2.26 0.013
ethnicity [black or
african] * education
[graduate degree] 		0.06 -0.63 – 0.75 0.871
ethnicity [coloured] *
education [graduate
degree] 		0.38 -0.57 – 1.33 0.436
ethnicity [black or
african] * education
[high school] 		0.02 -0.51 – 0.55 0.937
ethnicity [coloured] *
education [high school] 		-0.18 -0.75 – 0.40 0.547
ethnicity [other] *
education [high school] 		-0.00 -1.22 – 1.22 1.000
ethnicity [prefer not to
say] * education [high
school] 		0.18 -0.87 – 1.24 0.733
ethnicity [white or
caucasian] * education
[high school] 		0.33 -0.28 – 0.95 0.288
ethnicity [black or
african] * education
[prefer not to say] 		0.13 -0.56 – 0.81 0.714
ethnicity [coloured] *
education [prefer not to
say] 		-0.23 -1.00 – 0.54 0.556
ethnicity [prefer not to
say] * education [prefer
not to say] 		-0.57 -1.78 – 0.65 0.361
ethnicity [black or
african] * education
[some college] 		-0.36 -1.43 – 0.70 0.503
ethnicity [coloured] *
education [some college] 		-0.52 -1.62 – 0.58 0.354
ethnicity [prefer not to
say] * education [some
college] 		-0.33 -1.36 – 0.69 0.522
ethnicity [white or
caucasian] * education
[some college] 		-0.10 -1.08 – 0.88 0.841
(income [> R100,000]
ethnicity [black or
african]) education
[2-year degree] 		-0.70 -1.63 – 0.24 0.144
(income [> R100,000]
ethnicity [black or
african]) education
[4-year degree] 		-0.13 -1.21 – 0.94 0.806
(income [prefer not to
say] * ethnicity [black
or african]) * education
[4-year degree] 		0.48 -0.56 – 1.52 0.364
(income [> R100,000]
ethnicity [black or
african]) education
[graduate degree] 		-0.33 -1.25 – 0.59 0.483
(income [prefer not to
say] * ethnicity [black
or african]) * education
[graduate degree] 		0.12 -0.89 – 1.13 0.818
(income [> R100,000]
ethnicity [coloured])
education [graduate
degree] 		-0.47 -1.74 – 0.81 0.473
(income [> R100,000]
ethnicity [black or
african]) education
[high school] 		-0.46 -1.17 – 0.25 0.206
(income [prefer not to
say] * ethnicity [black
or african]) * education
[high school] 		0.03 -0.91 – 0.97 0.952
(income [prefer not to
say] * ethnicity
[coloured]) * education
[high school] 		0.06 -0.94 – 1.06 0.900
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[high school] 		-0.35 -1.63 – 0.93 0.591
(income [prefer not to
say] * ethnicity [white
or caucasian]) *
education [high school] 		-0.48 -1.52 – 0.56 0.366
(income [prefer not to
say] * ethnicity [black
or african]) * education
[prefer not to say] 		0.04 -0.75 – 0.84 0.912
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[prefer not to say] 		0.76 -0.43 – 1.96 0.211
(income [prefer not to
say] * ethnicity [black
or african]) * education
[some college] 		0.43 -0.17 – 1.04 0.162
(income [prefer not to
say] * ethnicity
[coloured]) * education
[some college] 		0.44 -0.29 – 1.17 0.235
Observations 933
R2 / R2 adjusted 0.176 / 0.112

Interaction model 2

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.33 -0.04 – 0.71 0.081
religion [hinduism] -0.16 -0.81 – 0.50 0.642
religion [islam] 1.59 0.45 – 2.72 0.006
religion [no religion] 0.17 -0.37 – 0.70 0.543
religion [other] 0.52 -0.30 – 1.34 0.215
religion [prefer not to
say] 		-0.45 -1.20 – 0.30 0.241
politics [liberal] -0.33 -0.86 – 0.20 0.217
politics [moderate] -0.13 -0.61 – 0.34 0.581
politics [prefer not to
say] 		0.05 -0.34 – 0.45 0.788
location [rural] -0.30 -0.70 – 0.09 0.126
location [suburban] -0.19 -0.59 – 0.21 0.351
location [urban] -0.18 -0.56 – 0.21 0.367
religion [hinduism] *
politics [liberal] 		0.03 -0.90 – 0.96 0.949
religion [islam] *
politics [liberal] 		-0.97 -1.78 – -0.16 0.019
religion [no religion] *
politics [liberal] 		0.20 -0.41 – 0.81 0.527
religion [other] *
politics [liberal] 		0.20 -0.41 – 0.81 0.527
religion [prefer not to
say] * politics [liberal] 		0.03 -0.90 – 0.96 0.949
religion [islam] *
politics [moderate] 		-1.79 -3.13 – -0.45 0.009
religion [no religion] *
politics [moderate] 		-0.37 -1.25 – 0.52 0.420
religion [other] *
politics [moderate] 		-0.24 -0.78 – 0.31 0.389
religion [prefer not to
say] * politics
[moderate] 		0.76 0.00 – 1.52 0.050
religion [hinduism] *
politics [prefer not to
say] 		0.01 -0.80 – 0.82 0.980
religion [islam] *
politics [prefer not to
say] 		-0.97 -1.90 – -0.05 0.039
religion [no religion] *
politics [prefer not to
say] 		-0.15 -0.58 – 0.28 0.489
religion [other] *
politics [prefer not to
say] 		0.09 -0.39 – 0.58 0.705
religion [prefer not to
say] * politics [prefer
not to say] 		0.23 -0.46 – 0.92 0.518
religion [hinduism] *
location [rural] 		0.41 -0.70 – 1.52 0.473
religion [islam] *
location [rural] 		-0.69 -1.49 – 0.12 0.095
religion [no religion] *
location [rural] 		-0.19 -0.91 – 0.52 0.593
religion [other] *
location [rural] 		-0.44 -1.17 – 0.30 0.247
religion [prefer not to
say] * location [rural] 		0.42 -0.58 – 1.42 0.409
religion [hinduism] *
location [suburban] 		0.01 -0.80 – 0.83 0.976
religion [islam] *
location [suburban] 		-1.73 -3.04 – -0.41 0.010
religion [no religion] *
location [suburban] 		0.19 -0.53 – 0.91 0.599
religion [other] *
location [suburban] 		-0.16 -1.11 – 0.79 0.738
religion [prefer not to
say] * location
[suburban] 		0.42 -0.06 – 0.90 0.085
religion [islam] *
location [urban] 		-0.74 -1.67 – 0.19 0.117
religion [no religion] *
location [urban] 		0.01 -0.35 – 0.38 0.946
religion [other] *
location [urban] 		-0.34 -1.06 – 0.38 0.355
religion [prefer not to
say] * location [urban] 		0.29 -0.07 – 0.65 0.111
politics [liberal] *
location [rural] 		0.30 -0.25 – 0.86 0.285
politics [moderate] *
location [rural] 		0.15 -0.34 – 0.65 0.539
politics [prefer not to
say] * location [rural] 		-0.01 -0.42 – 0.40 0.962
politics [liberal] *
location [suburban] 		0.33 -0.23 – 0.90 0.248
politics [moderate] *
location [suburban] 		0.18 -0.32 – 0.68 0.484
politics [prefer not to
say] * location
[suburban] 		-0.06 -0.49 – 0.36 0.766
politics [liberal] *
location [urban] 		0.30 -0.25 – 0.85 0.281
politics [moderate] *
location [urban] 		0.04 -0.45 – 0.53 0.875
politics [prefer not to
say] * location [urban] 		-0.08 -0.49 – 0.33 0.697
(religion [islam]
politics [liberal])
location [rural] 		0.07 -1.07 – 1.21 0.905
(religion [no religion]
politics [liberal])
location [rural] 		0.08 -0.74 – 0.90 0.845
(religion [other]
politics [liberal])
location [rural] 		-0.28 -1.21 – 0.65 0.553
(religion [prefer not to
say] * politics
[liberal]) * location
[rural] 		-0.00 -1.24 – 1.23 0.997
(religion [islam]
politics [moderate])
location [rural] 		0.84 -0.33 – 2.01 0.161
(religion [no religion]
politics [moderate])
location [rural] 		0.34 -0.73 – 1.42 0.530
(religion [other]
politics [moderate])
location [rural] 		0.11 -0.77 – 0.98 0.812
(religion [prefer not to
say] * politics
[moderate]) * location
[rural] 		-0.78 -1.98 – 0.42 0.201
(religion [no religion]
politics [prefer not to
say]) location [rural] 		0.33 -0.33 – 0.99 0.325
(religion [prefer not to
say] * politics [prefer
not to say]) * location
[rural] 		-0.27 -1.25 – 0.71 0.587
(religion [islam]
politics [liberal])
location [suburban] 		0.97 -0.27 – 2.21 0.125
(religion [no religion]
politics [liberal])
location [suburban] 		-0.70 -1.72 – 0.33 0.181
(religion [other]
politics [liberal])
location [suburban] 		-0.70 -1.72 – 0.33 0.181
(religion [prefer not to
say] * politics
[liberal]) * location
[suburban] 		-0.15 -1.03 – 0.74 0.745
(religion [islam]
politics [moderate])
location [suburban] 		2.24 0.67 – 3.81 0.005
(religion [no religion]
politics [moderate])
location [suburban] 		0.25 -0.80 – 1.29 0.641
(religion [other]
politics [moderate])
location [suburban] 		-0.31 -1.13 – 0.52 0.464
(religion [prefer not to
say] * politics
[moderate]) * location
[suburban] 		-0.92 -1.80 – -0.05 0.039
(religion [islam]
politics [prefer not to
say]) location
[suburban] 		0.98 -0.33 – 2.30 0.142
(religion [no religion]
politics [prefer not to
say]) location
[suburban] 		-0.14 -0.82 – 0.54 0.693
(religion [other]
politics [prefer not to
say]) location
[suburban] 		-0.18 -0.93 – 0.56 0.630
(religion [islam]
politics [moderate])
location [urban] 		0.88 -0.34 – 2.09 0.156
(religion [no religion]
politics [moderate])
location [urban] 		0.33 -0.53 – 1.18 0.455
Observations 933
R2 / R2 adjusted 0.118 / 0.046

Demographics and Ads

In this section, we are interested in the changes in the distribution of three demographic variables across pilots

Takeaway:

  • The proportion of females in Ads impressions is keep increasing, from 53% to 62%
  • The proportion of people with age > 45 in the conversation started is keep decreasing, from 40% to 20%
  • Most of the impressions/conversations started are made by young people (age: 18-34)

Age

Age and Impression count

Age and Conversation Started count

Gender

Gender and Impression count

Gender and Conversation Started count

Region

Region and Impression count

Region and Conversation Started count

Side-by-Side Chart on Key Metrics

Unnecessary vs Unsafe vs Inaccessible

This table compared three Ad impediment sources (vaccine is unnecessary vs vaccine is risky vs vaccine is inaccessible) in terms of the metrics described above.

Winner: unnecessary

Control vs Airtime

This table compared three Ad body text approaches - control (share your opinion) vs airtime (take a short survey and earn airtime) vs survey (take this short survey)- in terms of the metrics described above.

Winner: airtime

Compare by Image

This table compared nine images (provided below the table) in terms of the metrics described above.

Winner: image 1

Takeaway:

Best performing images that we should we use as baseline images for the next large survey/experiment:

  • unnecessary: image 1
  • risky: image 6, but the key cost metric of image 4, image 5, image 6 are close to each other
  • inaccessible: image 7, note: this image is supressed by FB algorithm due to higher cost per impression and higher cost per link click
Images used
Image 1

Image 2

Image 3

Image 4

Image 5

Image 6

Image 7

Image 8

Image 9