Current Pilot

This analysis is based on 1684 respondents who completed the current pilot survey wave (version 5), and 398 respondents who completed the last pilot survey wave (version 4b).

Key Takeaways:

  • FB’s A|B testing is NOT a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets)
  • FB’s A|B testing can pick the best Ad set, which is the best-performed Ad set based on our key cost metric when we look at the bottom of the funnel
  • When we adjust for the quality of responses (e.g., elicitation, time spent in survey, nonsense answers), airtime ads still outperform our control when using our key cost metric
  • Airtime participants will not skew responses
  • The proportion of females in Ads impressions is keep increasing, from 53% to 62%
  • The proportion of people with age > 45 in the conversation started is keep decreasing, from 40% to 20%
  • Most of the impressions/conversations started are made by young people (age: 18-34)
  • In Side-by-Side comparisons, unnecessary outperformed risk and inaccessible; airtime outperformed control; image 1 outperformed all other images

Setup

Detailed setting can be found here.

  • 1 campaign
  • 5 ad sets, 3 ads in each ad set
  • 15 ads split into:
    • 3 impediment themes (3 inaccessible, 6 risky, 6 unnecessary)
    • 2 different prompts (6 control and 9 airtime)
    • 9 different images (Images 1-6 used twice, images 7-9 used once)

Overview of the Ads Strategy

Ad Performance will be measured using the following indicators:

  • Click-through rate
  • Recruitment of non-vaccinated but potentially treatable participants
    • Total quantity
    • Percent of total participants recruited
    • Retention
  • Average participant elicitation
  • Cost
    • per impression
    • per Link Click
    • per Survey Complete

Key definitions:

  • unvax: participants with “vax_status” == “unvax”
  • unvax, open to treatment: participants that are unvax and chose “maybe” or “of course” when we ask them “would you ever consider getting a vaccine in the future?” in the Treatment section

Goals for this analysis script (version 5):

  • See whether Facebook A|B testing is a “true” randomized A|B test, and look into whether the best Ad set selected by Facebook aligns with the Ad set selected by our key metrics
  • Provide an overview on the performance of current Ads based on various metrics and compare the Ads performance between version 4b and version 5 in terms of the key cost metrics
  • Evaluate the strategy of using airtime text to recruit by estimating the advantages and disadvantages of mentioning airtime in our Ads
  • Analysis on Ad targeting: find demographic covariates that would predict unvax, open to treatment participants
  • Understand the variation in demographics distribution across pilots
  • Compare the Ads performance across three different themes, built upon hypothesized drivers of hesitancy, as well as different versions of creativity within those themes.

Goals of this pilot with ads:

  • Recruit full survey completes for unvaxxed, open to treatment participants at a low cost while maintaining participant response quality
  • Take the learnings here and apply them to our next large pilot

Part I: Version 5 Specific Analysis

In this part, we are going to answer the research questions that are specific to this pilot (version 5).

1. FB A|B testing algorithm analysis

Takeaways:

  • FB’s A|B testing can pick the best Ad set, which is the best-performed Ad set based on our key cost metric when we look at the bottom of the funnel
  • FB’s A|B testing is NOT a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets)

Winning Ad set comparison

To understand Facebook’s A|B test capabilities to draw conclusions of which ad performed best, we are going to see whether the winning Ad set selected by Facebook matches the winning Ad set selected by our bottom of the funnel key metrics.

  • Facebook A|B test selected pilot_v5_unnecessary_airtime
  • The metric table below suggests us pilot_v5_unnecessary_airtime is the best based on our funnel analysis

Conclusion:

  • FB’s winning Ad set matches the winning Ad set selected by our metrics. We can make a preliminary determination that Facebook’s A|B test can draw conclusions of which ad performed best

  • Interpretation: FB A|B test divides our budget to equally and randomly divide exposure between each version of our Ad sets and chooses the most cost-efficient Ad set as the winner, which aligns with our cost metrics. i.e. Given a budget, the more Ad audiences started the conversion, the more participants (that are unvaccinated and open to treatment) we can obtain, hence more cost-efficient when we look at the key cost metrics at the bottom of the funnel.

Demographic variables distribution

We are interested in:

  • is FB A|B testing a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets), OR
  • is it a different comparative test (FB divides up the total population into five equal samples, then runs an algorithm to identify attractive target populations to maximize cost efficiency on each sample)?

Metric:

  • if it is a true A|B test, we should see roughly the same demographic distributions in each of the five ad sets.
  • if it isn’t, we should see very different demographic distributions in each of the five ad sets.

Findings:

  • Using the Facebook ads data on demographics, if we combine the 18-24 and 25 - 34 population group, the age distribution is roughly balanced across five ad sets
  • Similarly, region (as identified by FB) is also roughly balanced
  • However, gender is not balanced across five ad sets, with two ad sets having about 10% lower female ratio (~57.5%) than the other three ads sets (~67.5%), which might suggest we are hitting different populations.

Age

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Gender

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Region

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Age | Gender

Since we found imbalanced distribution of gender, we are interested in will the distribution of Age is conditional on gender. Based on the plots below, the answer is No.

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

2. Airtime analysis

Our last pilot ads that mention airtime were a lot more cost-efficient in recruiting a large number of participants.

In this subsection, we are interested in:

  • When we adjust for the quality of responses (e.g., elicitation, time spent in survey, nonsense answers), do airtime ads still outperform our control when using our key cost metric (survey complete, unvaxxed, open to treatment)?

  • How does recruit airtime participants also skew responses

  • Should we continue to use airtime ads to recruit participants?

Takeaways:

  • After adjusting for the number of characters elicited or time spent in the survey, ads mentioning airtime are still a lot more cost-efficient.
  • When looking at the best treatment respondents select, the treatment selection distribution is roughly the same as the control group, suggesting financial incentives in ads do not translate to people selecting “rewards for vaxxing” in treatment.
  • We should continue to use airtime ads to recruit participants.

Adjusted cost analysis

  • We compared the cost per ten characters elicitation in treatment and impediment explanations between the airtime and control group.
  • We also compared the cost per minute spent in Survey Complete between the airtime and control groups.

In both comparisons, we found airtime is still a lot more cost-efficient.

Adjusted by character elicitation

Adjusted by time spent in survey

Skew responses analysis

We estimated the distribution of best treatment selected by participants recruited by airtime and participants recruited by control. We would like to see is there a skewed distribution of airtime’s. Based on the table below, the answer is NO.

Treatments

Part II: General Ads A|B Testing

This part includes the analysis that we will apply to every pilot.

1. Demographics and Ads

In this section, we are interested in the changes in the distribution of three demographic variables across pilots

Takeaway:

  • The proportion of females in Ads impressions is keep increasing, from 53% to 62%
  • The proportion of people with age > 45 in the conversation started is keep decreasing, from 40% to 20%
  • Most of the impressions/conversations started are made by young people (age: 18-34)

Age

Age and Impression count

Age and Conversation Started count

Gender

Gender and Impression count

Gender and Conversation Started count

Region

Region and Impression count

Region and Conversation Started count

2. Side-by-Side Chart on Key Metrics

Metrics explanation:

  • Impressions (Total Count) = the total number of times our ad has been viewed
  • Clickthrough (%) = #clicks / #impressions
  • Messages Sent (%) = #conversations / #clicks
  • Consent Obtained (%) = #consents / #conversations
  • Core Survey Complete (%) = #forking section completed / #consents
  • Treatment Complete (%) = #treatment section completed / #forking section completed
  • Demo Questions Complete (%) = #demog section completed / #treatment section completed
  • Full Survey Complete (%) = #full chat completed / #demog section completed
  • Total characters elicited per completed survey (treatment) = average #character in best treatment explanation per full chat completed
  • Avg characters elicited per completed survey (impediment explanations) = average #character in impediment explanations per full chat completed
  • Cost per Impression = amount spent / #impressions (in USD)
  • Cost per Link Click = amount spent / #clicks (in USD)
  • Cost per Survey Complete (All participants) = amount spent / #full chat completed (in USD)
  • Cost per Survey Complete (Unvax) = amount spent / #full chat completed with unvaccinated participants (in USD)
  • Cost per Survey Complete (Unvax, Open to Treatment) = amount spent / #full chat completed with unvaccinated and open to treatment participants (in USD)

Unnecessary vs Unsafe vs Inaccessible

This table compared three Ad impediment sources (vaccine is unnecessary vs vaccine is risky vs vaccine is inaccessible) in terms of the metrics described above.

Winner: unnecessary

Body texts used
Unnecessary
  • Do you actually NEED the COVID VACCINE? Take a short survey, earn mobile airtime! (Airtime)
  • Do you actually NEED the COVID VACCINE? Tell us about your experience, we’re here to listen (Control)
Risky
  • Is the COVID VACCINE too RISKY? Take a short survey, earn mobile airtime! (Airtime)
  • Is the COVID VACCINE too RISKY? Tell us about your experience, we’re here to listen (Control)
Inaccessible
  • IMPOSSIBLE to get a COVID VACCINE? Take a short survey, earn mobile airtime! (Airtime)

Control vs Airtime

This table compared three Ad body text approaches - control (share your opinion) vs airtime (take a short survey and earn airtime) vs survey (take this short survey)- in terms of the metrics described above.

Winner: airtime

Compare by Image

This table compared nine images (provided below the table) in terms of the metrics described above.

Winner: image 1

Takeaway:

Best performing images that we should we use as baseline images for the next large survey/experiment:

  • unnecessary: image 1
  • risky: image 6, but the key cost metric of image 4, image 5, image 6 are close to each other
  • inaccessible: image 7, note: this image is supressed by FB algorithm due to higher cost per impression and higher cost per link click
Images used
Image 1

Image 2

Image 3

Image 4

Image 5

Image 6

Image 7

Image 8

Image 9

3. Analysis on the best performing ad in each category

In this section, we are interested in the demographics, and funnel performance of the best performing ad in each category (unnecessary, risky, and inaccessible) in version 5.

  • Best Ad of unnecessary: Ad name: pilot_v5_ad1_image1
  • Best Ad of risky: Ad name: pilot_v5_ad3_image6
  • Best Ad of inaccessible: Ad name: pilot_v5_ad5_image7

Takeaways:

  • The best Ad of unnecessary has 10% more proportion of “impression/conversation started” for the age in 18-24 than the average proportion in pilot v5
  • The best Ad of inaccessible has 15% more proportion of “conversation started” for the age in 18-24 than the average proportion in pilot v5
  • The best Ad of inaccessible and the best Ad of risky has 10% more proportion of females than the average proportion in pilot v5
  • The cost per open to treatment survey of the best Ad of unnecessary is 50% less than the average cost of the whole unnecessary theme
  • The cost per open to treatment survey of the best Ad of risky is 37% less than the average cost of the whole risky theme
  • The cost per open to treatment survey of the best Ad of inaccessible is 30% less than the average cost of the whole inaccessible theme

Demographics

Age

Age and Impression count

Age and Conversation Started count

Gender

Gender and Impression count

Gender and Conversation Started count

Region

Region and Impression count

Region and Conversation Started count

Key metrics

4. Ad Targeting analysis

In this sub-section, we are interested in:

  • What demographic configurations predict vaxxed participants (with a low chance of false positives)?
  • What demographic configurations predict participants who are not open to treatment (with a low chance of false positives)?

Takeaway:

  • Using data provided by Ads Manager, demographic configurations in Ads Manager include age, gender, and region. Based on the regression result, we found there are NO significant demographic variables (all with high pvalue).
  • Using data provided by Chatfuel, found some significant demographic covariates and listed them in each tab.
  • We need more data and should consider more techniques to explore the targeting dimension

Hypotheses:

  • ethnicity [white or caucasian] -> not open to treatment

Predict vaxxed participants(from Ad configurations)

There is no significant covariate based on regression output.

Age

  vax status
Predictors Estimates CI p
(Intercept) 0.59 0.56 – 0.61 <0.001
Age25-34 0.00 -0.03 – 0.03 1.000
Age35-44 0.00 -0.03 – 0.03 1.000
Age45-54 0.00 -0.03 – 0.03 1.000
Age55-64 0.00 -0.03 – 0.03 1.000
Age [65+] 0.00 -0.03 – 0.03 1.000
Observations 10608
R2 / R2 adjusted 0.000 / -0.000

Gender

  vax status
Predictors Estimates CI p
(Intercept) 0.59 0.56 – 0.61 <0.001
Gender [male] -0.00 -0.03 – 0.03 1.000
Observations 3536
R2 / R2 adjusted 0.000 / -0.000

Region

  vax status
Predictors Estimates CI p
(Intercept) 0.59 0.56 – 0.61 <0.001
Region [Free State] -0.00 -0.03 – 0.03 1.000
Region [Gauteng] -0.00 -0.03 – 0.03 1.000
Region [KwaZulu-Natal] -0.00 -0.03 – 0.03 1.000
Region [Limpopo] -0.00 -0.03 – 0.03 1.000
Region [Mpumalanga] -0.00 -0.03 – 0.03 1.000
Region [North West] -0.00 -0.03 – 0.03 1.000
Region [Northern Cape] -0.00 -0.03 – 0.03 1.000
Region [Unknown] 0.00 -0.04 – 0.05 0.855
Region [Western Cape] -0.00 -0.03 – 0.03 1.000
Observations 16645
R2 / R2 adjusted 0.000 / -0.001

Age * Gender

  vax status
Predictors Estimates CI p
(Intercept) 0.59 0.56 – 0.61 <0.001
Age25-34 0.00 -0.03 – 0.03 1.000
Age35-44 0.00 -0.03 – 0.03 1.000
Age45-54 0.00 -0.03 – 0.03 1.000
Age55-64 0.00 -0.03 – 0.03 1.000
Age [65+] 0.00 -0.03 – 0.03 1.000
Gender [male] 0.00 -0.03 – 0.03 1.000
Gender [unknown] 0.00 -0.03 – 0.03 1.000
Age25-34:Gendermale -0.00 -0.05 – 0.05 1.000
Age35-44:Gendermale -0.00 -0.05 – 0.05 1.000
Age45-54:Gendermale -0.00 -0.05 – 0.05 1.000
Age55-64:Gendermale -0.00 -0.05 – 0.05 1.000
Age [65+] * Gender [male] -0.00 -0.05 – 0.05 1.000
Age25-34:Genderunknown -0.00 -0.05 – 0.05 1.000
Age35-44:Genderunknown -0.00 -0.05 – 0.05 1.000
Age45-54:Genderunknown 0.01 -0.04 – 0.05 0.791
Age55-64:Genderunknown 0.02 -0.04 – 0.07 0.550
Age [65+] * Gender
[unknown] 		-0.00 -0.05 – 0.05 0.969
Observations 30598
R2 / R2 adjusted 0.000 / -0.001

Predict not open to treatment(from Ad configurations)

There is no significant covariate based on regression output.

Age

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Age25-34 -0.00 -0.02 – 0.02 1.000
Age35-44 -0.00 -0.02 – 0.02 1.000
Age45-54 0.00 -0.02 – 0.02 1.000
Age55-64 -0.00 -0.02 – 0.02 1.000
Age [65+] -0.00 -0.02 – 0.02 1.000
Observations 10608
R2 / R2 adjusted 0.000 / -0.000

Gender

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Gender [male] 0.00 -0.02 – 0.02 1.000
Gender [unknown] 0.00 -0.02 – 0.02 1.000
Observations 5304
R2 / R2 adjusted 0.000 / -0.000

Region

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Region [Free State] 0.00 -0.02 – 0.02 1.000
Region [Gauteng] -0.00 -0.02 – 0.02 1.000
Region [KwaZulu-Natal] -0.00 -0.02 – 0.02 1.000
Region [Limpopo] -0.00 -0.02 – 0.02 1.000
Region [Mpumalanga] -0.00 -0.02 – 0.02 1.000
Region [North West] -0.00 -0.02 – 0.02 1.000
Region [Northern Cape] 0.00 -0.02 – 0.02 1.000
Region [Unknown] 0.01 -0.02 – 0.04 0.497
Region [Western Cape] -0.00 -0.02 – 0.02 1.000
Observations 16645
R2 / R2 adjusted 0.000 / -0.001

Age * Gender

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.11 0.10 – 0.13 <0.001
Age25-34 0.00 -0.02 – 0.02 1.000
Age35-44 0.00 -0.02 – 0.02 1.000
Age45-54 0.00 -0.02 – 0.02 1.000
Age55-64 0.00 -0.02 – 0.02 1.000
Age [65+] 0.00 -0.02 – 0.02 1.000
Gender [male] 0.00 -0.02 – 0.02 1.000
Gender [unknown] 0.00 -0.02 – 0.02 1.000
Age25-34:Gendermale -0.00 -0.03 – 0.03 1.000
Age35-44:Gendermale -0.00 -0.03 – 0.03 1.000
Age45-54:Gendermale -0.00 -0.03 – 0.03 1.000
Age55-64:Gendermale -0.00 -0.03 – 0.03 1.000
Age [65+] * Gender [male] -0.00 -0.03 – 0.03 1.000
Age25-34:Genderunknown -0.00 -0.03 – 0.03 1.000
Age35-44:Genderunknown -0.00 -0.03 – 0.03 1.000
Age45-54:Genderunknown -0.00 -0.03 – 0.03 0.795
Age55-64:Genderunknown -0.00 -0.04 – 0.03 0.886
Age [65+] * Gender
[unknown] 		0.00 -0.03 – 0.03 0.924
Observations 30598
R2 / R2 adjusted 0.000 / -0.001

Predict vaxxed participants(chatfuel data)

We found:

  • income != [prefer not to say] * ethnicity != [prefer not to say] -> vaxxed
  • religion [other] * politics [moderate] * location [rural] -> vaxxed

Tree model

Sum model

  vax status
Predictors Estimates CI p
(Intercept) 0.65 0.33 – 0.97 <0.001
cv age -0.00 -0.00 – 0.00 0.570
ethnicity [black or
african] 		-0.10 -0.39 – 0.19 0.515
ethnicity [coloured] -0.16 -0.46 – 0.14 0.304
ethnicity [other] 0.53 -0.10 – 1.16 0.102
ethnicity [prefer not to
say] 		-0.23 -0.56 – 0.10 0.178
ethnicity [white or
caucasian] 		-0.13 -0.43 – 0.18 0.409
income [> R100,000] -0.02 -0.15 – 0.10 0.721
income [prefer not to
say] 		-0.12 -0.19 – -0.05 0.001
education [2-year degree] -0.16 -0.33 – 0.01 0.064
education [4-year degree] -0.02 -0.21 – 0.18 0.858
education [graduate
degree] 		-0.11 -0.27 – 0.05 0.194
education [high school] -0.08 -0.16 – -0.01 0.027
education [prefer not to
say] 		-0.16 -0.30 – -0.01 0.040
education [some college] -0.12 -0.21 – -0.03 0.008
religion [hinduism] 0.27 -0.07 – 0.61 0.125
religion [islam] 0.04 -0.17 – 0.26 0.689
religion [no religion] -0.08 -0.19 – 0.03 0.150
religion [other] -0.17 -0.33 – -0.00 0.049
religion [prefer not to
say] 		-0.02 -0.16 – 0.12 0.830
politics [liberal] -0.03 -0.15 – 0.09 0.659
politics [moderate] 0.01 -0.09 – 0.10 0.910
politics [prefer not to
say] 		-0.01 -0.10 – 0.09 0.911
location [rural] 0.17 0.03 – 0.31 0.014
location [suburban] 0.16 0.02 – 0.30 0.023
location [urban] 0.16 0.02 – 0.29 0.022
Observations 1083
R2 / R2 adjusted 0.053 / 0.031

Interaction model 1

  vax status
Predictors Estimates CI p
(Intercept) 0.80 0.37 – 1.23 <0.001
income [> R100,000] -0.03 -1.73 – 1.67 0.973
income [prefer not to
say] 		-0.80 -1.85 – 0.25 0.134
ethnicity [black or
african] 		-0.11 -0.54 – 0.33 0.623
ethnicity [coloured] -0.18 -0.69 – 0.32 0.472
ethnicity [other] 0.10 -1.35 – 1.55 0.892
ethnicity [prefer not to
say] 		-0.17 -2.06 – 1.73 0.863
ethnicity [white or
caucasian] 		0.10 -0.42 – 0.62 0.708
education [2-year degree] 0.69 -1.26 – 2.65 0.486
education [4-year degree] 0.10 -0.90 – 1.10 0.845
education [graduate
degree] 		0.10 -0.90 – 1.10 0.845
education [high school] 0.20 -0.60 – 1.00 0.624
education [prefer not to
say] 		0.10 -0.90 – 1.10 0.845
education [some college] 0.20 -0.85 – 1.25 0.708
income [> R100,000] *
ethnicity [black or
african] 		0.00 -1.65 – 1.66 0.996
income [prefer not to
say] * ethnicity [black
or african] 		0.67 -0.39 – 1.72 0.215
income [> R100,000] *
ethnicity [coloured] 		-0.18 -1.97 – 1.61 0.841
income [prefer not to
say] * ethnicity
[coloured] 		0.73 -0.39 – 1.85 0.200
income [prefer not to
say] * ethnicity [other] 		0.85 -1.21 – 2.91 0.417
income [> R100,000] *
ethnicity [prefer not to
say] 		-1.25 -4.01 – 1.51 0.376
income [prefer not to
say] * ethnicity [prefer
not to say] 		0.67 -0.89 – 2.23 0.402
income [> R100,000] *
ethnicity [white or
caucasian] 		-0.10 -1.55 – 1.35 0.892
income [prefer not to
say] * ethnicity [white
or caucasian] 		0.10 -1.07 – 1.27 0.867
income [> R100,000] *
education [2-year degree] 		-0.47 -1.87 – 0.94 0.515
income [prefer not to
say] * education [2-year
degree] 		-0.23 -0.77 – 0.32 0.411
income [> R100,000] *
education [4-year degree] 		0.13 -1.49 – 1.74 0.876
income [prefer not to
say] * education [4-year
degree] 		0.70 -0.75 – 2.15 0.344
income [> R100,000] *
education [graduate
degree] 		-0.16 -1.51 – 1.19 0.820
income [prefer not to
say] * education
[graduate degree] 		-0.30 -1.75 – 1.15 0.685
income [> R100,000] *
education [high school] 		-0.59 -1.69 – 0.52 0.298
income [prefer not to
say] * education [high
school] 		0.30 -1.12 – 1.72 0.678
income [> R100,000] *
education [prefer not to
say] 		-0.87 -2.49 – 0.74 0.290
income [prefer not to
say] * education [prefer
not to say] 		-0.05 -1.19 – 1.09 0.932
income [> R100,000] *
education [some college] 		0.21 -0.29 – 0.71 0.403
income [prefer not to
say] * education [some
college] 		0.80 -0.91 – 2.51 0.359
ethnicity [black or
african] * education
[2-year degree] 		-0.78 -2.72 – 1.17 0.434
ethnicity [coloured] *
education [2-year degree] 		-1.31 -3.50 – 0.88 0.242
ethnicity [prefer not to
say] * education [2-year
degree] 		0.03 -2.13 – 2.20 0.976
ethnicity [white or
caucasian] * education
[2-year degree] 		-0.33 -2.08 – 1.41 0.708
ethnicity [black or
african] * education
[4-year degree] 		-0.08 -1.11 – 0.96 0.884
ethnicity [prefer not to
say] * education [4-year
degree] 		-1.30 -2.80 – 0.20 0.089
ethnicity [black or
african] * education
[graduate degree] 		-0.18 -1.22 – 0.86 0.740
ethnicity [coloured] *
education [graduate
degree] 		-0.35 -1.79 – 1.10 0.640
ethnicity [prefer not to
say] * education
[graduate degree] 		0.70 -0.80 – 2.20 0.359
ethnicity [black or
african] * education
[high school] 		-0.26 -1.07 – 0.54 0.524
ethnicity [coloured] *
education [high school] 		-0.22 -1.08 – 0.65 0.627
ethnicity [other] *
education [high school] 		-0.10 -1.96 – 1.76 0.916
ethnicity [prefer not to
say] * education [high
school] 		-0.50 -2.55 – 1.55 0.632
ethnicity [white or
caucasian] * education
[high school] 		-0.39 -1.31 – 0.54 0.415
ethnicity [black or
african] * education
[prefer not to say] 		-0.56 -1.60 – 0.48 0.290
ethnicity [coloured] *
education [prefer not to
say] 		-0.38 -1.56 – 0.79 0.524
ethnicity [prefer not to
say] * education [prefer
not to say] 		-0.73 -3.12 – 1.65 0.546
ethnicity [black or
african] * education
[some college] 		-0.35 -1.41 – 0.70 0.513
ethnicity [coloured] *
education [some college] 		-0.32 -1.45 – 0.82 0.585
ethnicity [prefer not to
say] * education [some
college] 		-0.83 -2.37 – 0.70 0.287
ethnicity [white or
caucasian] * education
[some college] 		-0.43 -1.66 – 0.79 0.487
(income [> R100,000]
ethnicity [black or
african]) education
[2-year degree] 		0.17 -1.21 – 1.54 0.813
(income [> R100,000]
ethnicity [black or
african]) education
[4-year degree] 		-0.32 -1.95 – 1.32 0.702
(income [prefer not to
say] * ethnicity [black
or african]) * education
[4-year degree] 		-0.95 -2.53 – 0.63 0.239
(income [> R100,000]
ethnicity [black or
african]) education
[graduate degree] 		0.23 -1.14 – 1.61 0.740
(income [prefer not to
say] * ethnicity [black
or african]) * education
[graduate degree] 		0.04 -1.47 – 1.55 0.959
(income [> R100,000]
ethnicity [coloured])
education [graduate
degree] 		0.60 -1.32 – 2.52 0.540
(income [> R100,000]
ethnicity [black or
african]) education
[high school] 		0.48 -0.59 – 1.55 0.381
(income [prefer not to
say] * ethnicity [black
or african]) * education
[high school] 		-0.29 -1.72 – 1.14 0.689
(income [prefer not to
say] * ethnicity
[coloured]) * education
[high school] 		-0.50 -2.02 – 1.02 0.522
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[high school] 		-0.17 -2.08 – 1.75 0.864
(income [prefer not to
say] * ethnicity [white
or caucasian]) *
education [high school] 		-0.15 -1.73 – 1.43 0.854
(income [prefer not to
say] * ethnicity [black
or african]) * education
[prefer not to say] 		0.55 -0.65 – 1.75 0.366
(income [prefer not to
say] * ethnicity
[coloured]) * education
[prefer not to say] 		-0.21 -1.85 – 1.42 0.798
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[prefer not to say] 		0.68 -1.76 – 3.13 0.584
(income [prefer not to
say] * ethnicity [black
or african]) * education
[some college] 		-0.68 -2.40 – 1.05 0.441
(income [prefer not to
say] * ethnicity
[coloured]) * education
[some college] 		-0.80 -2.62 – 1.02 0.388
(income [prefer not to
say] * ethnicity [white
or caucasian]) *
education [some college] 		-0.77 -2.70 – 1.17 0.436
Observations 1089
R2 / R2 adjusted 0.095 / 0.031

Interaction model 2

  vax status
Predictors Estimates CI p
(Intercept) 0.67 0.11 – 1.22 0.019
religion [hinduism] 0.34 -0.63 – 1.31 0.494
religion [islam] -0.93 -2.47 – 0.61 0.235
religion [no religion] -0.37 -1.14 – 0.41 0.356
religion [other] -0.02 -1.25 – 1.22 0.979
religion [prefer not to
say] 		-0.68 -1.76 – 0.40 0.218
politics [liberal] 0.00 -0.79 – 0.79 1.000
politics [moderate] -0.27 -0.97 – 0.44 0.457
politics [prefer not to
say] 		-0.32 -0.89 – 0.26 0.285
location [rural] -0.04 -0.61 – 0.54 0.895
location [suburban] -0.20 -0.79 – 0.38 0.498
location [urban] -0.01 -0.58 – 0.57 0.984
religion [hinduism] *
politics [liberal] 		0.11 -1.27 – 1.48 0.881
religion [islam] *
politics [liberal] 		0.61 -0.59 – 1.80 0.322
religion [no religion] *
politics [liberal] 		0.44 -0.37 – 1.25 0.290
religion [other] *
politics [liberal] 		-0.06 -0.97 – 0.84 0.894
religion [prefer not to
say] * politics [liberal] 		0.01 -1.54 – 1.57 0.986
religion [islam] *
politics [moderate] 		1.53 -0.33 – 3.40 0.107
religion [no religion] *
politics [moderate] 		0.97 -0.35 – 2.28 0.149
religion [other] *
politics [moderate] 		-0.61 -1.41 – 0.20 0.138
religion [prefer not to
say] * politics
[moderate] 		0.06 -1.07 – 1.18 0.919
religion [hinduism] *
politics [prefer not to
say] 		0.08 -1.29 – 1.45 0.907
religion [islam] *
politics [prefer not to
say] 		0.58 -0.61 – 1.77 0.338
religion [no religion] *
politics [prefer not to
say] 		0.42 -0.21 – 1.04 0.195
religion [other] *
politics [prefer not to
say] 		-0.33 -1.09 – 0.42 0.382
religion [prefer not to
say] * politics [prefer
not to say] 		0.62 -0.39 – 1.62 0.231
religion [hinduism] *
location [rural] 		-0.34 -1.46 – 0.78 0.554
religion [islam] *
location [rural] 		0.27 -0.92 – 1.46 0.659
religion [no religion] *
location [rural] 		0.41 -0.56 – 1.37 0.412
religion [other] *
location [rural] 		-0.61 -2.18 – 0.96 0.445
religion [prefer not to
say] * location [rural] 		1.05 -0.40 – 2.51 0.157
religion [hinduism] *
location [suburban] 		0.20 -0.94 – 1.33 0.734
religion [islam] *
location [suburban] 		0.47 -1.36 – 2.29 0.614
religion [no religion] *
location [suburban] 		0.40 -0.65 – 1.45 0.453
religion [other] *
location [suburban] 		0.05 -1.37 – 1.47 0.943
religion [prefer not to
say] * location
[suburban] 		-0.32 -0.96 – 0.31 0.320
religion [islam] *
location [urban] 		0.77 -0.60 – 2.15 0.271
religion [no religion] *
location [urban] 		0.04 -0.49 – 0.57 0.885
religion [other] *
location [urban] 		0.02 -1.07 – 1.11 0.968
religion [prefer not to
say] * location [urban] 		0.02 -0.46 – 0.50 0.934
politics [liberal] *
location [rural] 		0.00 -0.82 – 0.82 0.997
politics [moderate] *
location [rural] 		0.29 -0.43 – 1.02 0.431
politics [prefer not to
say] * location [rural] 		0.27 -0.33 – 0.87 0.378
politics [liberal] *
location [suburban] 		0.12 -0.71 – 0.95 0.779
politics [moderate] *
location [suburban] 		0.38 -0.36 – 1.11 0.317
politics [prefer not to
say] * location
[suburban] 		0.49 -0.13 – 1.11 0.121
politics [liberal] *
location [urban] 		-0.11 -0.92 – 0.71 0.800
politics [moderate] *
location [urban] 		0.21 -0.52 – 0.93 0.572
politics [prefer not to
say] * location [urban] 		0.23 -0.37 – 0.83 0.444
(religion [islam]
politics [liberal])
location [rural] 		-0.57 -2.39 – 1.25 0.540
(religion [no religion]
politics [liberal])
location [rural] 		-1.00 -2.06 – 0.07 0.066
(religion [other]
politics [liberal])
location [rural] 		1.06 -0.45 – 2.57 0.167
(religion [prefer not to
say] * politics
[liberal]) * location
[rural] 		-0.02 -1.98 – 1.95 0.987
(religion [islam]
politics [moderate])
location [rural] 		-0.52 -2.26 – 1.21 0.556
(religion [no religion]
politics [moderate])
location [rural] 		-0.99 -2.53 – 0.55 0.207
(religion [other]
politics [moderate])
location [rural] 		1.58 -0.01 – 3.18 0.051
(religion [prefer not to
say] * politics
[moderate]) * location
[rural] 		-0.08 -1.86 – 1.69 0.927
(religion [no religion]
politics [prefer not to
say]) location [rural] 		-0.54 -1.42 – 0.35 0.233
(religion [other]
politics [prefer not to
say]) location [rural] 		0.38 -0.91 – 1.67 0.565
(religion [prefer not to
say] * politics [prefer
not to say]) * location
[rural] 		-0.86 -2.28 – 0.57 0.240
(religion [islam]
politics [liberal])
location [suburban] 		0.28 -1.56 – 2.11 0.768
(religion [no religion]
politics [liberal])
location [suburban] 		-1.06 -2.52 – 0.40 0.155
(religion [other]
politics [liberal])
location [suburban] 		-0.56 -2.07 – 0.95 0.469
(religion [prefer not to
say] * politics
[liberal]) * location
[suburban] 		0.74 -0.67 – 2.15 0.304
(religion [islam]
politics [moderate])
location [suburban] 		-1.64 -3.86 – 0.58 0.147
(religion [no religion]
politics [moderate])
location [suburban] 		-1.15 -2.69 – 0.39 0.144
(religion [other]
politics [moderate])
location [suburban] 		0.75 -0.43 – 1.93 0.213
(religion [prefer not to
say] * politics
[moderate]) * location
[suburban] 		0.37 -0.89 – 1.63 0.563
(religion [hinduism]
politics [prefer not to
say]) location
[suburban] 		-0.26 -2.03 – 1.52 0.778
(religion [islam]
politics [prefer not to
say]) location
[suburban] 		-0.76 -2.58 – 1.07 0.416
(religion [no religion]
politics [prefer not to
say]) location
[suburban] 		-0.49 -1.49 – 0.51 0.338
(religion [other]
politics [prefer not to
say]) location
[suburban] 		0.23 -0.87 – 1.33 0.678
(religion [prefer not to
say] * politics
[liberal]) * location
[urban] 		1.09 -0.46 – 2.65 0.169
(religion [islam]
politics [moderate])
location [urban] 		-1.14 -2.92 – 0.64 0.209
(religion [no religion]
politics [moderate])
location [urban] 		-0.91 -2.16 – 0.34 0.154
Observations 1089
R2 / R2 adjusted 0.082 / 0.015

Predict not open to treatment(chatfuel data)

We found:

  • ethnicity != [asian or india, black or africa, coloured, other] -> not open to treatment
  • ethnicity [prefer not to say] -> not open to treatment
  • ethnicity [white or caucasian] -> not open to treatment
  • income [prefer not to say] -> not open to treatment
  • education [graduate degree] -> not open to treatment
  • education [prefer not to say] -> not open to treatment
  • education [some college] -> not open to treatment
  • religion [no religion] -> not open to treatment
  • religion [other] -> not open to treatment
  • ethnicity [prefer not to say] * education [4-year degree] -> not open to treatment
  • income [> R100,000] * education [prefer not to say] -> not open to treatment
  • religion [islam] -> not open to treatment
  • religion [prefer not to say] * politics[moderate] -> not open to treatment
  • religion [islam] * politics [moderate] *location [suburban] -> not open to treatment

Tree model

Sum model

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.09 -0.12 – 0.29 0.397
cv age -0.00 -0.00 – 0.00 0.553
ethnicity [black or
african] 		0.10 -0.09 – 0.28 0.304
ethnicity [coloured] 0.16 -0.03 – 0.35 0.106
ethnicity [other] -0.20 -0.60 – 0.20 0.326
ethnicity [prefer not to
say] 		0.32 0.11 – 0.53 0.003
ethnicity [white or
caucasian] 		0.31 0.11 – 0.50 0.002
income [> R100,000] -0.06 -0.14 – 0.02 0.136
income [prefer not to
say] 		0.05 0.01 – 0.10 0.020
education [2-year degree] 0.13 0.02 – 0.24 0.017
education [4-year degree] 0.01 -0.11 – 0.14 0.833
education [graduate
degree] 		0.11 0.01 – 0.21 0.035
education [high school] 0.04 -0.01 – 0.09 0.112
education [prefer not to
say] 		0.11 0.01 – 0.20 0.028
education [some college] 0.07 0.01 – 0.12 0.018
religion [hinduism] -0.01 -0.22 – 0.21 0.955
religion [islam] -0.01 -0.15 – 0.13 0.883
religion [no religion] 0.10 0.03 – 0.17 0.007
religion [other] 0.16 0.05 – 0.27 0.003
religion [prefer not to
say] 		-0.05 -0.14 – 0.04 0.284
politics [liberal] -0.01 -0.09 – 0.06 0.703
politics [moderate] -0.02 -0.09 – 0.04 0.455
politics [prefer not to
say] 		0.01 -0.05 – 0.06 0.841
location [rural] -0.18 -0.27 – -0.10 <0.001
location [suburban] -0.13 -0.22 – -0.04 0.004
location [urban] -0.13 -0.22 – -0.05 0.002
Observations 1083
R2 / R2 adjusted 0.120 / 0.099

Interaction model 1

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.00 -0.27 – 0.27 1.000
income [> R100,000] -0.12 -1.20 – 0.96 0.829
income [prefer not to
say] 		-0.00 -0.66 – 0.66 1.000
ethnicity [black or
african] 		0.06 -0.22 – 0.33 0.692
ethnicity [coloured] 0.23 -0.09 – 0.55 0.156
ethnicity [other] -0.00 -0.92 – 0.92 1.000
ethnicity [prefer not to
say] 		1.17 -0.03 – 2.37 0.057
ethnicity [white or
caucasian] 		-0.00 -0.33 – 0.33 1.000
education [2-year degree] -0.33 -1.57 – 0.91 0.605
education [4-year degree] -0.00 -0.64 – 0.64 1.000
education [graduate
degree] 		-0.00 -0.64 – 0.64 1.000
education [high school] -0.00 -0.51 – 0.51 1.000
education [prefer not to
say] 		0.00 -0.64 – 0.64 1.000
education [some college] -0.00 -0.66 – 0.66 1.000
income [> R100,000] *
ethnicity [black or
african] 		0.06 -0.98 – 1.11 0.906
income [prefer not to
say] * ethnicity [black
or african] 		0.02 -0.65 – 0.69 0.948
income [> R100,000] *
ethnicity [coloured] 		-0.04 -1.18 – 1.09 0.943
income [prefer not to
say] * ethnicity
[coloured] 		-0.05 -0.76 – 0.66 0.892
income [prefer not to
say] * ethnicity [other] 		0.10 -1.20 – 1.40 0.880
income [> R100,000] *
ethnicity [prefer not to
say] 		0.26 -1.49 – 2.01 0.773
income [prefer not to
say] * ethnicity [prefer
not to say] 		-0.67 -1.66 – 0.32 0.187
income [> R100,000] *
ethnicity [white or
caucasian] 		-0.00 -0.92 – 0.92 1.000
income [prefer not to
say] * ethnicity [white
or caucasian] 		0.60 -0.14 – 1.34 0.113
income [> R100,000] *
education [2-year degree] 		0.45 -0.44 – 1.34 0.326
income [prefer not to
say] * education [2-year
degree] 		0.06 -0.28 – 0.41 0.729
income [> R100,000] *
education [4-year degree] 		0.12 -0.91 – 1.14 0.820
income [prefer not to
say] * education [4-year
degree] 		-0.60 -1.52 – 0.32 0.201
income [> R100,000] *
education [graduate
degree] 		0.26 -0.60 – 1.12 0.549
income [prefer not to
say] * education
[graduate degree] 		0.40 -0.52 – 1.32 0.394
income [> R100,000] *
education [high school] 		0.50 -0.20 – 1.20 0.161
income [prefer not to
say] * education [high
school] 		0.00 -0.90 – 0.90 1.000
income [> R100,000] *
education [prefer not to
say] 		1.12 0.09 – 2.14 0.032
income [prefer not to
say] * education [prefer
not to say] 		-0.10 -0.82 – 0.62 0.786
income [> R100,000] *
education [some college] 		0.04 -0.28 – 0.35 0.826
income [prefer not to
say] * education [some
college] 		0.00 -1.08 – 1.08 1.000
ethnicity [black or
african] * education
[2-year degree] 		0.44 -0.79 – 1.67 0.485
ethnicity [coloured] *
education [2-year degree] 		1.10 -0.29 – 2.49 0.122
ethnicity [prefer not to
say] * education [2-year
degree] 		-0.23 -1.60 – 1.14 0.739
ethnicity [white or
caucasian] * education
[2-year degree] 		0.33 -0.77 – 1.44 0.555
ethnicity [black or
african] * education
[4-year degree] 		0.02 -0.64 – 0.67 0.962
ethnicity [prefer not to
say] * education [4-year
degree] 		1.10 0.15 – 2.05 0.023
ethnicity [black or
african] * education
[graduate degree] 		0.02 -0.64 – 0.68 0.949
ethnicity [coloured] *
education [graduate
degree] 		0.42 -0.50 – 1.34 0.371
ethnicity [prefer not to
say] * education
[graduate degree] 		-0.90 -1.85 – 0.05 0.063
ethnicity [black or
african] * education
[high school] 		0.02 -0.50 – 0.53 0.953
ethnicity [coloured] *
education [high school] 		-0.18 -0.73 – 0.37 0.520
ethnicity [other] *
education [high school] 		0.00 -1.18 – 1.18 1.000
ethnicity [prefer not to
say] * education [high
school] 		-0.67 -1.96 – 0.63 0.314
ethnicity [white or
caucasian] * education
[high school] 		0.29 -0.30 – 0.87 0.341
ethnicity [black or
african] * education
[prefer not to say] 		0.10 -0.56 – 0.76 0.770
ethnicity [coloured] *
education [prefer not to
say] 		-0.23 -0.98 – 0.51 0.544
ethnicity [prefer not to
say] * education [prefer
not to say] 		-1.17 -2.68 – 0.35 0.130
ethnicity [black or
african] * education
[some college] 		0.06 -0.61 – 0.72 0.870
ethnicity [coloured] *
education [some college] 		-0.11 -0.82 – 0.61 0.773
ethnicity [prefer not to
say] * education [some
college] 		-0.50 -1.47 – 0.47 0.314
ethnicity [white or
caucasian] * education
[some college] 		0.33 -0.44 – 1.11 0.399
(income [> R100,000]
ethnicity [black or
african]) education
[2-year degree] 		-0.56 -1.43 – 0.31 0.209
(income [> R100,000]
ethnicity [black or
african]) education
[4-year degree] 		-0.13 -1.17 – 0.90 0.799
(income [prefer not to
say] * ethnicity [black
or african]) * education
[4-year degree] 		0.51 -0.49 – 1.51 0.321
(income [> R100,000]
ethnicity [black or
african]) education
[graduate degree] 		-0.28 -1.16 – 0.59 0.525
(income [prefer not to
say] * ethnicity [black
or african]) * education
[graduate degree] 		-0.05 -1.01 – 0.91 0.911
(income [> R100,000]
ethnicity [coloured])
education [graduate
degree] 		-0.55 -1.77 – 0.67 0.375
(income [> R100,000]
ethnicity [black or
african]) education
[high school] 		-0.51 -1.20 – 0.17 0.138
(income [prefer not to
say] * ethnicity [black
or african]) * education
[high school] 		0.02 -0.89 – 0.93 0.967
(income [prefer not to
say] * ethnicity
[coloured]) * education
[high school] 		0.22 -0.74 – 1.19 0.653
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[high school] 		0.33 -0.88 – 1.55 0.590
(income [prefer not to
say] * ethnicity [white
or caucasian]) *
education [high school] 		-0.39 -1.39 – 0.62 0.450
(income [prefer not to
say] * ethnicity [black
or african]) * education
[prefer not to say] 		0.12 -0.64 – 0.88 0.749
(income [prefer not to
say] * ethnicity
[coloured]) * education
[prefer not to say] 		1.15 0.11 – 2.19 0.030
(income [prefer not to
say] * ethnicity [prefer
not to say]) * education
[prefer not to say] 		1.27 -0.28 – 2.82 0.109
(income [prefer not to
say] * ethnicity [black
or african]) * education
[some college] 		0.01 -1.08 – 1.11 0.981
(income [prefer not to
say] * ethnicity
[coloured]) * education
[some college] 		0.07 -1.09 – 1.22 0.910
(income [prefer not to
say] * ethnicity [white
or caucasian]) *
education [some college] 		-0.43 -1.66 – 0.79 0.488
Observations 1089
R2 / R2 adjusted 0.169 / 0.110

Interaction model 2

  not open to treatment
Predictors Estimates CI p
(Intercept) 0.33 -0.03 – 0.69 0.070
religion [hinduism] -0.12 -0.76 – 0.51 0.698
religion [islam] 1.14 0.14 – 2.14 0.026
religion [no religion] 0.24 -0.27 – 0.74 0.359
religion [other] 0.47 -0.33 – 1.27 0.250
religion [prefer not to
say] 		-0.35 -1.06 – 0.35 0.323
politics [liberal] -0.33 -0.84 – 0.18 0.201
politics [moderate] -0.13 -0.59 – 0.32 0.567
politics [prefer not to
say] 		0.02 -0.36 – 0.39 0.925
location [rural] -0.29 -0.66 – 0.09 0.132
location [suburban] -0.19 -0.57 – 0.19 0.326
location [urban] -0.21 -0.58 – 0.16 0.271
religion [hinduism] *
politics [liberal] 		0.01 -0.88 – 0.91 0.976
religion [islam] *
politics [liberal] 		-0.49 -1.26 – 0.29 0.220
religion [no religion] *
politics [liberal] 		0.01 -0.51 – 0.54 0.959
religion [other] *
politics [liberal] 		0.18 -0.41 – 0.77 0.546
religion [prefer not to
say] * politics [liberal] 		1.35 0.35 – 2.36 0.009
religion [islam] *
politics [moderate] 		-1.34 -2.55 – -0.12 0.031
religion [no religion] *
politics [moderate] 		-0.44 -1.29 – 0.42 0.315
religion [other] *
politics [moderate] 		-0.27 -0.79 – 0.25 0.313
religion [prefer not to
say] * politics
[moderate] 		0.73 0.00 – 1.46 0.050
religion [hinduism] *
politics [prefer not to
say] 		0.01 -0.88 – 0.90 0.979
religion [islam] *
politics [prefer not to
say] 		-0.49 -1.26 – 0.29 0.216
religion [no religion] *
politics [prefer not to
say] 		-0.19 -0.60 – 0.22 0.367
religion [other] *
politics [prefer not to
say] 		0.18 -0.31 – 0.67 0.472
religion [prefer not to
say] * politics [prefer
not to say] 		0.15 -0.51 – 0.80 0.662
religion [hinduism] *
location [rural] 		0.37 -0.35 – 1.10 0.315
religion [islam] *
location [rural] 		-0.72 -1.50 – 0.05 0.067
religion [no religion] *
location [rural] 		-0.28 -0.91 – 0.35 0.377
religion [other] *
location [rural] 		-0.52 -1.54 – 0.50 0.321
religion [prefer not to
say] * location [rural] 		0.31 -0.64 – 1.25 0.524
religion [hinduism] *
location [suburban] 		-0.02 -0.75 – 0.72 0.962
religion [islam] *
location [suburban] 		-1.28 -2.46 – -0.09 0.034
religion [no religion] *
location [suburban] 		0.12 -0.56 – 0.80 0.728
religion [other] *
location [suburban] 		-0.11 -1.03 – 0.81 0.811
religion [prefer not to
say] * location
[suburban] 		0.33 -0.08 – 0.74 0.119
religion [islam] *
location [urban] 		-0.76 -1.65 – 0.13 0.094
religion [no religion] *
location [urban] 		-0.03 -0.37 – 0.32 0.872
religion [other] *
location [urban] 		-0.26 -0.97 – 0.45 0.470
religion [prefer not to
say] * location [urban] 		0.23 -0.08 – 0.54 0.146
politics [liberal] *
location [rural] 		0.29 -0.25 – 0.82 0.292
politics [moderate] *
location [rural] 		0.13 -0.34 – 0.60 0.591
politics [prefer not to
say] * location [rural] 		0.01 -0.38 – 0.40 0.961
politics [liberal] *
location [suburban] 		0.32 -0.22 – 0.86 0.252
politics [moderate] *
location [suburban] 		0.15 -0.32 – 0.63 0.527
politics [prefer not to
say] * location
[suburban] 		-0.03 -0.43 – 0.37 0.882
politics [liberal] *
location [urban] 		0.32 -0.21 – 0.85 0.236
politics [moderate] *
location [urban] 		0.07 -0.40 – 0.54 0.775
politics [prefer not to
say] * location [urban] 		-0.03 -0.42 – 0.36 0.879
(religion [islam]
politics [liberal])
location [rural] 		0.07 -1.11 – 1.26 0.904
(religion [no religion]
politics [liberal])
location [rural] 		0.37 -0.32 – 1.06 0.298
(religion [other]
politics [liberal])
location [rural] 		-0.13 -1.11 – 0.84 0.788
(religion [prefer not to
say] * politics
[liberal]) * location
[rural] 		-1.31 -2.58 – -0.03 0.045
(religion [islam]
politics [moderate])
location [rural] 		0.88 -0.25 – 2.01 0.125
(religion [no religion]
politics [moderate])
location [rural] 		0.44 -0.56 – 1.44 0.387
(religion [other]
politics [moderate])
location [rural] 		0.27 -0.76 – 1.31 0.605
(religion [prefer not to
say] * politics
[moderate]) * location
[rural] 		-0.73 -1.88 – 0.43 0.216
(religion [no religion]
politics [prefer not to
say]) location [rural] 		0.36 -0.21 – 0.93 0.218
(religion [other]
politics [prefer not to
say]) location [rural] 		0.13 -0.71 – 0.97 0.767
(religion [prefer not to
say] * politics [prefer
not to say]) * location
[rural] 		-0.17 -1.10 – 0.75 0.714
(religion [islam]
politics [liberal])
location [suburban] 		0.50 -0.69 – 1.69 0.407
(religion [no religion]
politics [liberal])
location [suburban] 		-0.50 -1.44 – 0.45 0.304
(religion [other]
politics [liberal])
location [suburban] 		-0.66 -1.64 – 0.32 0.185
(religion [prefer not to
say] * politics
[liberal]) * location
[suburban] 		-1.45 -2.37 – -0.54 0.002
(religion [islam]
politics [moderate])
location [suburban] 		1.82 0.37 – 3.26 0.014
(religion [no religion]
politics [moderate])
location [suburban] 		0.34 -0.65 – 1.34 0.499
(religion [other]
politics [moderate])
location [suburban] 		-0.25 -1.02 – 0.51 0.518
(religion [prefer not to
say] * politics
[moderate]) * location
[suburban] 		-0.87 -1.69 – -0.05 0.037
(religion [hinduism]
politics [prefer not to
say]) location
[suburban] 		0.00 -1.16 – 1.16 1.000
(religion [islam]
politics [prefer not to
say]) location
[suburban] 		0.50 -0.68 – 1.68 0.407
(religion [no religion]
politics [prefer not to
say]) location
[suburban] 		-0.10 -0.75 – 0.55 0.763
(religion [other]
politics [prefer not to
say]) location
[suburban] 		-0.24 -0.95 – 0.48 0.513
(religion [prefer not to
say] * politics
[liberal]) * location
[urban] 		-1.34 -2.35 – -0.33 0.009
(religion [islam]
politics [moderate])
location [urban] 		0.90 -0.25 – 2.06 0.126
(religion [no religion]
politics [moderate])
location [urban] 		0.33 -0.48 – 1.15 0.418
Observations 1089
R2 / R2 adjusted 0.117 / 0.053

Part III: Miscellaneous

In this part, we are going to do some analysis that is not directly related to our research questions, but they will give us some general directional insights and hypotheses.

1. Ad Performance Summary Comparison

In this section, we are interested in:

  • Has our overall key cost metric (survey complete, unvaxxed, open to treatment) decreased for this pilot?
  • What were the drivers of this increase/decrease (and at which stages of the funnel)?

Note:

  • This is not a true Ad A|B test, since 1: version 5 chatbot setting is different from version 4b, 2: we applied FB AB test in version 5 (which is not budget optimized), while version 4b is not.
  • For version 5, when calculating the metrics, we include 276 respondents who completed our survey without clicking on the ads.

Takeaways:

  • The overall key cost metric has increased for this pilot (worse than version 4b)
  • The increasing cost could be caused by the A|B testing that we ran on FB Ads Manager. In the FB A|B test, all 5 ad sets have been controlled the same spending, instead of cost-benefit optimized, resulting in a lower clickthrough rate
  • So, we think we cannot compare the campaigns by simply looking at the key cost metric
  • Based on other metrics, we found the new pilot has better performance when we look at the drop-off rates of each section in the chatbot.

Metrics explanation:

  • Impressions (Total Count) = the total number of times our ad has been viewed
  • Clickthrough (%) = #clicks / #impressions
  • Messages Sent (%) = #conversations / #clicks
  • Consent Obtained (%) = #consents / #conversations
  • Core Survey Complete (%) = #forking section completed / #consents
  • Treatment Complete (%) = #treatment section completed / #forking section completed
  • Demo Questions Complete (%) = #demog section completed / #treatment section completed
  • Full Survey Complete (%) = #full chat completed / #demog section completed
  • Total characters elicited per completed survey (treatment) = average #character in best treatment explanation per full chat completed
  • Avg characters elicited per completed survey (impediment explanations) = average #character in impediment explanations per full chat completed
  • Cost per Impression = amount spent / #impressions (in USD)
  • Cost per Link Click = amount spent / #clicks (in USD)
  • Cost per Survey Complete (All participants) = amount spent / #full chat completed (in USD)
  • Cost per Survey Complete (Unvax) = amount spent / #full chat completed with unvaccinated participants (in USD)
  • Cost per Survey Complete (Unvax, Open to Treatment) = amount spent / #full chat completed with unvaccinated and open to treatment participants (in USD)

2. Ad Theme analysis

In our last pilot ads with an “unnecessary” theme were promoted by Facebook’s algorithm since the cost per messaging conversation is cheaper.

What we are interested in:

  • When we look at the bottom of the funnel using our key cost metric (survey complete, unvaxxed, open to treatment), do “unnecessary” ads still outperform “risky” and “inaccessible” ads?
  • If so, by how much?
  • How should we shift our ad creative using this new information?

Takeaway:

  • In version 5, we found “unnecessary” ads still outperform “risky” and “inaccessible” ads in all three metrics. “unnecessary” ads is 5%/15% more cost-efficient than “risky”/“inaccessible” ads in Cost per Survey Complete; 30%/45% more cost-efficient than “risky”/“inaccessible” ads in Cost per Unvax Survey Complete; 30%/33% more cost-efficient than “risky”/“inaccessible” ads in Cost per Unvax and Open to treatment Survey Complete
  • While in version 4b, we found “unnecessary” ads have the worst performance. “risky” ads are the best based on the cost metrics. Surprisingly, the (unvax, open to treatment survey completed) in “unnecessary” ads have a much higher cost (1.46) than that of in “unsafe” (0.662) and “inaccessible” (0.711).
  • Hypothesis: based on the detailed summary table in Version 4b tab, we deduced that Facebook promote Ad sets by comparing cost per impression/ cost per link clicked

Version 5

Takeaway: “unnecessary” theme is the best according to the key cost metrics

Version 4b

Takeaway: “unnecessary” theme is NOT the best