Highs and lows of using Facebook

to recruit hard-to-reach groups for research


Aneta Piekut, University of Sheffield

Anna Gawlewicz, University of Glasgow

‘Facebook as a research tool’ seminar series

07/04/2022

Agenda (1)

1 About the project

2 Online survey

  • sampling strategy: combining Facebook Ads recruitment with other Facebook- and social-media based recruitment techniques
  • online survey set-up

3 Sampling with the Facebook Ad Campaign

  • Facebook Ad Campaign structure
  • indirect recruitment through ad design and its reception among the Facebook population

Agenda (2)

4 Facebook Ad Campaign performance

  • Fb Ads precision
  • ad performance by an ad set + reach, ad clicks, conversions & cost per valid response

5 Facebook Ad Campaign performance over time

  • managing gender bias in sampling

6 Sample(ing) quality assessment

  • eligibility to participate: Clickers vs. Respondents
  • Reluctant vs. Engaged respondents

7 Challenges and key lessons learnt

1


The project on

Polish essential workers

in the UK


The project : Nov 2020-Mar 2023

www: www.migrantessentialworkers.com
Twitter: MigrEssentWork
Facebook: MigrantEssentialWorkers

Project logo

The team

  • Prof. Sharon Wright (PI, Glasgow)
  • Dr Anna Gawlewicz (Glasgow)
  • Dr Kasia Narkowicz (Middlesex)
  • Dr Aneta Piekut (Sheffield)
  • Dr Paulina Trevena (Glasgow)

Uni logos

  • Partners England: Centrala, POSK Polish Centre
  • Partners Scotland: Fife Migrants Forum, PKAVS

Essential work

Essential work is the type of work that is essential for keeping British society and economy running during the Covid-19 pandemic.

Facebook comments

UK Government, 8 January 2021 here
Scottish Government, 20 January 2021 here

Methods

  • Online migrant survey
    • (1,105 responses incl. 372 to open question)

  • Qualitative:
    • 40 migrant interviews
    • 10 expert interviews

  • This paper: focus on recruitment for the online survey.

2



The online survey


Survey fieldwork

  • Convenience sampling - 22 Feb - 12 Apr (7 weeks)

  • Facebook Ad Campaign - 27 Feb - 26 Mar (4 weeks)

Table 2.1. Survey fieldwork weeks by sampling type

Weeks 1 2 3 4 5 6 7 8
W/c 22/02 01/03 08/03 15/03 22/03 29/03 05/04 12/04
All - — —— —— —— —— —— —— -
FB-ad ✓ ✓ ✓ ✓

  • 2 language versions:
    • English and Polish
    • Polish version automatically linked to all Facebook Ads

  • Combining various online recruitment techniques

    • Promotion via our own social media (FB + Twitter + www)
    • Promotion on stakeholders’ social media (FB + Twitter + newsletters)
    • Personal research networks
    • AND: FB ads

  • Results

    • 66% respondents via FB ads
    • 34% respondents via other channels
    • More diversity within the sample

Fb ad example

Fig 2.1. Sample by recruitment channel (N/%)

Screening for eligibility (1)

  • To take part in this survey participants had to:
    • be Polish,

    • currently live or lived in the UK during the Covid-19 pandemic (since March 2020) for at least three months,

    • and have worked in one of the so called ‘essential work’ sectors during the Covid-19 pandemic.

Screening for eligibility (2)

  • Not eligible if ‘None of the above’ selected

Sector question

3


Sampling
with
the Facebook Ad Campaign


Facebook Ad Campaign: set up

  • Targeted population - Polish people living in the UK:

    • Geolocation: UK & Behaviour: Lived in Poland (past)
    • Aged 18-65 (‘working age’)

  • Buying type: auction

  • Campaign objective: Traffic/Clicks (not Reach or Conversion - FB Pixel not used)

  • Budget: total spent on Fb ads - £1,718

    • Extra costs for a marketing company /intermediary - Uni. of Glasgow requirement (+ £460)
    • Started with £1,200 (£200 per ad set)

Facebook Ad Campaign: ad sets (1)

Ad sets

Facebook Ad Campaign: ad sets (2)

  • Annual Population Survey (2019; N=2,271; ONS 2020) used to establish boundaries of strata / ad sets

    • For 16-64 years old, but all sectors (not possible to distinguish ‘essential workers’)
    • Data one year old (+ 1 year for calculations)
    • Aged 18-31 (33.6%)
    • Aged 32-39 (35.1%)
    • 40 years old and more (31.3%)

  • Why gender was not used to define ad sets:

    • Potentially too many as sets? 6 x 2 = 12 - budget would be spread too thinly?
    • No prior knowledge of gender split
    • Prioritized increasing sample size in Scotland

Facebook Ad Campaign: one ad example

One ad example
  • Ad text:

    • Are you a Pole [flag] living in the UK [flag]? Tell us how Covid affected your life

    • Within a pic:
      • ’Are you from Poland? Migration, Covid, Work (selected industries)
      • ‘Win a £100 or £50 voucher’

    • Take part in a study - 15-minute survey

    • Carousel with 5 pictures
      • 3 ads, with different lead picture

        .

Facebook Ad Campaign: pictures

  • Lead pictures for 3 ads within each ad set: 1, 3 and 5

Community responses (1)

Facebook comments
  • Misunderstanding of FB Ad campaign

    .

Community responses (2)

Facebook comment
  • FB Ad campaign creating space for conspiracy theories about Covid-19

    .

4


Facebook Ad Campaign performance
overall and by ad set


FB population: potential (P) vs. actual (A) reach

  • Estimated population of Polish migrants aged 18-64 in the UK 411,000:

    • 350K in England,
    • 61K in Scotland, Northern Ireland, Wales & Crown islands.

  • Our Fb Ad Campaign ‘reached’ 26% of Facebook users Polish migrants

Table 4.1. Facebook Ad Campaing reach - estimated vs. actual (budget-determined)
Ad set Reach-Potential Est. Daily reach Est. Daily clicks Reach-Actual Reach A/P (%)
1 England - 18-31 110000 1500-4300 7-25 20456 18.6
2 England - 32-39 120000 1700-5000 11-30 22424 18.7
3 England - 40+ 120000 1700-4900 10-28 23871 19.9
Total England 350000 NA NA 66751 19.1
4 Rest of the UK - 18-31 19000 1000-3000 6-23 12408 65.3
5 Rest of the UK - 32-39 21000 1100-3100 5-21 13681 65.1
6 Rest of the UK - 40+ 21000 943-2700 5-22 14045 66.9
Total Rest of the UK 61000 NA NA 40134 65.8
TOTAL 411000 NA NA 106885 26.0

** Note: Total Actual Reach reported by Facebook was 102,434

Fig 4.1: FB ad precision - region

* 1.8% (13 cases) mismatch: 1 Scotland as England, 12 England as Other

Fig 4.2: FB ad precision - age (1)

* 5% mismatch, but might be due the birth-year question used (age calculated)

Facebook Ad Campaign performance

  • ‘Facebook scrollers’ vs. ‘Ad clickers’ vs. ‘Survey respondents’ (Neundorf & Öztürk 2021)

    • Reach & Impressions / ‘Facebook scrollers’ - saw information, but not acted

    • Clicks on the ad / Traffic to survey website - demostrated some interest

    • Conversions - gave consent and started the survey:

      • but could be not eligible (eliminated via screening questions)
      • eligible (PL essential worker), but not engaged in the survey (>40% nonresponse)
      • eligible (PL essential worker) and survey finished –> valid response

Survey flow

FB reach -> impressions -> clicks -> sample

  • Much smaller population in the ‘Rest of the UK’ - more aggressive and expensive sampling

Table 4.2. Facebook Ad Campaing stats: reach, impressions, clicks & costs
Adset Reach [R] Impressions (I) Frequency (I/R) Link clicks Unique link clicks Amount spent (GBP) Valid FB surveys Cost per 1 valid survey (GDP)
1 England - 18-31 20456 90855 4.44 735 667 225.00 177 1.27
2 England - 32-39 22424 86649 3.86 673 606 225.00 153 1.47
3 England - 40+ 23871 89413 3.75 764 672 225.00 110 2.05
Total England 66751 266917 4.00 2172 1945 675.00 440 1.53
4 Rest of the UK - 18-31 12408 119309 9.62 596 528 347.66 109 3.19
5 Rest of the UK - 32-39 13681 119054 8.70 500 446 347.66 97 3.58
6 Rest of the UK - 40+ 14045 127318 9.07 631 555 347.66 89 3.91
Total Rest of the UK 40134 365681 9.11 1727 1529 1042.98 295 3.54
TOTAL 106885 632598 6.18 3899 3382 1717.98 735 2.34

5


Facebook Ad Campaign
performance over time

– sampling “harder-to-reach”


Changes made to the FB Campaign

  • The original campaign length was 3 weeks / 21 days

    • started with £200 per ad set (£1,200)

  • Day 7 revision - switched off sampling women

  • Day 18 revision - reusing remaining funds (£300)

    • sampling among all genders enabled
    • extended campaign into 28 days (4 weeks)
    • extra budget added: England - £25 per ad set & Rest - £75 per ad set

  • Day 26 revision - hoping to boost sampling in the short-time

    • Scotland switched-off
    • extra £218 found and placed into non-England ad sets
    • too late? - algorithm has not managed to re-learn

Fig.5.1: Daily reach - all

Fig.5.2: Daily reach - by ad set

Fig.5.3: Number of clicks - all

Fig.5.4: Number of clicks - by ad set

Fig.5.5: Number of respondents - by region

FB sampling - understanding gender bias

  • 67% respondents from FB sample identified as women, 32% as men, and 1% as non-binary

    • (ONS 2020, 60% key workers in 2019 - women)

  • Male respondents across stages:
    • 1 - 90 respondents
    • 2 - 162 respondents
    • 3 - 8 respondents
    • 4 - 8 respondents

  • More funds at Stages 3 & 4 resulted in less male ‘Clickers’ in comparison to Stage 1, and even less ‘Respondents’

Fig.5.6: Daily reach - gender

Fig.5.7: Number of clicks - by gender

Fig.5.8: Number of respondents - by gender

Fig.5.9: FB sampling - stage & gender

Fig.5.10: Sampling channel & gender

Reflections on sampling

  • The smaller the sample (i.e. outside England; 61K Poles on FB), more resources needed to recruit respondents (444 vs 279 respondents).

  • The final stages with extra budget - men still clicked, but not converted into respondents at the same rate as women

    • not interested?
    • not eligible / not essential workers?
    • needed more time to convert?
    • FB targeting algorithm changed?

  • Gender imbalance ‘better’ in FB sampling, in comparison to the convenience sampling

    • but which one is closer to the population of study?

6


What does it all mean

for sample(ing) quality?


Ad Clickers vs. Respondents

  • Clicks on the ad / Traffic to the survey website - demonstrated some interest, but might resign for various reasons

  • Conversions - moved to page 2, gave consent and started the survey:

    • Eligible (after the three screening questions)
    • Respondents:
      • Unit non-response (>40% non-responses to core questions)
      • Reluctant respondents - not fully engaged the survey (1-40% non-responses to core questions)
      • Engaged respondents - 100% valid responses to core questions

Fig.6.1: Campaing outcomes by day (N)

Campaign outcomes by ad set

Table 6.1. Campaign outcomes by adset (days in brackets, % FB Reach)
Adset N Reach Clicks Unique Clicks Conversions Eligible Respondent Engaged Resp.
ENG (18-31) N 20456 735 667 326 212 177 148
ENG (32-39) N 22424 673 606 270 176 153 122
ENG (40+) N 23871 764 672 219 124 110 89
REST (18-31) N 12408 596 528 202 135 109 93
REST (32-39) N 13681 500 446 177 116 97 79
REST (40+) N 14045 631 555 168 101 89 63
Total N 102434 3899 3382 1362 864 735 594
Adset % Reach Clicks Unique Clicks Conversions Eligible Respondent Engaged Resp.
ENG (18-31) % 100 3.59 3.26 1.59 1.04 0.87 0.72
ENG (32-39) % 100 3.00 2.70 1.20 0.78 0.68 0.54
ENG (40+) % 100 3.20 2.82 0.92 0.52 0.46 0.37
REST (18-31) % 100 4.80 4.26 1.63 1.09 0.88 0.75
REST (32-39) % 100 3.65 3.26 1.29 0.85 0.71 0.58
REST (40+) % 100 4.49 3.95 1.20 0.72 0.63 0.45
Total % 100 3.81 3.30 1.33 0.84 0.72 0.58

  • Youngest - most likely to convert into survey participants, conversion rate lowest in 40+.
    • Higher conversion rate outside England.

Fig.6.2. Conversions by ad set

Campaign outcomes by sampling stage

Table 6.2. Campaign outcomes by samling stage (target age group in brackets, % FB Reach)
Sampling stage N FB Reach* FB Clicks Unique Clicks Conversions Eligible Respondent Engaged Resp.
Stage 1 (1-7) N 108331 1630 1503 697 449 377 313
Stage 2 (8-18) N 151332 1208 1148 318 202 174 140
Stage 3 (19-25) N 75130 731 680 254 168 147 115
Stage 4 (26-28) N 45842 330 305 89 45 37 26
Total N 380635 3899 3636 1362 864 735 594
Sampling stage % FB Reach FB Clicks Unique Clicks Conversions Eligible Respondent Engaged Resp.
Stage 1 (1-7) % 100 1.50 1.39 0.64 0.41 0.35 0.29
Stage 2 (8-18) % 100 0.80 0.76 0.21 0.13 0.11 0.09
Stage 3 (19-25) % 100 0.97 0.91 0.34 0.22 0.20 0.15
Stage 4 (26-28) % 100 0.72 0.67 0.19 0.10 0.08 0.06
Total % 100 1.02 0.96 0.36 0.23 0.19 0.16

*Reach - sum of FB Reach per day

Fig.6.3. Conversions by sampling stage

Survey responses by sampling stage

  • Respondents per day (Mean=26.3):
    • Stage 1 (1-7) = 53.9
    • Stage 2 (8-18) = 15.8
    • Stage 3 (19-25) = 21.0
    • Stage 4 (26-28) = 12.3

  • Cost per respondent (Mean=£2.34):
    • Stage 1 (1-7) = £1.11
    • Stage 2 (8-18) = £3.69
    • Stage 3 (19-25) = £2.13
    • Stage 4 (26-28) = £9.36

  • Fully engaged respondents with 0 non-response to core questions (80.8% of Respondents):
    • Stage 1 (1-7) = 83.0%
    • Stage 2 (8-18) = 80.5%
    • Stage 3 (19-25) = 78.2%
    • Stage 4 (26-28) = 70.3%

Fig.6.4: Survey sample flow

Fig.6.5: Item nonreponse among eligible (%)

Reflections on the survey enegagement

  • Stage 1 - 377 respondents / 50% sample (of which 75% women), stage 2 - 174, stage 3 & 4 - 184.

  • More reliable respondents in the first 2 sampling stages.

  • FB-ad sampling ‘produced’ more dedicated respondents than convenience sampling.

  • Representativeness?

    • Some assessment of the sample quality is in our Blog 1 and 2

    • Comparisons can be made with 2021 Census data soon

7



Challenges

and key lessons learnt


Positives

  • Unexpectedly large sample size achieved - N=1,105 in total; 735 via FB Ad (66%).

  • Engaged audience recruited via FB ad:

    • 81% of FB respondents answered all core survey questions,
    • qualitative material collected alongside,
    • many long open-question responses.

  • Gained experience on the use of FB Ad Manager for recruiting even ‘harder-to-reach’ population (i.e. Polish essential workers in the UK).

Challenges

  • Working with an intermediary: not all requests picked up immediately or correctly

  • FB advertising creating unwanted space for conspiracy theories about Covid-19

  • Comments under FB ads: research team exposed to online hate (can be overwhelming and demotivating)

  • Misreading of FB ads: accusations of stereotyping of Polish migrants in the UK, complaints

What would we have done differently?

  • Run two different sampling pilots, separate from the main fieldwork, e.g. one with the optimization by clicks and another by conversions + Pixel

  • Maximize resources: shorter ad campaign, larger budget and more ad sets

  • Approach intermediary earlier in the process

  • Work with intermediary with experience/understanding of promotion for research (not only marketing)

  • Have more control over the FB ad campaign

  • Be more prepared for hateful responses to/criticism of FB ads


Thank you!

Aneta Piekut, a.piekut[at]sheffield.ac.uk

Anna Gawlewicz, Anna.Gawlewicz[at]glasgow.ac.uk

Extras

Survey: online mode - Qualtrics specs

Fig.8.1: Cost per click - all

Fig.8.2: Cost per click - by ad set

Fig.8.3: Frequency (Impressions/Reach) - all

  • How often people in the audience seen an ad per day

Fig.8.4: Frequency (Impressions/Reach) - by ad set

Fig.8.5: Cost per click - by gender

Fig.8.6: Frequency (Impressions/Reach) - by gender

Fig.8.7: Responses over time