RMarkdown setup

## Global options
knitr::opts_chunk$set(echo=TRUE,
               cache=TRUE,
               prompt=FALSE,
               tidy=TRUE,
               comment=NA,
               message=FALSE,
               warning=FALSE)

# Libraries
library(readxl)
library(readr)
library(ggplot2)
library(dplyr)
library(knitr)
library(ggthemes)
library(tidyverse)
library(kableExtra)

Agenda

• About the project and the survey
  • sampling strategy
  • online survey set-up

• Facebook Ad Campaign
  • sampling via the FB Ads
  • ad performance over time and by adset
  • Fb Ads precision

• Sample quality assessment (work in progress)
  • Eligibility to participate
  • Reluctant vs. Engaged respondents
    

• Key lessons learnt

The study

The project Migrant Essential Workers: November 2020-March 2022

The team

• Prof. Sharon Wright (PI, Glasgow)
• Dr Anna Gawlewicz (Glasgow)
• Dr Kasia Narkowicz (Middlesex)
• Dr Aneta Piekut (Sheffield)
• Dr Paulina Trevena (Glasgow)

• Partners England: Centrala, POSK Polish Centre
  
• Partners Scotland: Fife Migrants Forum, PKAVS

Population of study

Polish essential workers in the UK

• were adult Polish people (aged 18+, as of survey day)

AND

• who have worked in the essential work sectors in the UK during the Covid-19 pandemic (since March 2020, until February/March 2021)

AND

• lived in the UK for at least three months in 2020.

Population of study

Polish essential workers in the UK

As essential work we understand the type of work that is essential for keeping British society and economy running during the pandemic. Hence, essential workers are for example health and care workers, staff maintaining public safety (e.g. police, fire service) or delivering pivotal services (e.g. public transport, child care for essential workers, food production and sale, product delivery), among others. In the study, we are interested in Polish essential workers in the UK who are either formally (self-)employed or work informally (e.g. cash-in hand work).

UK Government, 8 January 2021 here
Scottish Government, 20 January 2021 here

Project so far

• Aug 21: Fieldwork finished
  • ~1,100 responses to survey incl. 372 to open Q (Feb/March 2021)
  • 40 migrant + 10 expert interviews

• Oct 21: Transcription complete
  • Since Nov 2021: Ongoing analysis

• Impact work:
  • Provided evidence on vaccine hesitancy (Scot. Gov Vaccine Inclusive Steering Group & Scot. Parl. Covid-19 Recovery Committee)
  • Materials co-produced with partners: policy rescommendation, toolkit and resources for migrants
  • Art exhibition (Sept 21: Secured £20k for spin-off project)

Survey fieldwork

• Convenience sampling - 22 February - 12 April (7 weeks)
  • via partner organisation (POSK + Polish Embassy, Polish Expats Association, PKAVS Minority Communities Hub, FIFE Migrants)
  • via our social media channels (Facebook, Twitter)

• Facebook Ad Campaign - 27 February - 26 March (4 weeks)

Survey: online mode - Qualtrics specs

• 2 language versions: English and Polish version
  • Polish version automatically linked to all Facebook Ads

• Cookies left to eliminate duplicates (‘Avoid ballot stuffing’ option in Qualtrics)
  • Full anonymisation (IP address not collected)

• URL parameters used to collect information on:
  • language
  • distribution channel
  • Facebook Ad set / strata (region X age group)
  • Facebook Ad photo
  • E.g. https://shefmethods.qualtrics.com/jfe/form/SV_3CNCjfj8iNGygIK?Q_Language=PL&Dis=8&Reg=1&Age=1&Pic=1

Survey sample by recruitment channel (N/%)

Screening for eligibility (1)

• To take part in this survey participants had to:
  • be Polish,
  • currently live or lived in the UK during the Covid-19 pandemic (since March 2020) for at least three months,
  • and have worked in one of the so called ‘essential work’ sectors during the Covid-19 pandemic.

• Survey preview is available here https://shefmethods.qualtrics.com/jfe/preview/SV_3CNCjfj8iNGygIK?Q_CHL=preview&Q_SurveyVersionID=current

Screening for eligibility (2)

• Not eligible if ‘None of the above’ selected

Link to the qualitative research

• The final question of the survey automatically redirected participants to a separate Qualtrics survey and collected email addresses for the voucher draw and a further consent to take part in a follow-up qualitative research, and emails:
  • 1,005 left emails for a voucher draw,
  • 640 agreed to be contacted about the qualitative study.

• As this personal data was disconnected from the main survey, the sampling for qualitative study was more challenging.

Facebook Ad Campaign: set up

• Targeted population - Polish people living in the UK:

  • Geolocation: UK & Behaviour: Lived in Poland (past)
  • Aged 18-65 (‘working age’)
  • Targeted men only in some weeks

• Buying type: auction

• Campaign objective: Traffic/Clicks (not Reach or Conversion - FB Pixel not used)

• Budget: total spent on Fb ads - £1,718

  • Extra costs for a marketing company /intermediary - Uni. of Glasgow requirement (+ £460)
  • Started with £1,200 (£200 per ad set)
  • £518 added in week 4, most into non-England ad sets

Facebook Ad Campaign: ad sets (1)

Facebook Ad Campaign: ad sets (2)

• Annual Population Survey (2019; N=2,271; ONS 2020) used to establish boundaries of strata / ad sets
  • For 16-64 years old, but all industries (not possible to distinguish ‘essential workers’)
  • Data one year old (+ 1 year for calculations)
  • Aged 18-31 (33.6%)
  • Aged 32-39 (35.1%)
  • 40 years old and more (31.3%)

• Why gender was not used as a strata / ad set?
  • Potentially too many as sets? 6 x 2 = 12 - budget would be spread too thinly?
  • No prior knowledge of gender split
  • Prioritized increasing sample size in Scotland

Facebook Ad Campaign: one ad

Ad text: Are you a Pole [flag] living in the UK [flag]? Tell us how Covid affected your life Within a pic: ’Are you from Poland? Migration, Covid, Work (selected industries) ‘Win a £100 or £50 voucher’ Take part in a study - 15-minute survey Carousel with 5 pictures 3 ads, with different lead picture .

Facebook Ad Campaign: pictures

• Lead pictures for 3 ads within each ad set: 1, 3 and 5

Key takeaways & reflections

FB population: potential (P) vs. actual (A) reach

** Note: Total Actual Reach reported by Facebook was 102,434

Facebook Ad Campaign performance

• ‘Facebook scrollers’ vs. ‘Ad clickers’ vs. ‘Survey respondents’ (Neundorf & Öztürk 2021)

  • Reach & Impressions / ‘Facebook scrollers’ - saw information, but not acted

  • Clicks on the ad / Traffic to survey website - demostrated some interest

  • Conversions - consent and started the survey:

    • but not eligible (eliminated via screening questions)
    • eligible (PL essential worker), but not engaged the survey (>40% nonresponse)
    • eligible (PL essential worker) and survey finished –> valid response

FB reach -> impressions -> clicks -> sample

FB reach & clicks by device type

FB ad precision

• We can check the match in terms of age and region in the UK in the final, valid sample (N=735)

• Match check is only for those who replied to these two questions at the end of the survey

  • Age question (drop down list with birth year) - 47 non-responses (8%)
  • Region question (single choice) - 11 non-responses (1.5%)

Fig 4.1: FB ad precision - region

* 1.8% (13 cases) mismatch: 1 Scotland as England, 12 England as Other

Fig 4.2: FB ad precision - age (1)

* some mismatch, but might be due the birth-year question used (age calculated)

FB ad precision - age (2)

• Age mismatch between Facebook-side data and respondent-provided age:

  • 5% if all counted (34 cases)
  • 3% excluding neighbour ages (24 cases)

• Yet, overall region and age match by Facebook Ad Manager ‘pretty good’.

Key takeaways & reflections

Sampling flowchart - whole sample

Sampling flowchart - convenience sample

Sampling flowchart - Facebook sample

Clickers vs. Respondents

• Clicks on the ad / Traffic to the survey website - demonstrated some interest, but might resign for various reasons

• Conversions - moved to page 2, gave consent and started the survey:

  • Eligible (after the three screening questions)
  • Respondents:
    • Unit non-response (>40% non-responses to key questions)
    • Reluctant respondents - not fully engaged the survey (<40% non-responses)
    • Engaged respondents - 100% valid responses

Fig.5.1: Survey sample flow

Fig.5.2: Item nonreponse among eligible (%)

Key takeaways & reflections

Faceook Ad Campaign -
changes made to sampling (ughrrr!!!)

Fig.6.1: Daily reach - all

Fig.6.2: Daily reach - gender

Fig.6.3: Daily reach - by ad set

Fig.6.4: Number of clicks - all

Fig.6.5: Number of clicks - by ad set

Fig.6.6: Cost per click - all

Fig.6.7: Cost per click - by ad set

Fig.6.8: Frequency (Impressions/Reach) - all

• How often people in the audience seen an ad per day

Fig.6.9: Frequency (Impressions/Reach) - by ad set

Key takeaways & reflections

Fig.7.1: Eligible responses over time

Fig.7.2: FB Ad: clicks -> respondents (N)

Fig.7.3: FB Ad: clicks -> respondents (%)

FB sampling by stage

Fig.7.4: FB sampling - stage & gender

FB sampling - understanding gender bias

Fig.7.5: Number of clicks - by gender

Fig.7.6: Cost per click - by gender

Fig.7.7: Frequency (Impressions/Reach) - by gender

Fig.7.8: Sampling channel & gender

Key takeaways & reflections

Positives

• Large sample size achieved - N=1,100 in total; 735 via FB Ad (66%).

• Engaged audience recruited via FB ad:

  • about 80% of FB sample had max. question not answered,
  • many long open-question responses.

• Collected good experience on how FB Ad Manager works among the Polish community in the UK, especially its effectiveness over time and outside England.

For the next time…

• Run two different sampling pilots, separate from the main fieldwork

  • e.g. one with the optimization by clicks and another by conversions + Pixel.

• Maximize resources: shorter survey fieldwork, larger budget and more adsets.

• Have more control over the survey management and direct access to the FB manager.

Thank you!

Follow us at:

https://twitter.com/MigrEssentWork

Aneta Piekut

a.piekut@sheffield.ac.uk