2 October 2018

Small-area estimates of public opinion: motivation

  • Why? A Holy Grail of political science. Studies of representation in district based political systems: do the legislative votes of elected politicians reflect the preferences of their districts?
  • Suppose a representative \(i \in 1, \ldots, n\) has a legislative voting history \(\mathcal{H}_i = \cup_{j=1}^m V_{ij}\), \(V_{ij} \in \{ Y, N \}\).
  • Reduce \(\mathcal{H}_i\) to \(\xi_i = h(\mathcal{H}_i) \in \mathbb{R}^d, d << m\), a voting score or ideal point estimate (e.g., Clinton, Jackman Rivers; Poole and Rosenthal).
  • Studies of representation: \(\xi_i = f(x_i)\), where \(x_i\) is a measure of preferences in district \(i\). Is \(f\) monotone? Outlier \(\xi_i\) or \(x_i\)?

Small-area estimates of public opinion: motivation

  • \(\xi_i = f(x_i)\), where \(x_i\) is a measure of preferences in district \(i\).
  • Parliamentary systems have few departures from party line voting in legislatures: \(\xi_i \approx \xi_{p(i)}\), where \(p(i)\) is the party of legislator \(i\).
  • In a strong, two-party, Westminister system, \(\xi_i \in \{ -c, c\}\).
  • Does party discipline oversmooth \(f\) with respect to district preferences?
  • United States: is a combination of primary voting, gerrymandering and polarisation sending more extreme politicians to Congress?
  • “Out of step, out of office”? Electoral sanctions for \(\xi_i \neq f(x_i)\)?

Measuring district preferences is not trivial

  • The Miller-Stokes problem: a public opinion survey with national coverage — and a conventional sample size — has very few respondents per legislative district.

Measuring district preferences is not trivial

  • US: \(N\) = 2,000 survey respondents, \(n\) = 435 CDs, 4.6 respondents per district.
  • Australia: AES 2016, \(N\) = 2,787, \(n\) = 150 CEDs, 18.6 respondents per district.

Respondents per CED, Australian Election Survey 2016

Other ideas…

  • Do we need surveys at all? Other measures: vote shares, demographics?
    • Need a model to link that information to attitude data (and hold that thought…)
  • Do we need to measure all districts? Stratify by type of district?
    • \(N\) typically not big enough for even a small number of sample districts.

Internet surveys

  • lower fixed costs than other survey modes
  • sometimes close to zero marginal cost (“opt-in”)
  • Hence, possible to get useable RPD?
  • No free lunch: low cost, opt-in recruitment typically generates bias through respondent self-selection.
  • But can we “make it up in volume”? ``Useful’’ data in the torrent of self-selected, less useful data?

Voter Advice Applications

  • Internet-administered tool for citizens to learn about candidates and issues ahead of an election
  • Media partners use to drive traffic to their web sites, generate news content from the resulting data
  • Also aligns with public service mission of public broadcasters
  • VoxPop Labs (Toronto) supplies VAAs for BBC, CBC, PBS (USA) and ABC, in conjunction with teams of local academics offering advice on issue, candidates, question-wording.
  • VAA \(\neq\) “survey”; no sampling per se, short, simple set of questions.

ABC Vote Compass 2016, 1.3M “responses”

ABC Vote Compass 2016

  • Truly massive amount of data by any standard, 1.3M partial cases.

  • 771K unique cases from citizen adults, with complete data on vote intention and demographics; still very big.

  • But… promoted by the ABC to ABC listeners and viewers, typically alongside news and current affairs content
    • on-line only
    • English language only

  • Suggests data will be quite biased; tempers initial excitment about large sample size.

Respondents per CED, 2016 Vote Compass

  • Geo-coded to CED by R supplying postcode: R asked to self-resolve ambiguities in postcode to CED lookup.

Top and bottom 5 CEDs by count, 2016 Vote Compass

Rank Division Count
1 Canberra 19,598
2 Melbourne 16,009
3 Fenner 15,209
4 Sydney 14,337
5 Brisbane 13,005
146 Chifley 1,561
147 McMahon 1,470
148 Blaxland 1,302
149 Werriwa 1,272
150 Fowler 1,054

Bias in vote intentions, national level

Bias in vote intentions, by division

Post-stratification

  • Post-stratification: after collecting survey data (“post”), allocate cases to “strata” (or cells) with known population prevalance; weight strata such that survey distribution matches population distribution.
  • Most common technique is rim-weighting or “raking”:
    • “strata” are cells of a \(J\)-dimensional table: \(\mathcal{T} = X_1 \times X_2 \times \ldots X_J\).
    • iteratively “rake” over dimension \(j\) of \(\mathcal{T}\), forming weights such that the weighted sample marginal distribution of \(X_j\) matches the population marginal distribution of \(X_j\).
  • Industry-standard since 1943 (Deming).

Post-stratification

  • Relies on having population marginal distributions for \(X_j\): e.g., demographics from the Census, votes from election returns.
  • After post-stratification adjustments, survey marginal distributions guaranteed to match population marginals \(\forall\ X_j\).
  • No guarantees that adjusted survey joint distributions match population analogues.
  • Weights in under-represented “corners” of the survey data can get very large.

Model-assisted post-stratification

  • Consider a variable \(y\): measured on survey, not available in trusted data source: e.g., political attitudes, issues preferences.
  • Small area estimation of \(y\) is a prediction problem (e.g., Royall et al, Brewer).
  • In small area \(s\), use the (biased) survey data to learn about a model \(\mathcal{M}_s\) relating \(y\) to predictors \(X\).
  • The population distribution of \(X\) is known, from the Census etc, available as a table or post-stratification frame \(X^C_s\).
  • Generate predictions by projecting \(X^C_s\) through \(\mathcal{M}_s\): \(\hat{y}{}_{s} = \mathcal{M}_s(X^C_s)\).

Theory: when will this work?

  • Let \(R_i\) = 1 if \(i\) responds to a survey \(R_i\) = 0 otherwise; let \(Y\) be a variable of interest.
  • Non-ignorable non-response: \(p(R,Y) = p(R|Y) \, p(Y) \neq p(R) \, p(Y)\), i.e., \(R\) and \(Y\) are not independent.
  • cf ignorability conditional on \(X\): \(p(R, Y | X=x) = p(R | Y, X=x) p(Y|X=x) = p(R|X=x) p(Y|X=x)\).
  • i.e., in each stratum defined by \(X=x\), \(R\) and \(Y\) are conditionally independent: \((Y \perp R) \, | \, X\)
  • Or, knowledge of \(R\) supplies no knowledge about \(Y\) given the information about \(Y\) provided by \(X\).

Theory: when will this work?

  • Suppose \(X\) is sufficient to induce ignorability.
  • Then \(p(Y | X, R) \, = \, p(Y | X)\).
  • Can (validly) learn about \(p(Y | X)\) from observed data (\(R_i = 1\))
  • A model needed to relate \(Y\) and \(X\); guard against model dependence by employing non-parametric methods.
  • Tree-based Bayesian classifier (Chipman, George, and McCulloch 2010; Kapelner and Bleich 2016).

Covariates to help induce ignorability

  • \(X\) here is age (5 categories) by religion (5) by income (6) by education by gender (2).

  • Limited by availability on both VAA and Australian 2016 Census.

  • From ABS TableBuilder, we have a frame \(X^C\) of 182,555 non-empty cells, spanning all 150 CEDs.

20 largest cells and a random 20 in Census frame

Division age religion income education gender n Division age religion income education gender n
Sydney Age 30-44 None Fifth University Male 3,915 Higgins Age 65 plus Other Religion Third Some tertiary Male 20
Melbourne Age 30-44 None Fifth University Male 3,250 Moncrieff Age 45-64 Catholic First Some school Female 231
Grayndler Age 30-44 None Fifth University Female 3,039 Solomon Age 30-44 Other Religion Not Stated University Female 9
Sydney Age 30-44 None Fifth University Female 3,026 Solomon Age 45-64 Centrist Protestant Fifth Some school Female 211
Melbourne Age 30-44 None Fifth University Female 3,001 Calwell Age 65 plus None Not Stated High school Male 17
Grayndler Age 30-44 None Fifth University Male 2,738 Swan Age 65 plus Centrist Protestant First Some school Female 845
Fenner Age 30-44 None Fifth University Female 2,719 Paterson Age 65 plus Other Religion First Some school Male 5
Lyne Age 65 plus Centrist Protestant Second Some school Female 2,682 Corio Age 45-64 Centrist Protestant Second University Female 87
Hinkler Age 65 plus Centrist Protestant Second Some school Female 2,635 Goldstein Age 45-64 Catholic Second Some tertiary Male 88
North Sydney Age 30-44 None Fifth University Male 2,634 Mallee Age 30-44 Catholic Fourth University Male 82
North Sydney Age 30-44 None Fifth University Female 2,547 Lilley Age 18-20 Conservative Christian Fifth Some school Male 5
Durack Age 30-44 None Not Stated Some tertiary Male 2,546 Bonner Age 45-64 Other Religion Fifth University Male 62
Wentworth Age 30-44 None Fifth University Male 2,507 Kingston Age 18-20 Centrist Protestant Fifth High school Female 49
Fenner Age 30-44 None Fifth University Male 2,484 Jagajaga Age 30-44 Centrist Protestant Second Some school Female 13
Wentworth Age 30-44 None Fifth University Female 2,426 Fairfax Age 45-64 Conservative Christian Fourth University Female 173
Canberra Age 45-64 None Fifth University Male 2,364 Fairfax Age 65 plus Catholic Not Stated Some school Female 342
Cowper Age 65 plus Centrist Protestant Second Some school Female 2,353 Hindmarsh Age 65 plus Catholic First Some school Female 971
Warringah Age 30-44 None Fifth University Female 2,254 Petrie Age 45-64 Conservative Christian Not Stated Some tertiary Male 81
Bradfield Age 45-64 None Fifth University Male 2,222 Kennedy Age 21-29 None First Some school Female 125
Canberra Age 30-44 None Fifth University Female 2,220 Mitchell Age 30-44 Other Religion Not Stated Some school Male 10

Adding a political variable to the frame

  • To make ignorability more plausible, we add 2016 vote to the post-stratification frame:
    • in survey, model 2016 vote intention as a function of \(X\).
    • projecting Census frames through model to generate predicted vote shares in each cell of the frame.
    • adjust vote predictions to match known CED-level election results (with adjustment for non-voting share of the adult citizenry).
    • use raking to preserve Census cell counts.
    • in each CED, augmented frame matches the joint distribution of \(X\) supplied by the Census and the marginal distribution of 2016 vote.

Model-assisted post-stratification

  • Denote the appended frame as \(\tilde{X}^C\).
  • Form predictions in district \(s\) as \(\hat{y}_s = \mathcal{M}_s(\tilde{X}^C_s)\), again using a tree-based classifier.
  • Procedure is Bayesian, so also obtain posterior uncertainty estimates.

Validation with the Australian Marriage Law Postal Survey

  • On 15 November 2017, 61.6 per cent of Australian voters ‘voted’ to change the law to legalize same-sex marriage.
  • AMLPS ‘plebiscite’ administered by ABS between 12 September and 7 November 2017, by mail; participation rate of 79.5 per cent.
  • Compare CED-level results from AMLPS to CED estimates of voters’ attitudes towards same-sex marriage from 2016 VAA.
  • n.b., lengthy time interval between 2016 VAA and 2017 AMLPS and differences in question-wording, best case is an monotone relationship between VAA and AMLPS

Comparison with AMLPS

Validation: ten largest departures from linear regression

Division Postal Survey (Yes %) Estimate Difference Regression Error
Watson 30.4 51.6 -21.2 -23.5
Blaxland 26.1 44.8 -18.7 -20.5
McMahon 35.1 51.3 -16.2 -18.5
Fowler 36.3 50.4 -14.1 -16.4
Werriwa 36.3 49.1 -12.8 -15.0
Parramatta 38.4 51.0 -12.6 -14.9
Barton 43.6 55.3 -11.7 -14.3
Chifley 41.3 49.9 -8.6 -10.9
Gorton 53.3 61.1 -7.8 -10.8
Greenway 46.4 53.3 -6.9 -9.3

Errors by % CED speaking English at home

Errors by % of CED Islam

Errors by % of CED not responding to AMLPS

Errors by VC sample size per division

Discussion and conclusion

  • Our estimates reveal variation in preferences across legislative district, vote, demographics. Highlights cross-cutting nature of social issues.

  • This approach offers access to data that is roughly the equivalent of a classical survey with approximately 68 thousand respondents. This would likely have cost more than $1M to collect through conventional means.

  • Considerations for design of future VAA tools.