• Let \(t\) index campaign days.
• Poll \(p\) fielded on day \(t\) by polling company \(j\) yields a estimated voting intention, a proportion \(\color{cyan}{y_p} \in [0,1]\), with sample size \(\color{cyan}{n_p}\).
  
• Variance of \(\color{cyan}{y_p}\) is approximately \(\color{cyan}{V_p = y_p (1-y_p)/n_p}\)
• True, latent voting intentions on day \(t\) are \(\color{orange}{\xi_t} \in [0,1]\). These are observed exactly on election days, \(\color{orange}{\xi_1}\) and \(\color{orange}{\xi_T}\), respectively.
• Polling company \(j\) has a time-invariant “house effect” \(\color{orange}{\delta_j}\), such that \(E(\color{cyan}{y_p}) = \color{orange}{\xi_{t(p)}} + \color{orange}{\delta_{j(p)}}\)
• \(E(\color{cyan}{y_p}) = \color{orange}{\xi_{t(p)}} + \color{orange}{\delta_{j(p)}}\) = truth + bias, both unobserved.

Arise from: - sampling methodology (e.g., RDD, landline/mobile mix; quotas from web panels) - weighting procedures and selection of weighting variables (raking; propensity score matching) - survey mode (live interviewer, IVR, web self-complete) - question wording and question ordering effects - response options (are minor parties or DK offered or volunteered?; are DKs pushed?) - field operations (time of day, day of week) - reporting conventions (DKs reported or not) - compounded in low or uncertain voter turnout environments

• Measurement model: \(\color{cyan}{y_{p}} \sim N(\color{orange}{\xi_{t(p)}} + \color{orange}{\delta_{j(p)}} \, , \, \color{cyan}{V_p})\)

• Dynamic model: \(\color{orange}{\xi_t} \sim N(\color{orange}{\xi_{t-1}}, \color{orange}{\omega^2})\), with the endpoint constraints from election results observed on \(\color{orange}{\xi_1}\) and \(\color{orange}{\xi_T}\).

• Given published polls, \(\color{cyan}{\boldsymbol{Y}}\), sample sizes, field dates and identity of polling companies — and the model — we seek

1. trajectory of latent voting intentions \(\color{orange}{\boldsymbol{\xi}} = (\color{orange}{\xi_1}, \ldots, \color{orange}{\xi_T})'\)
2. house effects: \(\color{orange}{\boldsymbol{\delta}} = (\color{orange}{\delta_1}, \ldots, \color{orange}{\delta_J})'\)
3. “pace of change” parameter (innovation variance), \(\color{orange}{\omega^2}\).

• Augment model with unknown step/discontinuities \(\color{orange}{\gamma}\) in \(\color{orange}{\boldsymbol{\xi}}\) trajectory, no “event” days (e.g., leadership changes).

• as written, the model is over-parameterised
• \(E(\color{cyan}{y_p}) = \color{orange}{\xi_{t(p)}} + \color{orange}{\delta_{j(p)}}\).
• Indistinguishable from \(E(\color{cyan}{y_p}) = [\color{orange}{\xi_{t(p)}} + \color{red}{c}] + [\color{orange}{\delta_{j(p)}} - \color{red}{c}], \quad \forall\ \color{red}{c} \neq 0\).
• Post-election, end-point constraints: anchor \(\xi_T\) to known election result, and/or \(\xi_1\) to past election result as may be appropriate.
• Sum-to-zero normalisation of house effects \(\color{orange}{\delta}\); i.e., set \(\color{red}{c} = \color{orange}{\bar{\delta}}\), such that \(\xi_t\) are identified up to a translation equal to the average bias of all pollsters.

• Gaussian law of motion: Kalman filter.
• in Bayesian statistics: dynamic linear model (West & Harrison).
• house effects and partially observed polling data makes the model slightly non-standard for off-the-shelf Kalman filtering (many packages in R)
• EM or MCMC via R and C/C++
• jags via rjags (Plummer 2019), see Jackman (2009).
• Stan via RStan (Stan Development Team 2020)
• nimble (de Valpine et al. 2017)
• pomp (King et al. 2016)

• suppose true level of voting intentions is \(\color{orange}{\pi} \in (0,1)\)
• special case of the Central Limit Theorem: under unbiased, simple random sampling (SRS), survey based estimates of \(\color{orange}{\pi}\) will be distributed normally around \(\color{orange}{\pi}\) with variance \(V(\color{orange}{\pi}) = \color{orange}{\pi}(1-\color{orange}{\pi})/\color{cyan}{n}\).
• put the question of bias to one side (dealt with previously with house effects estimates)
• we compare observed dispersion of the polls with theoretically expected dispersion, given (1) stated sample sizes \(\color{cyan}{n}\); (2) assumption about \(\color{orange}{\pi}\)

• latent \(\xi_t\) is a \(K\) vector, subject to restriction that \(\sum_{k=1}^K \xi_{tk} = 1 \forall t\).

• 2 approaches:

  • Dirichlet model for transitions and multinomial for polls.
  • \(K-1\) Gaussian model for log-odds of \(\xi_t\) and poll estimates (with \(K-1\) by \(K-1\) covariance matrices)

• examples: multi-candidate elections (e.g., Iowa caucuses), with drop-out and drop-in.

• multiple jurisdictions: e.g., each state in US presidential elections, \(M \approx 50\) filters running in parallel, with high dependencies in trajectories (voters in different states consuming same information, the “nationalisation” of politics and campaigns)

• not just a “poll average”, but a forecast
• with great public interest, comes great power, and …
• participant, not observer
• communication of results to lay public of vital importance