Modeling information effects in R with the informationeffects package

Why information matters in politics

In politics like elsewhere, what we know matters for what we want. Lucy wants harsh sentencing because she believes it will reduce crime rates. Were she to find out that it does not, she wouldn’t want harsh sentencing anymore. Bob wants less immigration because he believes that it hurts the economy. Were he to learn that immigration tends to have a positive, economic impact he would no longer want to see it reduced.

That’s why political scientists, rightly, are concerned with studying what voters know, and what difference it would make had they known more. The former question has been extensively investigated in the literature on public ignorance – as it turns out, most of us know very little when it comes to politically relevant matters (Achen and Bartels 2016; Delli Carpini and Keeter 1996). The latter (what difference knowledge makes in politics) has been studied in the literature under the heading of ‘information effects’ (Althaus 2003; Bartels 1996).

The information effects literature makes clear that knowledge does matter for politics, and can in some cases even change the electoral outcome. For example, Ahlstrom-Vij (2020) models an informed EU referendum in the UK, and sees the proportion of ‘Remain’ swing from a minority to a majority. Blais et al. (2009) simulate the outcome of six past Canadian elections, involving three to four parties, with fully informed voters, and see a likely difference in outcome in one. Oscarsson (2007) simulates six past Swedish elections, involving eight main parties, and sees a likely difference in outcome in two of them.

Information effects modeling can also be used to look at the influence of knowledge on political opinion over time. For example, Ahlstrom-Vij (2021) uses ANES data to evaluate the idea that we have entered a “post-truth era”, by we arguing that, if we have, we should expect to see decreasing information effects on central political issues over time. This turns out to be the case: Ahlstrom-Vij shows that, at least in a US context, we see a decrease in information effects on party preferences as well as on key, political issues – immigration, same-sex adoption, and gun laws, in particular – in the period of 2004 to 2016, which offers some novel, empirical evidence for the “post-truth” narrative.

Whether explicitly framed in those terms, modeling of information effects involves a form of counterfactual or causal modeling (Morgan and Winship 2015): a model is fitted, not for purposes of making a straightforward prediction (as in predictive modeling), e.g., concerning how some particular respondent might respond, but in order to estimate how a respondent would have responded, under some counterfactual condition. In our case, such an estimation is performed by fitting the model on the relevant data, and then using the model to make a “prediction,” once the value on the political knowledge variable for each respondent has been set to whatever value designates being “informed,” thereby estimating what each respondent would have responded, had they been fully informed, with reference to some relevant measure of political knowledge (Delli Carpini and Keeter 1996).

The informationeffects package

How does this work in practice? That’s the question this vignette is looking to answer. Using the informationeffects package, it will walk through each step in a complete pipeline from the construction of a political knowledge scale from a set of items to modeling the relevant effects. The package contains five functions, each of which will be demonstrated below:

• info_scale() constructs a knowledge scale using Item Response Theory (IRT) modeling by way of the mirt package on the basis of a set of binary knowledge items.
• info_emmeans() estimates marginal mean levels of knowledge using the emmeans package for different demographic variables in order to evaluate construct validity for the underlying knowledge scale.
• info_prop_scores() calculates propensity scores to be used as weights in subsequent, counterfactual modeling, in order to improve balance.
• info_bal_plots() evaluates propensity scores using balance plots via the cobalt package.
• info_effect() calculates information effects on the basis of survey data with a binary knowledge variable, propensity scores, and survey weights, while controlling for a set of covariates. It can also generate bootstrapped confidence intervals.

The package’s functionality will be illustrated by way of subset of Wave 17 of the British Election Study Internet Panel (Fieldhouse 2019) (N = 34,366). As our outcome, we will use the following attitudinal variable (immigSelf in the original data set; renamed immig_self in our subset): “On a scale from 0-10, how would you prefer immigration levels to Britain changed?” (0 = reduced a lot, 5 = kept the same, 10 = increased a lot). For purposes of modeling, this variable has been re-coded as a binary one, with 1 for responses below 5, and 0 otherwise:

df %>% 
  count(immig_self)
#> # A tibble: 2 × 2
#>   immig_self     n
#>        <dbl> <int>
#> 1          0 16596
#> 2          1 17770

In what follows, we will use these variables to estimate what difference information would make to anti-immigration sentiments in the UK. To that end, we’ll also use a set of demographic and socioeconomic covariates, as follows:

df %>% 
  select(education, income, gender, age, religion, ethnicity, party_id, eu_ref_vote)
#> # A tibble: 34,366 × 8
#>    education income gender age    religion    ethnicity     party_id eu_ref_vote
#>    <ord>     <ord>  <chr>  <chr>  <chr>       <chr>         <chr>    <chr>      
#>  1 no_qual   Q1     female over65 c_of_e      white_british libdem   remain     
#>  2 a-level   Q1     male   over65 no_religion white_british no_party remain     
#>  3 a-level   Q3     male   56-65  no_religion white_british plaid    leave      
#>  4 gcse      Q1     female over65 no_religion white_british cons     remain     
#>  5 a-level   Q2     male   over65 c_of_e      white_british no_party remain     
#>  6 postgrad  Q3     female 26-35  no_religion white_british labour   leave      
#>  7 gcse      Q4     female 56-65  c_of_e      white_british no_party remain     
#>  8 gcse      Q3     female 56-65  c_of_e      white_british cons     remain     
#>  9 no_qual   Q2     male   56-65  c_of_e      white_british plaid    leave      
#> 10 gcse      Q5     male   56-65  no_religion white_british cons     leave      
#> # … with 34,356 more rows

We will also make use of four knowledge items, coded as 1 for correct, and 0 for incorrect or “Don’t know” responses (Zaller 1992: 339; Althaus 2003: 105):

• k1: “Polling stations close at 10.00pm on election day” (True)
• k2: “No-one may stand for parliament unless they pay a deposit” (True)
• k3: “MPs from different parties are on parliamentary committees” (True)
• k4: “The number of MPs in Parliament is about 100” (False)

If we sum up the number of correct answers, we get the following distribution:

df %>% 
  mutate(total_score = k1 + k2 + k3 + k4) %>% 
  count(total_score)
#> # A tibble: 5 × 2
#>   total_score     n
#>         <dbl> <int>
#> 1           0  1994
#> 2           1  4611
#> 3           2  5225
#> 4           3  6869
#> 5           4 15667

Finally, we will use the survey weight variable (renamed survey_wt in the subset we will be using) included with the data set, in order to have our results be representative of the UK population.

info_scale(): Construct a knowledge scale

The first thing we need in order to model information effects is, naturally enough, some measure of participant’s level of political knowledge. Following the work of Michael Delli Carpini and Scott Keeter (Delli Carpini and Keeter 1993, 1996), this typically takes the form of a number of TRUE / FALSE items, where “Don’t know” responses, as already noted, are generally coded as FALSE, i.e., as respondents not knowing the relevant answer (Zaller 1992: 339; Althaus 2003: 105).

One straightforward way to create such a scale is to simply add up all correct answers, for a total knowledge score (Althaus 2003). One downside with doing so is that, outside of answering all questions correctly or answering all questions incorrectly, there is more than one way to get a particular number of responses correct. Since some questions are more difficult than others, and getting those right thereby is more diagnostic of being informed, a purely additive scale risks grouping together people of different abilities.

A better way to construct a scale is therefore to use Item Response Theory (IRT) modeling. IRT modeling is an established method for modeling underlying, latent traits, such as abilities. Such models are able to discriminate between the ability of respondents with the same number of correct responses but different response patterns. As we shall see, an IRT model also offers a clear window into the performance both of individual items and the scale as a whole, thereby helping the researcher construct a good knowledge scale.

The latent traits modeled by way of IRT are assumed to fall on a continuous scale. Values on that scale are usually referred to by way of the Greek letter \(\theta\) (theta), and taken to range from -\(\infty\) to +\(\infty\), with a mean of 0 and standard deviation of 1. This means that, while the individual \(\theta\) values ascribed to any particular respondent has no intrinsic meaning, it can nevertheless be interpreted relative to an estimated population mean.

To construct a knowledge scale using informationeffects, we first attach the package (assuming we have already installed it):

# install.packages("devtools")
# devtools::install_github("ahlstromvij/informationeffects")
library(informationeffects)

Then, we can build the scale using `know_scale``

know_scale <- info_scale(items = c("k1","k2","k3","k4"),
                         data = df)

info_scale returns a list of elements. To begin with, it uses R’s mirt package (Chalmers 2012) to generate an IRT scale on the basis of a set of knowledge items. The scale values of each response pattern in the data is accessible as know_scores:

head(know_scale$know_scores)
#> [1] -1.19012867 -0.01425591  0.75952465 -1.62074073 -0.09175294 -0.60738411

Additionally, info_scale takes as an optional parameter a percentile cut-off for a corresponding binary knowledge scale, accessible as binary_cutoff, and set at 90th percentile by default. The binary values of each response pattern in the data is accessed by way of know_scores_binary:

head(know_scale$know_scores_binary)
#> [1] 0 0 1 0 0 0

This binary variable will be used later on when calculating so-called propensity scores, for purposes of balancing the data set and break any correlation between the knowledge variable and demographic variables. For such balancing to work, we ideally want to set the bar for being “informed” at a level that’s demanding enough to be conceptually plausible, yet not so demanding that very few people quality. This can be evaluated by consulting the proportion of observations that end up in each of the two categories, available as a prop.table for each category via know_scores_binary_tbl:

know_scale$know_scores_binary_tbl
#> Proportion of observations in each category:
#>         0         1 
#> 0.5441134 0.4558866

As we can see, about 46% of the sample ends up in the “informed” category, which suggests two things: first, it likely won’t be too difficult to balance the data set with propensity scores, since ‘informed’ respondents are not in a small minority; and, second, the items in the scale are fairly easy. The latter fact in particular should be kept in mind when eventually interpreting any information effect.

info_scale also returns the IRT model itself as model, alongside a model_summary, providing the factor loadings (F1), and model coefficients (model_coef), with a corresponding to the discrimination parameter, and b to the difficulty parameter:

know_scale$model_summary
#> $rotF
#>           F1
#> X1 0.6249458
#> X2 0.8564270
#> X3 0.8021189
#> X4 0.8375793
#> 
#> $h2
#>           h2
#> X1 0.3905572
#> X2 0.7334672
#> X3 0.6433947
#> X4 0.7015391
#> 
#> $fcor
#>    F1
#> F1  1
know_scale$model_coef
#> $X1
#>         a      b g u
#> par 1.362 -1.909 0 1
#> 
#> $X2
#>         a      b g u
#> par 2.823 -0.442 0 1
#> 
#> $X3
#>         a      b g u
#> par 2.286 -0.492 0 1
#> 
#> $X4
#>         a      b g u
#> par 2.609 -0.566 0 1
#> 
#> $GroupPars
#>     MEAN_1 COV_11
#> par      0      1

We ideally want to see discrimination values greater than 1, which would indicate that the relevant item discriminates well between people of different levels of knowledge. This discrimination value is also reflected in the item probability function below (trace_plot), with steeper curves representing greater discrimination:

know_scale$trace_plot

The b value designates the difficulty of the item, and represents the point on the ability (i.e., \(\theta\)) scale on which a respondent becomes more than 50% likely to answer that question correctly. The same value can be plotted on the trace plot by drawing a straight line from 0.5 on the y-axis out to the line, and then tracing a vertical line down to \(\theta\) value on the x-axis, representing the relevant level of ability.

The test information plot (info_plot) shows at what point on the ability spectrum the test offers most information, which we in this case can see is just below a \(\theta\) of 0, representing mean ability:

know_scale$info_plot

An IRT scale needs to satisfy three conditions:

1. It needs (in the present context) to be unidimensional, i.e., the items involved should tap into a single trait. This can be evaluated using parallel analysis, accessed as par_analysis.
2. It should exhibit local independence, meaning that, conditional on the latent variable, item responses should be unrelated to one another. This is evaluated using Yen’s Q3 (Yen 1993), returned by info_scale as q3.
3. It should fit the data, which can be evaluated visually by inspecting the empirical_plots element.

Let’s look at each in turn:

know_scale$par_analysis

Parallel analysis is related to the traditional scree method, whereby we plot eigenvalues of a principal axis in descending order. These eigenvalues indicate the amount of variance accounted for by each of the factors, out of the total variance. In traditional scree plotting, we simply look at where we get a steep drop in the graph, suggesting that bringing in further factors fails to explain much (further) variance. However, in parallel analysis, we compare the scree plot to eigenvalues from principal axis factoring of random correlation matrices of the same size as the data, and look at how many factors have eigenvalues greater than the corresponding average eigenvalues of the random matrices (Andrews 2021). As can be seen in this graph, one factor has such an eigenvalue, suggesting that the unidimensionality assumption is likely satisfied.

Turn now to local independence:

know_scale$q3
#>             X1          X2         X3         X4
#> X1  1.00000000 -0.09945417 -0.1165377 -0.1642594
#> X2 -0.09945417  1.00000000 -0.4130960 -0.4549003
#> X3 -0.11653775 -0.41309604  1.0000000 -0.3345753
#> X4 -0.16425944 -0.45490029 -0.3345753  1.0000000

The largest Q3 value is -0.45. Yen (1993) suggests a cut-off value of 0.2, but as pointed out by Ayala (2009), a Q3 test tends to give inflated negative values for short tests. Indeed, Yen’s own suggestion was in the context of scales with at least 17 items. For that reason, a value of -0.45 would seem acceptable, given the short scale.

Finally, let’s consider model fit:

know_scale$empirical_plots

The empirical plots for all items suggest an acceptable fit, with some possible reservations about item 1.

info_emmeans(): Evaluate construct validity

By performing the type of diagnostics covered in the previous section on our knowledge scale, we can get a good sense of whether the model performs well from a formal perspective, i.e., in regards to unidimensionality, local independence, and model fit. However, we also would like to be able to validate that the score plausibly is measuring not just any trait, but specifically a form of political knowledge.

One way of doing this is to investigate the relationship between our knowledge scale and demographic factors that we know to be associated with political knowledge. Specifically, we expect that men should score more highly on our scale than women (vanHeerde-Hudson 2020; Plutzer 2020), and that the same should go for people who are older (Plutzer 2020), who have higher levels of education (Rasmussen 2016), and who earn more (Vowles 2020; Plutzer 2020).

One way to investigate this is to look at the estimated marginal means for each level of education, income, gender, and age, holding the others constant, which we can do using info_emmeans() (which, as the name suggests, employs emmeans under the hood):

df$knowledge <- know_scale$know_scores
info_emmeans(knowledge_var = "knowledge",
            covariates = c("income", "education","gender","age"), 
            data = df)
#> [[1]]
#>  income  emmean      SE    df lower.CL upper.CL
#>  Q1     -0.2221 0.00780 34350  -0.2374 -0.20685
#>  Q2     -0.1662 0.00886 34350  -0.1836 -0.14887
#>  Q3     -0.1362 0.00995 34350  -0.1558 -0.11674
#>  Q4     -0.1068 0.00911 34350  -0.1247 -0.08895
#>  Q5     -0.0118 0.01018 34350  -0.0317  0.00816
#> 
#> Results are averaged over the levels of: education, gender, age 
#> Confidence level used: 0.95 
#> 
#> [[2]]
#>  education   emmean      SE    df lower.CL upper.CL
#>  no_qual    -0.4023 0.01418 34350   -0.430  -0.3745
#>  below_gcse -0.3117 0.01731 34350   -0.346  -0.2778
#>  gcse       -0.2207 0.00831 34350   -0.237  -0.2044
#>  a-level    -0.0905 0.00810 34350   -0.106  -0.0746
#>  undergrad   0.0497 0.00646 34350    0.037   0.0623
#>  postgrad    0.2036 0.01155 34350    0.181   0.2263
#> 
#> Results are averaged over the levels of: income, gender, age 
#> Confidence level used: 0.95 
#> 
#> [[3]]
#>  gender  emmean      SE    df lower.CL upper.CL
#>  female -0.3515 0.00609 34350  -0.3634   -0.340
#>  male    0.0942 0.00663 34350   0.0812    0.107
#> 
#> Results are averaged over the levels of: income, education, age 
#> Confidence level used: 0.95 
#> 
#> [[4]]
#>  age    emmean      SE    df lower.CL upper.CL
#>  18-25  -0.417 0.01511 34350  -0.4462  -0.3870
#>  26-35  -0.562 0.01093 34350  -0.5833  -0.5404
#>  36-45  -0.383 0.00970 34350  -0.4020  -0.3640
#>  46-55  -0.049 0.00908 34350  -0.0668  -0.0312
#>  56-65   0.242 0.00866 34350   0.2249   0.2589
#>  over65  0.397 0.00781 34350   0.3814   0.4120
#> 
#> Results are averaged over the levels of: income, education, gender 
#> Confidence level used: 0.95

We can see that the scale value increases in an (almost exclusively) monotonic fashion for education, income, and age, which is what we should expect if our scale measures political knowledge. We also see that the mean level of knowledge is greater for men than for women. This all offers some evidence for construct validity.

info_prop_scores(): Calculate propensity scores

As noted earlier, information modeling is a type of counterfactual modeling, estimating the causal effect that we would have seen, had we been able to intervene on (i.e., increase) the knowledge variable. Best practice in counterfactual modeling is to rely on so-called doubly-robust estimation, which looks to approximate the situation we would have found ourselves in, had our data been the result of a randomized experimental design with a single treatment (Morgan and Winship 2015). The ‘double robustness’ owes to how effects are estimated in a context where we have both (a) controlled for assumed confounds (as in standard regression), and (2) taken steps to make up for the fact that the data have not come about as a result of randomized assignment. In the present case, this second layer of ‘robustness’ is achieved by using so-called ‘propensity scores’ as weights in the subsequent models.

This is where the binary knowledge variable from before comes in. In our case, propensity scores measure the probability (i.e. propensity) that an observation will be found in the ‘fully informed’, binary category, as a function of someone’s demographic features. (It is possible to do propensity scoring with a continuous ‘treatment’, e.g., as in Hirano and Imbens (2004); however, at present, informationeffects only implements binary propensity scoring.) The idea is to then use these scores to remove any correlation between these features and the ‘informed’ category, to justify a counterfactual inference.

To see why, return to the paradigm of a randomized experimental design, where the random allocation of participants to a treatment and a control group means that no feature of the participant is predictive of being found in the treatment as opposed to in the control. Whether female or male, rich or poor (and so on), you are equally likely to end up in one group as opposed to in the other. In the case of observational data, by contrast, this might not be the case. It might (for example) be that some features of the observations – such as, their level of education – are predictive of ending up in the ‘informed’ category.

In fact, let’s look at the data at hand, to determine whether the demographic factors that we have reason to believe influence someone’s degree of political knowledge – again, gender, level of education, income, and age (Plutzer 2020) – are in fact predictive of knowledge, as measured by our scale:

df$knowledge_binary <- know_scale$know_scores_binary
m <- glm(knowledge_binary ~ 
           age +
           education +
           gender +
           income,
         data = df,
         family = "binomial")
summary(m)
#> 
#> Call:
#> glm(formula = knowledge_binary ~ age + education + gender + income, 
#>     family = "binomial", data = df)
#> 
#> Deviance Residuals: 
#>     Min       1Q   Median       3Q      Max  
#> -2.3806  -0.9137  -0.4555   0.9573   2.5502  
#> 
#> Coefficients:
#>                      Estimate Std. Error z value Pr(>|z|)    
#> (Intercept)         -2.987678   0.074296 -40.213  < 2e-16 ***
#> age26-35            -0.473751   0.064920  -7.297 2.93e-13 ***
#> age36-45             0.004427   0.060290   0.073  0.94147    
#> age46-55             0.915762   0.058653  15.613  < 2e-16 ***
#> age56-65             1.638358   0.058433  28.038  < 2e-16 ***
#> ageover65            2.032476   0.058251  34.892  < 2e-16 ***
#> educationbelow_gcse  0.249039   0.073126   3.406  0.00066 ***
#> educationgcse        0.508128   0.052563   9.667  < 2e-16 ***
#> educationa-level     0.904158   0.054356  16.634  < 2e-16 ***
#> educationundergrad   1.314389   0.052504  25.034  < 2e-16 ***
#> educationpostgrad    1.812900   0.063408  28.591  < 2e-16 ***
#> gendermale           1.323824   0.025377  52.166  < 2e-16 ***
#> incomeQ2             0.183424   0.035888   5.111 3.20e-07 ***
#> incomeQ3             0.261691   0.039723   6.588 4.46e-11 ***
#> incomeQ4             0.366727   0.038345   9.564  < 2e-16 ***
#> incomeQ5             0.591538   0.041868  14.129  < 2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> (Dispersion parameter for binomial family taken to be 1)
#> 
#>     Null deviance: 47374  on 34365  degrees of freedom
#> Residual deviance: 38619  on 34350  degrees of freedom
#> AIC: 38651
#> 
#> Number of Fisher Scoring iterations: 4

Looking at the coefficient values for age, education, and income to begin with, we see that the difference in effect between the lowest age bracket, level of income, and income bracket increases as we move up the factor levels. These differences are in virtually all cases also significant. The same goes for the difference in knowledge between men and women, with men knowing more than women.

Let’s now calculate the propensity scores with info_prop_scores, and visualise them as a histogram to get a sense of their distribution:

df$prop_score <- info_prop_scores(knowledge_var = "knowledge_binary", 
                                  covariates = c("age","gender","education","income"), 
                                  data = df)

df %>% 
  ggplot() +
  aes(x = prop_score) +
  geom_histogram(binwidth=0.1, color = "black", fill = "salmon")

What we want to see in this distribution is a clustering of propensity scores towards the low end, and not too many extreme scores. That said, extreme scores should not automatically be assumed to be incorrect (Levy et al. 2008), although one should be mindful that they of course have a disproportionate influence when subsequently using them as weights in our regression model (more on this below). When properly estimated, however, such weighting will counteract any correlations between demographics and levels of political knowledge. Specifically, since propensity scores measure the probability of ending up in the ‘treatment’ category, given a set of covariates – in our case, the probability that you would be ‘informed’, given your age, income, level of education and gender – we can use the inverse of those scores as weights (such that an observation with a low propensity is weighted heavily) in fitting the model. Given an appropriately chosen set of covariates when calculating the scores, this recreates a situation that would have been expected in a randomized experiment, thereby allowing greater confidence in any counterfactual inference.

info_bal_plots(): Evaluate propensity scores

Since the whole point of propensity scores is to balance the sample, we want to inspect whether we have been successful on that score. We can do that using balance plots by way of info_bal_plots, in turn relying on the cobalt package (Greifer 2022):

bal_plots <- info_bal_plots(knowledge_var = "knowledge_binary", 
                            covariates = c("age", "gender", "income", "education"), 
                            prop_score ="prop_score", 
                            data = df)

We see that, in each case, balance has been improved by the propensity scores (the right pane in each of the graphs). Had that not been the case, we might have wanted to revisit the cut-off for being ‘informed’ (currently, 90th percentile or above), in case the balance between the two groups is so lopsided that it’s difficult to balance them using propensity scores.

info_effect(): Calculate information effects

We now need to specify our model – or models, in case we want to use several specifications. Since care needs to be taken when specifying and interpreting causal models, the first thing to do is to justify one’s choices regarding what variables are to be included and excluded. By transparently presenting the rationale for model specifications, and visualizing this as a directed acyclic graph (DAG), we adhere to good practice for political scientists who use observational data to address causal questions (Keele, Stevenson, and Elwert 2020).

As in traditional regression, we need to control for any confounders. In our case, those are variables that have a causal effect on both someone’s degree of political knowledge and their political attitudes or preferences. Existing evidence suggests gender (vanHeerde-Hudson 2020; Plutzer 2020), level of education (Rasmussen 2016), income (Vowles 2020; Plutzer 2020), and possibly age (Plutzer 2020) fall in this category.

Moreover, to reduce noise in these models, we also do well to control for variables that can be expected to change someone’s political preferences, whether also their degree of knowledge, such as ethnicity (e.g., through a “shared faith”; Dawson (1995)) and religion (Geoffrey Evans 2020).

What about partisanship, as measured by party identification? Despite being a prominent variable in political scientific modeling, we probably do well to exclude it in this context for two reasons:

First, it is likely affected by political knowledge, specifically, knowledge of parties’ and candidates’ positions (Brader and Tucker 2018; Fowler and Margolis 2014). This would make it a mediator in the language of counterfactual/causal modeling. Controlling for a mediator, or a causal node located on a direct or indirect pathway between (in this case) political knowledge and political preference, will mean misestimating the relevant causal effect (Rohrer 2018).

Second, even if partisanship is not a mediator, controlling for it in this context is likely unnecessary. Partisanship is heavily influenced by socialization early in life (Campbell et al. 1980), including around the group-identities of religion, ethnicity, gender, and the like – all of which shape individuals’ conceptions of what positions “people like us” take in politics (Green, Palmquist, and Schickler 2004). Consequently, controlling for such group-level variables (here: gender, ethnicity, and religion) would already account for partisanship.

In the UK context, EU referendum vote is likely to in this respect be similar to partisanship, as far as causal modeling is concerned, as research suggests that UK voters’ identification with “Leave” or “Remain” camps have become political identities in their own rights (Hobolt, Leeper, and Tilley 2021). If correct, then referendum vote choice, too, is an unnecessary control because it is a function of socialization variables for which our model already controls.

The DAG below uses the R package ggdag (Barrett 2022) to summarise the assumptions made for the purpose of modeling. Note the status of partisanship (“Party” in the graph) as a mediator for knowledge (“Know.”). If partisanship is an unnecessary control, the edge between knowledge and partisanship should be removed. Depending on the position one takes in relation to whether the EU referendum has become a political identity in its own right, EU referendum vote would either replace the partisanship node or (perhaps more plausibly) inhabit a structurally identical node (i.e., a mediator for knowledge) alongside it.

theme_set(ggdag::theme_dag())

coords <- tibble::tribble(
  ~name,            ~x,  ~y,
  "Y",              4,   0,
  "Know.",          0,   0,
  "Edu.",           0,   1,
  "Gend.",          1,   1,
  "Inc.",           2,   1,
  "Age",            3,   1,
  "Ethn.",          1,   -1.5,
  "Rel.",           2,   -1.5,
  "Party",          2,   -0.5
)

pt_dag <- ggdag::dagify(Y ~ Know. + Gend. + Edu. + Inc. + Ethn. + Rel. + Age + Party,
                  Party ~ Know. + Gend. + Ethn. + Rel.,
                  Know. ~ Gend. + Edu. + Inc. + Age,
                         exposure = "Know.",
                         outcome = "Y",
                    coords = coords)

ggdag::ggdag(pt_dag, stylized = TRUE, node_size = 20)

We now have everything we need to actually calculate information effects, i.e., differences in proportions between actual and estimated informed levels of support for some particular statement, policy, or the like. In light of the discussion above about different model specifications, and questions about the exact causal role of partisanship and any identities tied up with the UK’s 2016 EU referendum vote, we fit and display the results for three models, in the interest of robustness: one purely demographic, one that also controls for partisanship, and one that additionally controls for the respondent’s EU referendum vote.

inf1 <- info_effect(outcome = "immig_self", 
                    knowledge_var = "knowledge_binary", 
                    covariates = c("age",
                                   "gender",
                                   "education",
                                   "income",
                                   "religion",
                                   "ethnicity"), 
                    prop_weight = "prop_score",
                    survey_weight = "survey_wt",
                    data = df,
                    boot_ci = TRUE)

inf2 <- info_effect(outcome = "immig_self", 
                    knowledge_var = "knowledge_binary", 
                    covariates = c("age",
                                   "gender",
                                   "education",
                                   "income",
                                   "religion",
                                   "ethnicity",
                                   "party_id"), 
                    prop_weight = "prop_score",
                    survey_weight = "survey_wt",
                    data = df,
                    boot_ci = TRUE)

inf3 <- info_effect(outcome = "immig_self", 
                    knowledge_var = "knowledge_binary", 
                    covariates = c("age",
                                   "gender",
                                   "education",
                                   "income",
                                   "religion",
                                   "ethnicity",
                                   "party_id",
                                   "eu_ref_vote"), 
                    prop_weight = "prop_score",
                    survey_weight = "survey_wt",
                    data = df,
                    boot_ci = TRUE)

Each model estimates actual (actual_proportion) and informed support (informed_proportion) for the idea that levels of immigration coming into the UK should be reduced, in order to estimate what difference (difference) political knowledge makes on this issue. For both actual and informed support, we apply the survey weights included in the data set to approximate representativeness. Let’s plot the results:

plot_df <- tibble::tibble(
  scenario = c("Actual", "Informed (demographic)","Informed (partisanship)", "Informed (EU vote)"),
  support = c(inf1$actual_proportion, inf1$informed_proportion, inf2$informed_proportion, inf3$informed_proportion),
  lwr = c(inf1$actual_lwr, inf1$informed_lwr, inf2$informed_lwr, inf3$informed_lwr),
  upr = c(inf1$actual_upr, inf1$informed_upr, inf2$informed_upr, inf3$informed_upr)
)

plot_df <- within(plot_df, scenario <- factor(scenario, levels=c("Actual", 
                                                                 "Informed (EU vote)", 
                                                                 "Informed (partisanship)", 
                                                                 "Informed (demographic)")))

theme_set(ggplot2::theme_minimal())
ggplot2::ggplot(plot_df) +
  aes(x = scenario,
      y = support,
      fill = scenario) +
  geom_bar(stat = "identity", color = "black") +
  geom_errorbar(aes(ymin=lwr,
                    ymax=upr), width=.1,
                position=position_dodge(.9)) +
  geom_hline(yintercept=0.5, linetype="dashed") +
  scale_fill_brewer(palette="Blues") +
  ylab("Proportion of support") +
  xlab("Scenario") +
  ggtitle("Proportion wanting to see immigration levels reduced by scenario") +
  theme(legend.position = "none") +
  annotate("text", x = 1, y = plot_df$support[1]+0.025, label = paste(round(plot_df$support[1] * 100, 2),"%", sep="")) +
  annotate("text", x = 2, y = plot_df$support[4]+0.025, label = paste(round(plot_df$support[4] * 100, 2),"%", sep="")) +
  annotate("text", x = 3, y = plot_df$support[3]+0.025, label = paste(round(plot_df$support[3] * 100, 2),"%", sep="")) +
  annotate("text", x = 4, y = plot_df$support[2]+0.025, label = paste(round(plot_df$support[2] * 100, 2),"%", sep=""))

The function returns the formula, the model, as well as the actual and estimated informed proportions (in each case weighted using the survey weights), as well as the difference between the two, i.e., the information effect. We see that, if people in the UK were all to become informed, the idea that immigration levels should be reduced would likely go from a majority to a minority position (with the dashed line signifying 50%), irrespective of the particular modeling assumptions made, suggesting some robustness of the results. The size of the effect – close to 10 percentage points for the demographic model – is noteworthy, especially in the context of the low bar set by the knowledge scale, suggesting that even a low level of political knowledge makes a difference.

There is no established way to compute confidence intervals for the type of aggregate, counterfactual estimates reported here. For purposes of giving a sense of the variability of the individual estimates, bootstrapped confidence intervals are therefore returned, constructed as follows: 1,000 bootstrap samples are drawn from the (weighted) predictions, as well as from the weighted responses in the data set. Using R’s boot package (Davison and Hinkley 1997), basic 95% confidence intervals are then computed for the mean weighted prediction, the upper and lower bounds of which are then divided in each case by the mean weight in the total sample.

Conclusion

We started out by noting that information matters for politics, and that one of the main ways of determining the difference that knowledge will make in any given instance is by modeling information effects. The informationeffects package offers a complete pipeline for calculating such effects. If you spot any bugs or issues, please log them on the GitHub repo here, so that I can address them and hopefully make the package as useful as possible to researchers working on this topic.