Marginal effects lab
Katie Gouge
2025-03-20
Hi all! This week we will be working through the marginaleffects package. While we have seen this package before, a refresher never hurt anybody.
Now we have had chats before about why one might use the marginaleffects package over the emmeans package. Essentially, both of these packages allow us to visualize the important bits that an lm reports back to us. Or as the great Thomas J. Wood says: “You estimated an lm, but you probably want to tidy and plot an emmeans” (2024), meaning that we want to run an lm and then visualize the effect that x had on y.
However, as we have already discussed, the emmeans package is currently maintained by Russel Lenth, an emeritus professor of statistics at Iowa. Unfortunately, Russel won’t live forever (at least on this Earth), and this poses some real challenges for the health of the emmeans package. Russel acknowledges his inability to maintain the package forever and encourages others to supersede his package with another when the time must come. You can read more about this from Russel here: https://github.com/rvlenth/emmeans?tab=readme-ov-file#supersession-plan.
Now that I have done sufficient (?) throat clearing, let’s hop into the lab. Importantly, like many of you, I am still learning how to use the marginaleffects package, so I am hopeful we can learn from each other throughout lab and breathe through the frustration.
Let’s go begin by installing the most recent version of the package.
remotes::install_github("vincentarelbundock/marginaleffects")## Skipping install of 'marginaleffects' from a github remote, the SHA1 (8e35cf2d) has not changed since last install.
## Use `force = TRUE` to force installationpackageVersion("marginaleffects")## [1] '0.25.0.9'We have talked about Vincent Arel-Bundock (pronounced “Arel” as in “barrel”; “Bundock” as in “bun” + “duck”) before, but just to reiterate, he is a very impressive Michigan PhD who also created the modelsummary package! He is pictured below with his dog, Chip. We can also see that he has exquisite taste in both art and furniture.
Given how useful his other package is for creating regression tables, we might expect that it is worth it to try to understand his other package marginaleffects.
Just to give us a brief overview (and review from previous labs), the marginaleffects package allows us to estimate the following:
Fitted values
Contrasts/risk difference/difference in odds ratios
Slopes
Linear and non-linear hypothesis tests
Equivalence tests
This package is compatible with a ton of models that social scientists use, such as lm(), glm(), lmer(), mlogit(), etc.
The website for this package is very fun, it looks like it could be a children’s storybook. Check it out here: https://marginaleffects.com/.
The really cool thing about this package is that it allows us to engage in counterfactual thinking, that is we can “combine predictions into counterfactual comparisons that quantify the strength of association between two variables, or the effect of a cause.” In order to address counterfactual questions, we have to make counterfactual predictions, and then we are able to compare those predictions.
Not to beat a dead horse, but here’s an example of what I mean. Let’s imagine we want to estimate the effect of a vaccine mandate on number of COVID-19 deaths. We would first start by asking: what is the number of COVID-19 deaths in states without a vaccine mandate? Next, we would ask: what would be the number of COVID-19 deaths in the same state if a vaccine mandate was implemented. Lastly, we would compare these two numbers to measure the difference between the expected number of COVID-19 deaths in counterfactual worlds with and without vaccine mandates.And even if we don’t think there is an air tight causal relationship, counterfactual comparisons can serve as descriptive measures of how strongly one variable is associated to another.
#load our packages
library(tidyverse)## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 3.5.1 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
## ✔ purrr 1.0.4
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(magrittr)##
## Attaching package: 'magrittr'
##
## The following object is masked from 'package:purrr':
##
## set_names
##
## The following object is masked from 'package:tidyr':
##
## extractlibrary(marginaleffects)
library(broom)We can start off easy by using a dataset that we are all familiar with… please welcome to the stage miss mtcars!!!
Let’s construct a model with multiplicative interactions. Here we have gross horsepower (hp) and displacement (disp) predict’s a cars mpg.
mod <- lm(mpg ~ hp * disp, data = mtcars)
mod %>% summary##
## Call:
## lm(formula = mpg ~ hp * disp, data = mtcars)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.5153 -1.6315 -0.6346 0.9038 5.7030
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.967e+01 2.914e+00 13.614 7.18e-14 ***
## hp -9.789e-02 2.474e-02 -3.956 0.000473 ***
## disp -7.337e-02 1.439e-02 -5.100 2.11e-05 ***
## hp:disp 2.900e-04 8.694e-05 3.336 0.002407 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.692 on 28 degrees of freedom
## Multiple R-squared: 0.8198, Adjusted R-squared: 0.8005
## F-statistic: 42.48 on 3 and 28 DF, p-value: 1.499e-10Now we want to use the comparisons() function to compute the difference in predicted outcome when each of the predictors is incremented by 1 unit (one predictor at a time, holding all others constant), Comparisons are unit-level values. Since we have 2 predictors and our data has 32 rows, we have 64 comparisons.
cmp <- comparisons(mod)
nrow(cmp)## [1] 64cmp##
## Term Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %
## disp -0.04147 0.00719 -5.764 <0.001 26.9 -0.0556 -0.02737
## disp -0.04147 0.00719 -5.764 <0.001 26.9 -0.0556 -0.02737
## disp -0.04640 0.00799 -5.809 <0.001 27.2 -0.0621 -0.03074
## disp -0.04147 0.00719 -5.764 <0.001 26.9 -0.0556 -0.02737
## disp -0.02261 0.00678 -3.334 <0.001 10.2 -0.0359 -0.00932
## --- 54 rows omitted. See ?print.marginaleffects ---
## hp -0.07031 0.01785 -3.939 <0.001 13.6 -0.1053 -0.03533
## hp 0.00391 0.01439 0.272 0.786 0.3 -0.0243 0.03212
## hp -0.05584 0.01480 -3.772 <0.001 12.6 -0.0848 -0.02682
## hp -0.01059 0.01229 -0.862 0.389 1.4 -0.0347 0.01350
## hp -0.06280 0.01619 -3.878 <0.001 13.2 -0.0945 -0.03106
## Type: response
## Comparison: +1What if we are interested in what happens to the predicted outcome when the horsepower variable increases from 100 to 120?
We can do the following:
comparisons(
mod,
variables =
list(
hp = c(120,100)
)
)##
## Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %
## -1.0297 0.280 -3.672 <0.001 12.0 -1.579 -0.48012
## -1.0297 0.280 -3.672 <0.001 12.0 -1.579 -0.48012
## -1.3314 0.340 -3.914 <0.001 13.4 -1.998 -0.66471
## -0.4612 0.231 -1.999 0.0456 4.5 -0.913 -0.00904
## 0.1305 0.297 0.439 0.6609 0.6 -0.452 0.71343
## --- 22 rows omitted. See ?print.marginaleffects ---
## -1.4062 0.357 -3.939 <0.001 13.6 -2.106 -0.70658
## 0.0783 0.288 0.272 0.7857 0.3 -0.486 0.64234
## -1.1167 0.296 -3.772 <0.001 12.6 -1.697 -0.53649
## -0.2118 0.246 -0.862 0.3889 1.4 -0.694 0.26996
## -1.2560 0.324 -3.878 <0.001 13.2 -1.891 -0.62118
## Term: hp
## Type: response
## Comparison: 120 - 100What if horsepower increases by 1 standard deviation about its mean?
comparisons(
mod,
variables =
list(
hp = "sd"
)
)##
## Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %
## -3.530 0.961 -3.672 <0.001 12.0 -5.41 -1.646
## -3.530 0.961 -3.672 <0.001 12.0 -5.41 -1.646
## -4.564 1.166 -3.914 <0.001 13.4 -6.85 -2.279
## -1.581 0.791 -1.999 0.0456 4.5 -3.13 -0.031
## 0.447 1.020 0.439 0.6609 0.6 -1.55 2.446
## --- 22 rows omitted. See ?print.marginaleffects ---
## -4.821 1.224 -3.939 <0.001 13.6 -7.22 -2.422
## 0.268 0.987 0.272 0.7857 0.3 -1.67 2.202
## -3.828 1.015 -3.772 <0.001 12.6 -5.82 -1.839
## -0.726 0.843 -0.862 0.3889 1.4 -2.38 0.925
## -4.306 1.110 -3.878 <0.001 13.2 -6.48 -2.129
## Term: hp
## Type: response
## Comparison: (x + sd/2) - (x - sd/2)Pretty neat! While it’s not at all surprising that an increase in a car’s horsepower is negatively associated with miles per gallon, it’s good to know what we expect and see that pan out in the r console.
Let’s have a go using some other, perhaps more relevant data. Let’s look at the framing effect of support for welfare vs. aid to the poor, aka the most famous framing effect in the history of American public opinion research! Looking at you Martin Gilens!
##Tom has helpfully uploaded this to his github so we have easy access to this data
t1 <- "https://github.com/thomasjwood/code_lab/raw/main/data/gss_welfare.rds" %>% url %>% readRDS
#let's peak at the data and see what's up (##uncomment "t1" to view on your machine)
#t1Okay great! Looks like we have loaded the appropriate dataset above. On to the exercise…
What if we want to estimate the overall framing effect of “welfare” vs. “aid to the poor”?
The comparisons() function allows us to predict the outcome variable at different regressor values and compare those predictions by computing a difference, ratio or some other function. For more information, see https://marginaleffects.com/man/r/comparisons.html.
##we are interested in whether one's support (ans_num) is a function of the item's wording (version)
##In other words, does changing the wording of a question affect how people respond?
#let's make the linear model, outcome variable is ans_num and the predictor is version
lm1 <- lm(ans_num ~ version,
data = t1)
#does version have an impact on support?
lm1 %>% summary##
## Call:
## lm(formula = ans_num ~ version, data = t1)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.2020 -0.4342 -0.2020 0.7980 1.5658
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.434227 0.004573 313.6 <2e-16 ***
## versionwelfare 0.767815 0.006538 117.4 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.7282 on 49650 degrees of freedom
## Multiple R-squared: 0.2174, Adjusted R-squared: 0.2174
## F-statistic: 1.379e+04 on 1 and 49650 DF, p-value: < 2.2e-16#the following function estimates how support (ans_num) changes for each version of the question
lm1 %>%
comparisons(
variables = "version",
newdata = datagrid(
version = "unique"
)
)##
## version Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %
## unique 0.768 0.00654 117 <0.001 Inf 0.755 0.781
##
## Term: version
## Type: response
## Comparison: welfare - assistance to the poor##just a note:
# datagrid(version = "unique") tells the model to compare all unique versions of the question.
#in our case, welfare vs. assistance to the poorLet’s just take a second to review the output together.
The output tells us the estimated effect of question wording (version) on support (ans_num). Specifically, the comparison being made is between two different versions of the question wording. In this case, welfare is the baseline and the estimate tells us how much support differs when the question is worded as “assistance to the poor” instead of “welfare”. We see that the estimate is 0.768, which tells us that on average, support (ans_num) is 0.768 units higher when the question is framed as “assistance to the poor” compared to “welfare”.
This is good news because that totally fits with the long-standing finding that Americans support spending when it is framed as “assistance to the poor” and do not support spending for “welfare”.
The output also shows us other quantities that we are typically interested in, such as p-value, standard error, z-score, 95% confidence interval (we are 95% confident that the true effect lies between 0.755 and 0.781, pretty tight CI). This suggests that there is a strong and consistent framing effect.
Any questions?
Previous work on this suggests that framing effect may vary across one’s ideology or their race. Imagine we want to estimate the framing effect by respondent’s ideology or race.
#interaction of pol ideology and version
lm2 <- lm(ans_num ~ version * polviews ,
data = t1)
##we can see how version and ideology separately influence support
##AND
##the interaction of polviews with version allows us to assess whether the effect of version changes depending on ideology. I.e., does ideology modify the framing effect?
lm2 %>% summary##
## Call:
## lm(formula = ans_num ~ version * polviews, data = t1)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.4877 -0.3783 -0.2061 0.6217 1.8285
##
## Coefficients:
## Estimate Std. Error t value
## (Intercept) 1.17149 0.02405 48.707
## versionwelfare 0.53418 0.03401 15.704
## polviewsliberal 0.09034 0.02723 3.318
## polviewsslightly liberal 0.16718 0.02730 6.125
## polviewsmoderate, middle of the road 0.20677 0.02515 8.221
## polviewsslightly conservative 0.36786 0.02670 13.776
## polviewsconservative 0.51724 0.02668 19.387
## polviewsextremely conservative 0.57069 0.03370 16.934
## polviewsdon't know 0.19053 0.03416 5.577
## versionwelfare:polviewsliberal 0.09140 0.03865 2.365
## versionwelfare:polviewsslightly liberal 0.19653 0.03870 5.078
## versionwelfare:polviewsmoderate, middle of the road 0.29360 0.03560 8.248
## versionwelfare:polviewsslightly conservative 0.27072 0.03783 7.155
## versionwelfare:polviewsconservative 0.25221 0.03777 6.677
## versionwelfare:polviewsextremely conservative 0.21132 0.04795 4.407
## versionwelfare:polviewsdon't know 0.20436 0.04953 4.126
## Pr(>|t|)
## (Intercept) < 2e-16 ***
## versionwelfare < 2e-16 ***
## polviewsliberal 0.000909 ***
## polviewsslightly liberal 9.16e-10 ***
## polviewsmoderate, middle of the road < 2e-16 ***
## polviewsslightly conservative < 2e-16 ***
## polviewsconservative < 2e-16 ***
## polviewsextremely conservative < 2e-16 ***
## polviewsdon't know 2.46e-08 ***
## versionwelfare:polviewsliberal 0.018025 *
## versionwelfare:polviewsslightly liberal 3.82e-07 ***
## versionwelfare:polviewsmoderate, middle of the road < 2e-16 ***
## versionwelfare:polviewsslightly conservative 8.45e-13 ***
## versionwelfare:polviewsconservative 2.47e-11 ***
## versionwelfare:polviewsextremely conservative 1.05e-05 ***
## versionwelfare:polviewsdon't know 3.69e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.7066 on 49636 degrees of freedom
## Multiple R-squared: 0.2635, Adjusted R-squared: 0.2632
## F-statistic: 1184 on 15 and 49636 DF, p-value: < 2.2e-16#what are the levels for political ideology
t1$polviews %>% levels %>% dput## c("extremely liberal", "liberal", "slightly liberal", "moderate, middle of the road",
## "slightly conservative", "conservative", "extremely conservative",
## "don't know")#estimates how the effect of wording differs across ideologies
lm2 %>%
comparisons(variables = "version",
newdata = datagrid(
polviews = c(
"extremely liberal",
"liberal",
"slightly liberal",
"moderate, middle of the road",
"slightly conservative",
"conservative",
"extremely conservative")
)
) ##
## polviews Estimate Std. Error z Pr(>|z|) S 2.5 %
## extremely liberal 0.534 0.0340 15.7 <0.001 182.2 0.468
## liberal 0.626 0.0183 34.1 <0.001 844.5 0.590
## slightly liberal 0.731 0.0185 39.6 <0.001 Inf 0.695
## moderate, middle of the road 0.828 0.0105 78.9 <0.001 Inf 0.807
## slightly conservative 0.805 0.0166 48.6 <0.001 Inf 0.772
## conservative 0.786 0.0164 47.9 <0.001 Inf 0.754
## extremely conservative 0.746 0.0338 22.1 <0.001 355.7 0.679
## 97.5 %
## 0.601
## 0.662
## 0.767
## 0.848
## 0.837
## 0.819
## 0.812
##
## Term: version
## Type: response
## Comparison: welfare - assistance to the poor##we can use the above code and graph!
lm2 %>%
comparisons(variables = "version",
newdata = datagrid(
polviews = c(
"extremely liberal",
"liberal",
"slightly liberal",
"moderate, middle of the road",
"slightly conservative",
"conservative",
"extremely conservative")
)
) %>%
tidy() %>%
ggplot(
aes(x = polviews,
y = estimate,
ymin = conf.low,
ymax = conf.high)) +
geom_point() +
geom_errorbar(width = 0.2) +
labs(
title = "Framing effect by political ideology",
x = "Political ideology",
y = "Effect of wording on support"
) +
theme_minimal()
##we could also plot it using
plot_comparisons(
lm2,
variables = "version",
by = "polviews"
) +
labs(
x = "Political ideology",
y = "Effect of question wording on support"
)
Remember that the estimate column reports how much higher support is when the question is framed as “assistance to the poor” instead of “welfare”. Hopefully, we can see that “assistance to the poor” consistently receives more support than “welfare” across all groups. However, we see that the strongest framing effect is for moderates (0.828), and the weakest framing effect if for extremely liberal respondents (0.534). Ultimately, we find that moderates and conservatives react more strongly to the question wording than liberals.
These findings are reassuring, as we might expect moderates to be more impacted by framing effects and extremely liberal folks to not be as impacted by question wording, especially on this question.
Moving on to race… Some argue that the word welfare has been racialized to communicate “black American”. That being said, we can test whether race moder
#interaction for race and version
lm3 <- lm(ans_num ~ version * race,
data = t1)
##again, we can measure whether wording impacts support overall and whether racial groups have different baseline support levels
##the interaction term lets us see if race moderates the framing effect
lm3 %>% summary##
## Call:
## lm(formula = ans_num ~ version * race, data = t1)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.2741 -0.4898 -0.2741 0.7259 1.8390
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.489808 0.005096 292.354 < 2e-16 ***
## versionwelfare 0.784287 0.007273 107.841 < 2e-16 ***
## raceblack -0.328839 0.012813 -25.665 < 2e-16 ***
## raceother -0.100267 0.017319 -5.790 7.10e-09 ***
## versionwelfare:raceblack -0.069301 0.018494 -3.747 0.000179 ***
## versionwelfare:raceother -0.117235 0.024677 -4.751 2.03e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.7165 on 49646 degrees of freedom
## Multiple R-squared: 0.2424, Adjusted R-squared: 0.2423
## F-statistic: 3177 on 5 and 49646 DF, p-value: < 2.2e-16#whites are the baseline group!
#the intercept communicates the average level of support for "assistance to the poor" among white respondents
#whites on average rate support at 1.49 when the question wording is "assistance to the poor"
t1$race %>% levels %>% dput## c("white", "black", "other")lm3 %>%
comparisons(variables = "version",
newdata = datagrid(
race = c("white", "black", "other")
)
) ##
## race Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %
## white 0.784 0.00727 107.8 <0.001 Inf 0.770 0.799
## black 0.715 0.01700 42.0 <0.001 Inf 0.682 0.748
## other 0.667 0.02358 28.3 <0.001 582.4 0.621 0.713
##
## Term: version
## Type: response
## Comparison: welfare - assistance to the poor##let's add on to the above and graph
lm3 %>%
comparisons(variables = "version",
newdata = datagrid(race = unique(t1$race))) %>%
tidy() %>%
ggplot(aes
(x = race,
y = estimate,
ymin = conf.low,
ymax = conf.high)) +
geom_point() +
geom_errorbar(width = 0.2) +
labs(
title = "Framing effect by race",
x = "Race",
y = "Effect of wording on support"
) +
theme_minimal()
## we could also plot it
plot_comparisons(
lm3,
variables = "version",
by = "race"
) +
labs(
x = "Race",
y = "Version"
)
We can see that white respondents are highly sensitive to question wording, so there is a stronger framing effect for whites than the other racial groups. However, all racial groups show a significant framing effect. Super interesting that the effect is still strong for non-white respondents.
Bonus question!
What if we want to see how it changes over time?
#race interaction changing over time
t1 %>% glimpse## Rows: 49,652
## Columns: 7
## $ year <fct> "1984", "1984", "1984", "1984", "1984", "1984", "1984", "1984…
## $ id <dbl> 1, 2, 3, 4, 7, 8, 9, 11, 12, 14, 16, 17, 19, 21, 22, 24, 26, …
## $ polviews <fct> "conservative", "slightly liberal", "liberal", "moderate, mid…
## $ race <fct> white, white, black, white, white, white, other, black, white…
## $ version <chr> "welfare", "welfare", "welfare", "assistance to the poor", "w…
## $ ans <fct> too much, too little, too little, too little, about right, to…
## $ ans_num <dbl> 3, 1, 1, 1, 2, 1, 1, 1, 3, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 1…t1$year %>% levels %>% dput## c("1972", "1973", "1974", "1975", "1976", "1977", "1978", "1980",
## "1982", "1983", "1984", "1985", "1986", "1987", "1988", "1989",
## "1990", "1991", "1993", "1994", "1996", "1998", "2000", "2002",
## "2004", "2006", "2008", "2010", "2012", "2014", "2016", "2018",
## "2021", "2022", "don't know", "iap", "I don't have a job", "dk, na, iap",
## "no answer", "not imputable", "refused", "skipped on web", "uncodeable",
## "not available in this release", "not available in this year",
## "see codebook")#make a new model
lm5 <- lm(ans_num ~ version * race * year,
data = t1)
##three-way interaction (WOAH) which tells us whether the framing effect changes over time, and whether those changes differ by race.
lm5 %>% summary##
## Call:
## lm(formula = ans_num ~ version * race * year, data = t1)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.5662 -0.4593 -0.2000 0.5900 1.9048
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.521197 0.035418 42.950 < 2e-16 ***
## versionwelfare 0.704934 0.050183 14.047 < 2e-16 ***
## raceblack -0.375742 0.101982 -3.684 0.000229 ***
## raceother -0.164054 0.192833 -0.851 0.394908
## year1985 -0.039490 0.044958 -0.878 0.379748
## year1986 -0.017982 0.045422 -0.396 0.692195
## year1987 -0.061909 0.043525 -1.422 0.154917
## year1988 -0.106681 0.045450 -2.347 0.018920 *
## year1989 -0.076753 0.045309 -1.694 0.090272 .
## year1990 -0.079756 0.046484 -1.716 0.086212 .
## year1991 -0.058387 0.045671 -1.278 0.201107
## year1993 0.006928 0.045171 0.153 0.878104
## year1994 0.104987 0.041081 2.556 0.010604 *
## year1996 0.192830 0.041348 4.664 3.12e-06 ***
## year1998 0.018046 0.041460 0.435 0.663373
## year2000 0.002114 0.041630 0.051 0.959498
## year2002 -0.035742 0.046573 -0.767 0.442814
## year2004 -0.073980 0.047116 -1.570 0.116382
## year2006 -0.089549 0.041538 -2.156 0.031102 *
## year2008 -0.071393 0.043706 -1.633 0.102379
## year2010 -0.061171 0.043746 -1.398 0.162021
## year2012 0.023472 0.044489 0.528 0.597789
## year2014 0.035344 0.042569 0.830 0.406384
## year2016 -0.111168 0.041821 -2.658 0.007860 **
## year2018 -0.130984 0.043423 -3.016 0.002559 **
## year2021 -0.077184 0.039749 -1.942 0.052170 .
## year2022 -0.128340 0.040776 -3.147 0.001648 **
## versionwelfare:raceblack -0.177974 0.142608 -1.248 0.212037
## versionwelfare:raceother 0.295066 0.272724 1.082 0.279293
## versionwelfare:year1985 0.147217 0.063853 2.306 0.021140 *
## versionwelfare:year1986 0.040998 0.064695 0.634 0.526273
## versionwelfare:year1987 0.131293 0.066974 1.960 0.049959 *
## versionwelfare:year1988 0.136364 0.065034 2.097 0.036016 *
## versionwelfare:year1989 0.102641 0.064260 1.597 0.110211
## versionwelfare:year1990 0.071670 0.066107 1.084 0.278305
## versionwelfare:year1991 0.055806 0.064870 0.860 0.389644
## versionwelfare:year1993 0.195739 0.064033 3.057 0.002238 **
## versionwelfare:year1994 0.235077 0.058055 4.049 5.15e-05 ***
## versionwelfare:year1996 0.113775 0.058521 1.944 0.051879 .
## versionwelfare:year1998 0.083770 0.058830 1.424 0.154468
## versionwelfare:year2000 -0.003292 0.058927 -0.056 0.955448
## versionwelfare:year2002 0.086208 0.066377 1.299 0.194031
## versionwelfare:year2004 0.146129 0.066675 2.192 0.028410 *
## versionwelfare:year2006 0.043323 0.058912 0.735 0.462113
## versionwelfare:year2008 0.039477 0.062185 0.635 0.525544
## versionwelfare:year2010 0.104903 0.062182 1.687 0.091605 .
## versionwelfare:year2012 0.041178 0.062902 0.655 0.512703
## versionwelfare:year2014 0.104862 0.060244 1.741 0.081757 .
## versionwelfare:year2016 0.141700 0.059241 2.392 0.016764 *
## versionwelfare:year2018 0.117232 0.061318 1.912 0.055900 .
## versionwelfare:year2021 -0.056476 0.056357 -1.002 0.316293
## versionwelfare:year2022 -0.052411 0.057807 -0.907 0.364590
## raceblack:year1985 0.107368 0.133694 0.803 0.421925
## raceother:year1985 0.143885 0.239371 0.601 0.547778
## raceblack:year1986 0.034689 0.134189 0.259 0.796014
## raceother:year1986 0.110839 0.251286 0.441 0.659152
## raceblack:year1987 0.086152 0.112092 0.769 0.442144
## raceother:year1987 0.110172 0.226759 0.486 0.627072
## raceblack:year1988 0.059992 0.131992 0.455 0.649462
## raceother:year1988 0.049538 0.234016 0.212 0.832352
## raceblack:year1989 0.054755 0.131943 0.415 0.678152
## raceother:year1989 0.127017 0.237936 0.534 0.593462
## raceblack:year1990 0.022909 0.132944 0.172 0.863188
## raceother:year1990 0.089279 0.234219 0.381 0.703072
## raceblack:year1991 0.128842 0.130184 0.990 0.322331
## raceother:year1991 -0.058756 0.241116 -0.244 0.807478
## raceblack:year1993 0.026189 0.131052 0.200 0.841610
## raceother:year1993 -0.022607 0.224141 -0.101 0.919660
## raceblack:year1994 -0.099091 0.116416 -0.851 0.394674
## raceother:year1994 0.037870 0.216640 0.175 0.861234
## raceblack:year1996 -0.213285 0.116084 -1.837 0.066166 .
## raceother:year1996 -0.094091 0.212220 -0.443 0.657504
## raceblack:year1998 0.119518 0.115054 1.039 0.298903
## raceother:year1998 0.047034 0.207938 0.226 0.821054
## raceblack:year2000 0.075732 0.115416 0.656 0.511717
## raceother:year2000 0.098190 0.207399 0.473 0.635906
## raceblack:year2002 0.031797 0.126729 0.251 0.801887
## raceother:year2002 0.037574 0.225826 0.166 0.867855
## raceblack:year2004 0.023764 0.131735 0.180 0.856847
## raceother:year2004 -0.033163 0.225224 -0.147 0.882940
## raceblack:year2006 0.053547 0.115645 0.463 0.643345
## raceother:year2006 0.120466 0.200395 0.601 0.547748
## raceblack:year2008 0.053598 0.120928 0.443 0.657610
## raceother:year2008 0.016575 0.209020 0.079 0.936794
## raceblack:year2010 0.121196 0.120436 1.006 0.314274
## raceother:year2010 0.118662 0.209710 0.566 0.571506
## raceblack:year2012 0.036553 0.120708 0.303 0.762025
## raceother:year2012 0.057588 0.208714 0.276 0.782612
## raceblack:year2014 -0.014132 0.116183 -0.122 0.903189
## raceother:year2014 0.018803 0.204448 0.092 0.926721
## raceblack:year2016 0.103924 0.113753 0.914 0.360933
## raceother:year2016 0.131977 0.204059 0.647 0.517789
## raceblack:year2018 0.113734 0.116666 0.975 0.329630
## raceother:year2018 0.149781 0.203956 0.734 0.462721
## raceblack:year2021 0.112725 0.114026 0.989 0.322869
## raceother:year2021 0.133187 0.199679 0.667 0.504773
## raceblack:year2022 0.100532 0.112023 0.897 0.369496
## raceother:year2022 0.102719 0.200815 0.512 0.608997
## versionwelfare:raceblack:year1985 -0.090408 0.189455 -0.477 0.633219
## versionwelfare:raceother:year1985 -0.808756 0.358914 -2.253 0.024242 *
## versionwelfare:raceblack:year1986 0.104932 0.184357 0.569 0.569236
## versionwelfare:raceother:year1986 -0.303498 0.364192 -0.833 0.404654
## versionwelfare:raceblack:year1987 -0.052192 0.163942 -0.318 0.750217
## versionwelfare:raceother:year1987 -0.382853 0.358657 -1.067 0.285769
## versionwelfare:raceblack:year1988 0.162642 0.183486 0.886 0.375406
## versionwelfare:raceother:year1988 -0.472078 0.332900 -1.418 0.156174
## versionwelfare:raceblack:year1989 0.077929 0.187731 0.415 0.678065
## versionwelfare:raceother:year1989 -0.545763 0.335537 -1.627 0.103842
## versionwelfare:raceblack:year1990 0.159985 0.188528 0.849 0.396105
## versionwelfare:raceother:year1990 -0.403854 0.332180 -1.216 0.224079
## versionwelfare:raceblack:year1991 -0.061301 0.181174 -0.338 0.735100
## versionwelfare:raceother:year1991 0.085146 0.346616 0.246 0.805954
## versionwelfare:raceblack:year1993 0.298731 0.183916 1.624 0.104323
## versionwelfare:raceother:year1993 -0.342758 0.319691 -1.072 0.283656
## versionwelfare:raceblack:year1994 0.151477 0.163178 0.928 0.353260
## versionwelfare:raceother:year1994 -0.413648 0.305824 -1.353 0.176200
## versionwelfare:raceblack:year1996 0.112439 0.162532 0.692 0.489070
## versionwelfare:raceother:year1996 -0.329151 0.298412 -1.103 0.270029
## versionwelfare:raceblack:year1998 0.189092 0.163183 1.159 0.246556
## versionwelfare:raceother:year1998 -0.303865 0.293703 -1.035 0.300862
## versionwelfare:raceblack:year2000 0.173245 0.162507 1.066 0.286395
## versionwelfare:raceother:year2000 -0.309418 0.295470 -1.047 0.295010
## versionwelfare:raceblack:year2002 0.115912 0.181350 0.639 0.522721
## versionwelfare:raceother:year2002 -0.337074 0.320561 -1.052 0.293029
## versionwelfare:raceblack:year2004 0.039893 0.187466 0.213 0.831482
## versionwelfare:raceother:year2004 -0.258198 0.312337 -0.827 0.408432
## versionwelfare:raceblack:year2006 0.150362 0.161968 0.928 0.353234
## versionwelfare:raceother:year2006 -0.425733 0.284034 -1.499 0.133912
## versionwelfare:raceblack:year2008 0.110416 0.170398 0.648 0.516995
## versionwelfare:raceother:year2008 -0.441802 0.295238 -1.496 0.134549
## versionwelfare:raceblack:year2010 -0.016829 0.168390 -0.100 0.920392
## versionwelfare:raceother:year2010 -0.452871 0.295720 -1.531 0.125673
## versionwelfare:raceblack:year2012 0.302772 0.169452 1.787 0.073980 .
## versionwelfare:raceother:year2012 -0.350667 0.294023 -1.193 0.233011
## versionwelfare:raceblack:year2014 0.190276 0.163751 1.162 0.245248
## versionwelfare:raceother:year2014 -0.365359 0.289030 -1.264 0.206206
## versionwelfare:raceblack:year2016 0.179855 0.159996 1.124 0.260967
## versionwelfare:raceother:year2016 -0.389882 0.288393 -1.352 0.176410
## versionwelfare:raceblack:year2018 0.122341 0.164620 0.743 0.457382
## versionwelfare:raceother:year2018 -0.501236 0.288893 -1.735 0.082743 .
## versionwelfare:raceblack:year2021 -0.005461 0.159677 -0.034 0.972719
## versionwelfare:raceother:year2021 -0.412849 0.283238 -1.458 0.144957
## versionwelfare:raceblack:year2022 0.103117 0.157689 0.654 0.513160
## versionwelfare:raceother:year2022 -0.441672 0.283573 -1.558 0.119352
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.7092 on 49508 degrees of freedom
## Multiple R-squared: 0.2598, Adjusted R-squared: 0.2577
## F-statistic: 121.5 on 143 and 49508 DF, p-value: < 2.2e-16#let's compare how it has changed over time
# lm6 %>%
# comparisons(
# variables = "version",
# newdata = datagrid(
# year = c("1972", "1973", "1974", "1975", "1976", "1977", "1978", "1980",
# "1982", "1983", "1984", "1985", "1986", "1987", "1988", "1989",
# "1990", "1991", "1993", "1994", "1996", "1998", "2000", "2002",
# "2004", "2006", "2008", "2010", "2012", "2014", "2016", "2018",
# "2021", "2022")
# )
# )
plot_comparisons(
lm5,
variables = "version",
by = "year"
) +
labs(
x = "",
y = "Effect of wording on support"
)