### recode likely voter items here:
df$up_901 <- yougov::recode(df$UML901, "1:2=1; else=2")
df$up_902 <- yougov::recode(df$UML902, "1:2=1; else=2")
# need to flip these
#df$up_903 <- yougov::recode(df$UML903, "4:6=1; else=2")
#df$down_903 <- yougov::recode(df$UML903, "1:2=1; else=2")
df$up_903 <- yougov::recode(df$UML903, "1:2=1; else=2")
df$down_903 <- yougov::recode(df$UML903, "4:5=1; else=2")
## original item
# prop.table(xtabs(~ df$UML901))
## reduced item
props_up1 = round(prop.table(xtabs(~ df$up_901)), 2)How to implement likely voter weights
Introduction
This is an introduction in how to implement the different types of likely voter weights for YouGov polls. The purpose of likely voter weighting is to upweight those more likely to vote, and downweight those who are less likely to turnout. While there is no universally agreed upon standard, there will always be some set of questions that will either be used to (a) weight respondents on a spectrum or (b) exclude them altogether via a Census Clicks approach.
This tutorial shall guide you through the first version, that being the standardized John Cluverius (JC) aproach as commonly implemented on University of Massachusetts at Lowell (UMAS) projects. For the purpose of today’s lesson, we will be looking at portions of the UMAS0138 project for the JC/UMAS method.1
The JC/UMAS method
The JC/UMAS method follows the common method of asking a set of questions that at the end of the standard YouGov weighting process – matching, propensity scoring, and post-stratification – is followed by a set of raking on standardized questions. The JC method employs three questions that remain constant across all surveys, which consist of (1) attention to candidates, (2) how often they vote, and (3) Intention/interest to vote in the upcoming election. They always see the same variable aliases of UML901, UML902, and UML903, respectively:
- [UML901] {single varlabel=“Attention to candidates”} As you know, there will be an election for INSERT RACE HERE2 this November. How closely are you following news about candidates running for INSERT RACE HERE?
- Very closely
- Somewhat closely
- Just a bit
- Haven’t really been following it much at all
- [UML902] {single varlabel=“How often they vote”} How often would you say that you vote when there’s a Presidential election?
- Always
- Almost always
- Just sometimes
- Hardly ever
- Never
- Just registered to vote for the first time
- [UML903] {single varlabel=“Likelihood to vote in election”} Many people don’t vote when there’s an election. At this point, how likely are you to vote in the Presidential election in November?
- I will definitely vote
- I will probably vote
- I may or may not vote
- I will probably not vote
- I will definitely not vote
After reading in the data frame from crunch and recoding the standard set of variables – along with any necessary imputation – the user needs to recode and simplify each of the three items into four binaries. UML901 and UML902 shall each become their own singular item, whereas UML903 shall be split into two, as seen below.
In the above chunk of code, we see that the four categories for UML901 are now two, with proportions of around 0.72, 0.28. In the context of likely voter weights, it is fine to be over the proportion for those likely to vote, but we do not want to downweight these. The theory being, it is good if we collect more people who are likely to vote. Ideally, we’d only sample people who sincerely max out the ends of the spectrum related to wanting to vote. It is only when we undersample these individuals that we’d want to upweight those likely to vote. Upweighting to these proportions arise from John Cluverius’s academic work suggesting that these are approximately the probabilities of turning out, all else equal.3 As can be seen right now, the results are not quite what we want, though close. We can ignore these for now, and turn our attention back to them after weighting on the traditional demographics.
c(.79,.21) - (round(prop.table(xtabs( ~ up_901, data=df)),2)) up_901
1 2
0.07 -0.07 c(.91,.09) - (round(prop.table(xtabs( ~ up_902, data=df)),2)) up_902
1 2
0.14 -0.14 c(.9,.1) - (round(prop.table(xtabs( ~ up_903, data=df)),2)) up_903
1 2
0.02 -0.02 c(.02,.98) - (round(prop.table(xtabs( ~ down_903, data=df)),2)) down_903
1 2
-0.02 0.02 Post-post statification
After completing the matching, propensity scoring, and post-stratification process – along with a review of the sample report – now it is time to rake upon the UML items created above. Generally speaking, the goal will be to attempt to get them within two percentage points of relevant proportions. However, there are two additional constraints:
Ensure that the normal marginals and joints do not fall outside the accepted thresholds for the registered voter frame
The topline horse race items adhere to what’s “reasonable” as defined by poll aggregators, other similar YouGov surveys, and checked by Rebecca, Xinling, or Julissa.4
setwd(paste0(proj_path, "/weighting"))
load(file = paste(proj, ".RData", sep = ""))
c(.79,.21) - (round(prop.table(xtabs(psweight ~ up_901, data=dfM)),2)) up_901
1 2
0.11 -0.11 c(.91,.09) - (round(prop.table(xtabs(psweight ~ up_902, data=dfM)),2)) up_902
1 2
0.19 -0.19 c(.9,.1) - (round(prop.table(xtabs(psweight ~ up_903, data=dfM)),2)) up_903
1 2
0.06 -0.06 c(.02,.98) - (round(prop.table(xtabs(psweight ~ down_903, data=dfM)),2)) down_903
1 2
-0.04 0.04 For the first constraint, keep in mind that the likely voter population will be a perfect subset of the registered voter population. One cannot vote if they are not registered, afterall.5 While we expect some deviations that will cause the marginals and joints to shift slightly, they should largely hold. In the event that there is tension and to balance the likely voter items comes at the cost of the normal marginal/joint checks, please consult with Rebecca, Xinling, or Julissa as to either:
Allow for greater flexibility for the joints
Increase the likely voter threshold to 5 points
Likewise, keep in mind that for the vast majority of cases, the likely voter weight exists in order to better estimate election horse race results. Both might be loosened so as to add an imperfect Bayesian prior against results that might otherwise contain some anomalous elements. Regardless, do not make these decisions by yourself, but rather through the aforementioned official channels.
Going back to the example at hand, we will be reading in a source file titled “lv_cleaning.R”6 which will have the lv_recodR and “lv_target_creatR” functions. These functions accept the data frame, and does the following features, if the fields of “UML901”,“UML902”, and “UML903” are present:
Code the four indicator variables recoded from the three UML fields to match that needed for weighting
Create factor variables from the four indicator UML fields
Create prop.table objects from the accepted thresholds for the LV weight
Add the relevant prop.table objects if and only if they fall below the accepted thresholds.7
now proceed to run the 4 post stratification commands on each of the UMAS likely voter items. Please note that in this case, the previous weight is named “psweight2” since there was an oversample for the UMAS0121 project. Regardless, you will want to create a new iteration of the psweight in order to compare the marginals and joints for the likely voter weight compared to the registered voter weight.8
### read in the frame
source(paste0(source_path,"/lv_cleaning.R"))
# recode the vars
dfM <- lv_recodR(dfM)
targets_lv <- lv_target_creatR(dfM)[1] "Setting up the prop_up_n objects"
[1] "Will add up_901_f to raking target"
[1] "Will add up_902_f to raking target"
[1] "Not adding up_903_f to raking"
[1] "Will add down_903_f to raking target"## Apply raking here
dfM$psweight2 <- rake_ygds(dfM, targets_lv, verbose = TRUE, maxit=100, eps=0.15, weight = dfM$psweight) [1] 0.274853434526033 [2] 0.122845695309497### trim and such
dfM$psweight2 <- trim.weight(dfM$psweight2, trimMin, max = trimMax_func(dfM))
dfM$psweight2 <- dfM$psweight2/mean(dfM$psweight2)
print(summary(dfM$psweight2)) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.001308 0.418431 0.838535 1.000000 1.305390 6.011556 As can be seen, the weighting now has the LV weight in alignment with what is expected from the JC method of applying likely voter weights.
### check the LV parts again
c(.79,.21) - (round(prop.table(xtabs(psweight2 ~ up_901, data=dfM)),2)) # off by .01up_901
1 2
-0.01 0.01 c(.91,.09) - (round(prop.table(xtabs(psweight2 ~ up_902, data=dfM)),2)) # perfect up_902
1 2
0 0 c(.9,.1) - (round(prop.table(xtabs(psweight2 ~ up_903, data=dfM)),2)) # off by .05, but higherup_903
1 2
-0.05 0.05 c(.02,.98) - print(round(prop.table(xtabs(psweight2 ~ down_903, data=dfM)),2)) # perfect down_903
1 2
0.02 0.98 down_903
1 2
0 0 We can now check how these horse race vars fare. At the time, these results would be considered fairly in line, and thus nothing out of the ordinary.
dfM$UML900c_d <- dfM$UML900c # need to fix; logic is not working
dfM$UML900c_d[is.na(dfM$UML900d)==F] <- dfM$UML900d[is.na(dfM$UML900d)==F]
dfM$UML900c_d_f <- factor(dfM$UML900c_d, labels=c("Platner", "Collins", "Other", "Don't know", "Will not vote"))
round(prop.table(xtabs(psweight2 ~ UML900c_d_f, data=dfM)),2)UML900c_d_f
Platner Collins Other Don't know Will not vote
0.48 0.43 0.02 0.06 0.01 The horse race results look fine, but let’s now see how the results fare for the marginals and such relative to the RV frame via the margin and joint checks.
#####################################
#### CHECK WEIGHTED DATA ####
#####################################
marginalDistReview(dfM,
frame,
demos=c("gender", "age4", "race2") ,
accept_margin = 1,
psweight = "psweight2",
banner = "",
banner_sub_cat = NULL)[1] "gender is not within the acceptable threshold of 1. It is off by 2.9. MOE is 4.9. Cases in df: 650. Raw numbers and Frame are off by 5"
[1] "age4 is not within the acceptable threshold of 1. It is off by 3.5. MOE is 4.9. Cases in df: 650. Raw numbers and Frame are off by 6"[1] 2# wave 1:
marginalDistReview(dfM,
frame,
demos=c("educ4", "region", "ownhome") ,
accept_margin = 2,
psweight = "psweight2",
banner = "",
banner_sub_cat = NULL)[1] "educ4 is not within the acceptable threshold of 2. It is off by 2.9. MOE is 4.9. Cases in df: 650. Raw numbers and Frame are off by 4"
[1] "ownhome is not within the acceptable threshold of 2. It is off by 3.1. MOE is 4.9. Cases in df: 650. Raw numbers and Frame are off by 17"[1] 2# wave 1:
jointDistReview(dfM,
frame,
demos=c("gender", "age4", "educ2"),
banner = c("gender", "age4", "educ2"),
accept_margin = 5,
psweight = "psweight2")[1] "Everything looked great with the joint distributions :D"# wave 1:
### now the presvote checks
cnn_pres = c(.524, .455, .017)
cnn_pres - prop.table(xtabs(dfM$psweight2 ~ dfM$presvote24 )) # looks good dfM$presvote24
1 2 3
-0.0154784557 0.0123830970 -0.0009046412 # age and male
prop.table(xtabs(dfM$psweight2[dfM$gender %in% 1 & dfM$age %in% 18:29] ~
dfM$presvote24[dfM$gender %in% 1 & dfM$age %in% 18:29] ))dfM$presvote24[dfM$gender %in% 1 & dfM$age %in% 18:29]
1 2
0.3808496 0.6191504 It unfortunately appears that the margins and joints are off. Therefore, let’s see what happens if we attempt to rake once more on the demos to the RV frame.
targets24c <- list(
# prop.table(xtabs( ~ gender_f + age4_f + race4_f + educ4_f, data = frame)),
prop.table(xtabs( ~ gender_f, data = frame)),
prop.table(xtabs( ~ age4_f, data = frame)),
prop.table(xtabs( ~ educ4_f, data = frame)))
## Apply raking here
dfM$psweight2 <- rake_ygds(dfM, targets24c, verbose = TRUE, maxit=100, eps=0.15, weight = dfM$psweight2) [1] 0.0826235846341112### check the LV parts again
c(.79,.21) - (round(prop.table(xtabs(psweight2 ~ up_901, data=dfM)),2)) # off by .02up_901
1 2
0 0 c(.91,.09) - (round(prop.table(xtabs(psweight2 ~ up_902, data=dfM)),2)) # off by .02 up_902
1 2
0.01 -0.01 c(.9,.1) - (round(prop.table(xtabs(psweight2 ~ up_903, data=dfM)),2)) # off by .01up_903
1 2
-0.04 0.04 c(.02,.98) - print(round(prop.table(xtabs(psweight2 ~ down_903, data=dfM)),2)) # ibid down_903
1 2
0.02 0.98 down_903
1 2
0 0 It looks like the LV items are fairly close following this raking. Now let’s run the margin and joint checks once more.
#####################################
#### CHECK WEIGHTED DATA ####
#####################################
marginalDistReview(dfM,
frame,
demos=c("gender", "age4", "race2") ,
accept_margin = 1,
psweight = "psweight2",
banner = "",
banner_sub_cat = NULL)[1] "Marginals look great (gender, age4, race2) :D "[1] 0# wave 1:
marginalDistReview(dfM,
frame,
demos=c("educ4", "region", "ownhome") ,
accept_margin = 2,
psweight = "psweight2",
banner = "",
banner_sub_cat = NULL)[1] "Marginals look great (educ4, region, ownhome) :D "[1] 0# wave 1: "Marginals look great (educ4, region, ownhome) :D "
# wave 2:
jointDistReview(dfM,
frame,
demos=c("gender", "age4", "educ2"),
banner = c("gender", "age4", "educ2"),
accept_margin = 5,
psweight = "psweight2")[1] "Everything looked great with the joint distributions :D"# wave 1: "Everything looked great with the joint distributions :D"
# wave 2:
### now the presvote checks
cnn_pres = c(.524, .455, .017)
cnn_pres - prop.table(xtabs(dfM$psweight2 ~ dfM$presvote24 )) # looks good dfM$presvote24
1 2 3
-0.0062862155 0.0026335905 -0.0003473751 # age and male
prop.table(xtabs(dfM$psweight2[dfM$gender %in% 1 & dfM$age %in% 18:29] ~
dfM$presvote24[dfM$gender %in% 1 & dfM$age %in% 18:29] ))dfM$presvote24[dfM$gender %in% 1 & dfM$age %in% 18:29]
1 2
0.3868561 0.6131439 horse_race = round(prop.table(xtabs(psweight2 ~ UML900c_d_f, data=dfM)),2)
horse_race_diff = (horse_race[1] - horse_race[2])*100In this most recent run, we now see that the margin and joints are also in line with the RV frame! Therefore, with the demographics, presidential vote, and LV items all in alignment, we can now proceed to deliver these results, with a Dem advantage at this point of 5 points, pending approval from Rebecca, Xinling, or Julissa.
Conclusion
So, as can be seen, the JC/UMAS method effectively acts as a wrapper to the normal US weighting process: first, the relevant UML variables are transformed at the beginning of the script, followed by the raking on the four items on top of the registered voter weight. The process is involved, given the usual demands of any horse race project. These are the type of results that will be picked up by the media at large, so it is crucial that we get it right. However, so long as you follow this guide and stay in contact with Rebecca, Xinling, or Julissa, all will work out. Just expect a few drafts at least.
Note, in the event that this is a UMAS project, the next step is followed by a standardized tabs report. It is very important that any and all issues are taken care of here and documented, lest mistakes carry over to the tabs report. For insight on this process, please see the next tutorial titled, “Tips on Tabs for UMAS projects: Pitfalls to avoid and stress reduction techniques.”
Footnotes
For the purposes of this documentation, I am running it with the updated YouGov package, which allows for R markdown documents. Therefore, some of the file paths will be different than what is standard on the server. Therefore, please be considerate when applying this code elsewhere.↩︎
i.e., President, Governor, Senate, etc.↩︎
INSERT LINK TO PUBLICATION ONCE FOUND.↩︎
There will be some wiggle room, though check first for recent YouGov surveys, or the overall window in the event that there is not a recent YouGov poll to benchmark against: Real Clear Polling↩︎
The main reason to not expect this would be that since the creation of the CPS frame, demographics significantly changed due to a new data generating process (DGP)↩︎
The source path is: “/home/surveys/projects/Internal_Projects/Analytics/weighting_document/R_WG_Development/weighting_comparison”↩︎
For example, if the up_901_f proportion is lower than 0.79 for those that closely follow the news and such.↩︎
WARNING: In the event that there are different psweights, MAKE SURE TO UPDATE THE TABS.R PREAMBLE TO REFLECT THE DIFFERENT NAME. Otherwise, the incorrect weight will be applied.↩︎