Predicting the Results of Republican Primaries
Imali Nadya Wanigasundara
1. Introduction
1.1 Objective
In the spirit of the past November elections, this paper will attempt to predict the outcome of Republican primaries based on the demographic features of each county. Various classfication methods are used to predict the winner of each county.
1.2 Data
The data explored in this study is extracted from Kaggle 2016 US Presidential Election in Primary Results. There are two datasets: primary results that contains votes by each county for each candidate and county facts that contains demographic data such as percentage of senior citizens and bachelors degree holders and median household income per county. The analysis is only based on Republican candidates performances here.
2. Data Exploration
Load all relevant libraries.
# load libraries
library(readr)
library(magrittr)
library(dplyr)
library(ggplot2)
library(caret)
library(C50)After loading the 2 datasets, a new dataframe is created by filtering for only Republican votes and creating a new data frame where the winner of each county is defined as the candidate with the maximum percentage of votes. Then, another dataset is extracted from county demogrpahics, leaving variables such as percentage of non-native English speakers and retail sales per capita. In this dataset, we have only included demographc attributes such as percent of white population, population density, poverty rate and percent of veterans per county. Finally, the two datasets, primary results and county facts, are joined based on county name and state.
# load 2 datasets
county_facts <- read_csv("~/Downloads/county_facts.csv")
primary_results <- read_csv("~/Downloads/primary_results.csv.zip")
# winners as candidate with max vote percent from each county
votes <- primary_results %>%
filter(party == "Republican") %>%
group_by(state_abbreviation, county) %>%
summarize(winner = candidate[which.max(fraction_votes)],
frac_vote = max(fraction_votes),
votes = max(votes))
# extracting only relevant attributes for county
county_facts %<>%
select(state_abbreviation = state_abbreviation, county = area_name,
income = INC110213, hispanic = RHI725214, white= RHI825214,
college = EDU685213, density = POP060210, over65 = AGE775214,
female=SEX255214, poverty_level = PVY020213, veterans =VET605213) %>%
mutate(county = gsub(" County", "", county))
#join on state abb & county
repubvotes <- inner_join(votes, county, by = c("state_abbreviation","county"))Error in is.data.frame(y) : object 'county' not foundThere are only 2 counties which recorded a win for Ben Carson, so a new dataset is created excluding Ben Carson results.
table(repubvotes$winner)
Ben Carson Donald Trump John Kasich Marco Rubio Ted Cruz
2 1945 59 27 596 repub_votes <- repubvotes %>%
filter(winner=="Donald Trump" | winner=="Ted Cruz" | winner=="Marco Rubio"| winner=="John Kasich")Unfortunately it is important to note that as evident from the above values, the data is not uniformly distributed between 4 target variables.
Lets explore some county attributes and the wining candidate’s performances before we delve into some classfication methods.
By visualizing the candidates performance by Income and Education, we can observe that Donald Trump and Ted Cruz seem to have won in counties with lowe percent of college educated persons earning lower medium income while Marco Rubio has outperformed all candidates in highly educated counties.
qplot(income,college, data = repub_votes, geom="point",
color = winner,
main = "Candidate Performance by Income and Education", xlab = "household median income", ylab = "college")
In the below boxplot, we can see that Donald Trump and Ted Cruz have done well in communites with few percentage of white population indicating these candidates ability to appeal to other ethnicities.
qplot(winner, white, data=repub_votes, geom="boxplot", fill=winner, main="Candidate Performance by % of White Pop", xlab="candidate", ylab="% of white population")
When we plot candidate performance by age, once again we notice that Trump has a wider appeal than his rivals.
qplot(winner, over65, data=repub_votes, geom="boxplot", fill=winner, main="Candidate Performance by Senior Citizens", xlab="candidate",ylab="% of people 65 or over")
3. Classification Methods & Results
First, we will remove state abbreviation and county from the dataset. The target variable (variable predicted) is winner with 4 levels. There are 1945 occurences where Donald Trump became the winner followed by Ted Cruz winning in 596 counties.
repub <- subset(repub_votes, select = -c(state_abbreviation,county))
names(repub) [1] "winner" "frac_vote" "votes" "income" "hispanic" "white"
[7] "college" "density" "over65" "female" "poverty_level" "veterans" repub$winner <- as.factor(repub$winner)
summary(repub$winner)Donald Trump John Kasich Marco Rubio Ted Cruz
1945 59 27 596 We will use 4 classification methods to predict the winner of each county.
3.1 Decision Tress
Decision tree is built by repeatedly dividing data into groups based on demographic features. The attribute on which to divide is based on a statistical technique called, information gain, that determines which attribute split will divide the data in a clean manner.
We will use 5 fold cross validation to train the model. A training test is created that contains 75% (or 1972 records) of data from the repub dataset. The remaning will data is used for testing the model results.
# stratified sampling to create a data partition
set.seed(40)
TrainingDataIndex <- createDataPartition(repub$winner, p=0.75, list = FALSE)
train <- repub[TrainingDataIndex,]
# 25% of data is used to train the model
test <- repub[-TrainingDataIndex,]
# decision tree built wit c5.0 algorithm
DecTreeModel <- train(winner ~ ., data = train,
method = "C5.0",
trControl= trainControl(method = "cv", number = 5),
na.action = na.omit)Lets see how well the model classified each county winner using test data.
DTPredictions <-predict(DecTreeModel, test, na.action = na.pass)
dt_cf <- confusionMatrix(DTPredictions, test$winner)
dt_cfConfusion Matrix and Statistics
Reference
Prediction Donald Trump John Kasich Marco Rubio Ted Cruz
Donald Trump 447 12 4 57
John Kasich 3 2 0 0
Marco Rubio 1 0 1 2
Ted Cruz 35 0 1 90
Overall Statistics
Accuracy : 0.8244
95% CI : (0.7931, 0.8528)
No Information Rate : 0.742
P-Value [Acc > NIR] : 3.388e-07
Kappa : 0.5216
Mcnemar's Test P-Value : NA
Statistics by Class:
Class: Donald Trump Class: John Kasich Class: Marco Rubio Class: Ted Cruz
Sensitivity 0.9198 0.142857 0.166667 0.6040
Specificity 0.5680 0.995320 0.995378 0.9289
Pos Pred Value 0.8596 0.400000 0.250000 0.7143
Neg Pred Value 0.7111 0.981538 0.992320 0.8885
Prevalence 0.7420 0.021374 0.009160 0.2275
Detection Rate 0.6824 0.003053 0.001527 0.1374
Detection Prevalence 0.7939 0.007634 0.006107 0.1924
Balanced Accuracy 0.7439 0.569088 0.581022 0.7664The decision tree model classified class variable winner with about 82% accuracy. In 57 counties, the model misclassified the winner as Donald Trump when Ted Cruz won those counties. The confusion matrix also shows that the model misclassified Ted Cruz as the winner in 35 instances when Donald Trump won those counties. Sensitivity is another important metric that measures the true positive rate, i.e. when a candidate actually wins a county, how well does model predict that same candidate won that county. Sensitivity for Donald Trump is highest and this could be because our class variables are not uniformly distributed.
3.2 Naive Bayes
Naive Bayes method calculates probabilities given certain event occuring. This method assumes that predictor variables are independent from each other. For instance, the model assumes that median household income and poverty rate are independent from each other as a predictor of which candidate wins the county.
NBModel <- train(train[,-1], train$winner,
method = "nb",
trControl= trainControl(method = "cv", number = 5))Below is the confusion matrix using test data to evaluate Naive Bayes method in predicting the winner of the county elections. The model seems to have a lesser accuracy 75% than that from Decision Trees. For instance, the model predicted that Donald Trump was the winning candidate in 89 counties but Ted Cruz truned out to be the actual winner. In 20 counties, the Naive Bayes model misclassified target variable predicting that Marco Rubio would win when Donald Trump won those counties. Further analysis on the sensitivity shows that the true positive rate has decreased for Trump by about 1% in absolute terms and has increased for Marco Rubio to 67%. Although the true positive rate has gone up for Marco Rubio it is important to note that the model misclassified Rubio in 31 counties when he only won 6 counties in total.
NBPredictions <-predict(NBModel, test)
confusionMatrix(NBPredictions, test$winner)Confusion Matrix and Statistics
Reference
Prediction Donald Trump John Kasich Marco Rubio Ted Cruz
Donald Trump 439 12 1 89
John Kasich 15 2 0 5
Marco Rubio 20 0 4 11
Ted Cruz 12 0 1 44
Overall Statistics
Accuracy : 0.7466
95% CI : (0.7114, 0.7795)
No Information Rate : 0.742
P-Value [Acc > NIR] : 0.4144
Kappa : 0.3078
Mcnemar's Test P-Value : NA
Statistics by Class:
Class: Donald Trump Class: John Kasich Class: Marco Rubio Class: Ted Cruz
Sensitivity 0.9033 0.142857 0.666667 0.29530
Specificity 0.3964 0.968799 0.952234 0.97431
Pos Pred Value 0.8115 0.090909 0.114286 0.77193
Neg Pred Value 0.5877 0.981043 0.996774 0.82441
Prevalence 0.7420 0.021374 0.009160 0.22748
Detection Rate 0.6702 0.003053 0.006107 0.06718
Detection Prevalence 0.8260 0.033588 0.053435 0.08702
Balanced Accuracy 0.6499 0.555828 0.809450 0.634813.3 Support Vector Machines (SVM)
Support Vector Machine (SVM) classification plots each data point in a coordinate plane with each feautre being the value of a particular coordinate. Then the model performs classification by finding the hyperplane between two classes.
When we perform SVM model to predict the county winner, the model does not make any predictions for Marco Rubio, partly because there are so few data points for Rubio. In terms of accuracy, the SVM model displays 82% accuracy which is significantly improved from Naive Bayes model. However, the model incorrectly classifies Donald Trump as the winner in 13 instance when indeed John Kasich won those 13 counties. The true positive rate of SVM model has improved for Trump correctly classifiying Trump as the winner in 97% of instances he actually won.
set.seed(60)
SVMMmodel <- train(winner ~., train,
method='svmPoly',
trControl=trainControl(method = "cv", number = 5))
SVMPredictions <-predict(SVMMmodel, test)
confusionMatrix(SVMPredictions, test$winner)Confusion Matrix and Statistics
Reference
Prediction Donald Trump John Kasich Marco Rubio Ted Cruz
Donald Trump 474 13 6 89
John Kasich 0 1 0 0
Marco Rubio 0 0 0 0
Ted Cruz 12 0 0 60
Overall Statistics
Accuracy : 0.8168
95% CI : (0.785, 0.8457)
No Information Rate : 0.742
P-Value [Acc > NIR] : 3.682e-06
Kappa : 0.4196
Mcnemar's Test P-Value : NA
Statistics by Class:
Class: Donald Trump Class: John Kasich Class: Marco Rubio Class: Ted Cruz
Sensitivity 0.9753 0.071429 0.00000 0.4027
Specificity 0.3609 1.000000 1.00000 0.9763
Pos Pred Value 0.8144 1.000000 NaN 0.8333
Neg Pred Value 0.8356 0.980122 0.99084 0.8473
Prevalence 0.7420 0.021374 0.00916 0.2275
Detection Rate 0.7237 0.001527 0.00000 0.0916
Detection Prevalence 0.8885 0.001527 0.00000 0.1099
Balanced Accuracy 0.6681 0.535714 0.50000 0.68953.4 Neural Networks
Neural networks mimic the structure and learning process of neurons in the brains. They group feature vectors into classes, takes new input data and find out which label fits best.
NNModel <- train(train[,-1], train$winner,
method = "nnet",
trControl= trainControl(method = "cv", number = 5))When we evaluate the NN model with test data, the model does not make any recommendations for either Marco Rubio or Ted Cruz. The model contains a relatively low accuracy at about 74% compared to other models.
NNPredictions <-predict(NNModel, test, na.action = na.pass)
confusionMatrix(NNPredictions, test$winner)Confusion Matrix and Statistics
Reference
Prediction Donald Trump John Kasich Marco Rubio Ted Cruz
Donald Trump 486 13 6 149
John Kasich 0 1 0 0
Marco Rubio 0 0 0 0
Ted Cruz 0 0 0 0
Overall Statistics
Accuracy : 0.7435
95% CI : (0.7083, 0.7766)
No Information Rate : 0.742
P-Value [Acc > NIR] : 0.4851
Kappa : 0.0101
Mcnemar's Test P-Value : NA
Statistics by Class:
Class: Donald Trump Class: John Kasich Class: Marco Rubio Class: Ted Cruz
Sensitivity 1.000000 0.071429 0.00000 0.0000
Specificity 0.005917 1.000000 1.00000 1.0000
Pos Pred Value 0.743119 1.000000 NaN NaN
Neg Pred Value 1.000000 0.980122 0.99084 0.7725
Prevalence 0.741985 0.021374 0.00916 0.2275
Detection Rate 0.741985 0.001527 0.00000 0.0000
Detection Prevalence 0.998473 0.001527 0.00000 0.0000
Balanced Accuracy 0.502959 0.535714 0.50000 0.50004. Conclusion
This study attempts to predict the results of 2016 Republican primary elections in each county based on four classificiation methods. Based on the accuracy of all four models, we can conclude that Decision Trees model produced the highest accuracy of 82% while neural networks produced the lowest accuracy of 74%. The decision tree model correctly classified the runner up in Republican primaries, Ted Cruz in 60% of the counties that he won, the highest sensitivity produced among all models. In an ideal classification problem, the training and testing dataset would contain an equal number of size in each class. It is important to highlight that the training tends to pull the classifier towards the dominant class Donald Trump in making predictions.
These model performances for all four candidates could have been improved by choosing a higher value for k in k-cross fold validation. Recall that all models in above methods use 5-fold cross validation to estimate the accuracy of the model. We could also even out the number of records for each candidate, however this would create a small dataset to perform classification models on.
5. Future Recommendations
Additionally, we hope to use ensemble methods such as boosting, bagging and stacking to improve the accuracy of the models explored above. It may also be useful to perform k means and Kohonen clusering methods on county demographics to determine a holisitic view of voting trends. This would provide a good context to understand which voting groups tend to lean towards Marco Rubio versus Ted Cruz and how are voting demographcs different between Trump and Ted Cruz.
6. References
1: Hammer, Ben. “2016 US Election.” Kaggle. N.p., n.d. Web. https://www.kaggle.com/benhamner/2016-us-election
2: Papiu, Alexandru. “Predictions in the Republican Primary.” N.p., n.d. Web.
https://www.kaggle.com/apapiu/d/benhamner/2016-us-election/predictions-in-the-republican-primary