AS4-3
Problem 1.1 - A Logistic Regression Model
Let’s begin by building a logistic regression model to predict whether an individual’s earnings are above $50,000 (the variable “over50k”) using all of the other variables as independent variables. First, read the dataset census.csv into R.
Then, split the data randomly into a training set and a testing set, setting the seed to 2000 before creating the split. Split the data so that the training set contains 60% of the observations, while the testing set contains 40% of the observations.
Next, build a logistic regression model to predict the dependent variable “over50k”, using all of the other variables in the dataset as independent variables. Use the training set to build the model.
Which variables are significant, or have factors that are significant? (Use 0.1 as your significance threshold, so variables with a period or dot in the stars column should be counted too. You might see a warning message here - you can ignore it and proceed. This message is a warning that we might be overfitting our model to the training set.) Select all that apply.
library(caTools)
census <- read.csv("census.csv")
set.seed(2000)
spl <- sample.split(census$over50k, SplitRatio = 0.6)
train <- subset(census, spl == TRUE)
test <- subset(census, spl == FALSE)
modelCensus <- glm( over50k ~ . , family = "binomial", data = train)glm.fit: fitted probabilities numerically 0 or 1 occurredsummary(modelCensus)
Call:
glm(formula = over50k ~ ., family = "binomial", data = train)
Deviance Residuals:
Min 1Q Median 3Q Max
-5.1065 -0.5037 -0.1804 -0.0008 3.3383
Coefficients: (1 not defined because of singularities)
Estimate Std. Error z value Pr(>|z|)
(Intercept) -8.658e+00 1.379e+00 -6.279 3.41e-10 ***
age 2.548e-02 2.139e-03 11.916 < 2e-16 ***
workclass Federal-gov 1.105e+00 2.014e-01 5.489 4.03e-08 ***
workclass Local-gov 3.675e-01 1.821e-01 2.018 0.043641 *
workclass Never-worked -1.283e+01 8.453e+02 -0.015 0.987885
workclass Private 6.012e-01 1.626e-01 3.698 0.000218 ***
workclass Self-emp-inc 7.575e-01 1.950e-01 3.884 0.000103 ***
workclass Self-emp-not-inc 1.855e-01 1.774e-01 1.046 0.295646
workclass State-gov 4.012e-01 1.961e-01 2.046 0.040728 *
workclass Without-pay -1.395e+01 6.597e+02 -0.021 0.983134
education 11th 2.225e-01 2.867e-01 0.776 0.437738
education 12th 6.380e-01 3.597e-01 1.774 0.076064 .
education 1st-4th -7.075e-01 7.760e-01 -0.912 0.361897
education 5th-6th -3.170e-01 4.880e-01 -0.650 0.516008
education 7th-8th -3.498e-01 3.126e-01 -1.119 0.263152
education 9th -1.258e-01 3.539e-01 -0.355 0.722228
education Assoc-acdm 1.602e+00 2.427e-01 6.601 4.10e-11 ***
education Assoc-voc 1.541e+00 2.368e-01 6.506 7.74e-11 ***
education Bachelors 2.177e+00 2.218e-01 9.817 < 2e-16 ***
education Doctorate 2.761e+00 2.893e-01 9.544 < 2e-16 ***
education HS-grad 1.006e+00 2.169e-01 4.638 3.52e-06 ***
education Masters 2.421e+00 2.353e-01 10.289 < 2e-16 ***
education Preschool -2.237e+01 6.864e+02 -0.033 0.973996
education Prof-school 2.938e+00 2.753e-01 10.672 < 2e-16 ***
education Some-college 1.365e+00 2.195e-01 6.219 5.00e-10 ***
maritalstatus Married-AF-spouse 2.540e+00 7.145e-01 3.555 0.000378 ***
maritalstatus Married-civ-spouse 2.458e+00 3.573e-01 6.880 6.00e-12 ***
maritalstatus Married-spouse-absent -9.486e-02 3.204e-01 -0.296 0.767155
maritalstatus Never-married -4.515e-01 1.139e-01 -3.962 7.42e-05 ***
maritalstatus Separated 3.609e-02 1.984e-01 0.182 0.855672
maritalstatus Widowed 1.858e-01 1.962e-01 0.947 0.343449
occupation Adm-clerical 9.470e-02 1.288e-01 0.735 0.462064
occupation Armed-Forces -1.008e+00 1.487e+00 -0.677 0.498170
occupation Craft-repair 2.174e-01 1.109e-01 1.960 0.049972 *
occupation Exec-managerial 9.400e-01 1.138e-01 8.257 < 2e-16 ***
occupation Farming-fishing -1.068e+00 1.908e-01 -5.599 2.15e-08 ***
occupation Handlers-cleaners -6.237e-01 1.946e-01 -3.204 0.001353 **
occupation Machine-op-inspct -1.862e-01 1.376e-01 -1.353 0.176061
occupation Other-service -8.183e-01 1.641e-01 -4.987 6.14e-07 ***
occupation Priv-house-serv -1.297e+01 2.267e+02 -0.057 0.954385
occupation Prof-specialty 6.331e-01 1.222e-01 5.180 2.22e-07 ***
occupation Protective-serv 6.267e-01 1.710e-01 3.664 0.000248 ***
occupation Sales 3.276e-01 1.175e-01 2.789 0.005282 **
occupation Tech-support 6.173e-01 1.533e-01 4.028 5.63e-05 ***
occupation Transport-moving NA NA NA NA
relationship Not-in-family 7.881e-01 3.530e-01 2.233 0.025562 *
relationship Other-relative -2.194e-01 3.137e-01 -0.699 0.484263
relationship Own-child -7.489e-01 3.507e-01 -2.136 0.032716 *
relationship Unmarried 7.041e-01 3.720e-01 1.893 0.058392 .
relationship Wife 1.324e+00 1.331e-01 9.942 < 2e-16 ***
race Asian-Pac-Islander 4.830e-01 3.548e-01 1.361 0.173504
race Black 3.644e-01 2.882e-01 1.265 0.206001
race Other 2.204e-01 4.513e-01 0.488 0.625263
race White 4.108e-01 2.737e-01 1.501 0.133356
sex Male 7.729e-01 1.024e-01 7.545 4.52e-14 ***
capitalgain 3.280e-04 1.372e-05 23.904 < 2e-16 ***
capitalloss 6.445e-04 4.854e-05 13.277 < 2e-16 ***
hoursperweek 2.897e-02 2.101e-03 13.791 < 2e-16 ***
nativecountry Canada 2.593e-01 1.308e+00 0.198 0.842879
nativecountry China -9.695e-01 1.327e+00 -0.730 0.465157
nativecountry Columbia -1.954e+00 1.526e+00 -1.280 0.200470
nativecountry Cuba 5.735e-02 1.323e+00 0.043 0.965432
nativecountry Dominican-Republic -1.435e+01 3.092e+02 -0.046 0.962972
nativecountry Ecuador -3.550e-02 1.477e+00 -0.024 0.980829
nativecountry El-Salvador -6.095e-01 1.395e+00 -0.437 0.662181
nativecountry England -6.707e-02 1.327e+00 -0.051 0.959686
nativecountry France 5.301e-01 1.419e+00 0.374 0.708642
nativecountry Germany 5.474e-02 1.306e+00 0.042 0.966572
nativecountry Greece -2.646e+00 1.714e+00 -1.544 0.122527
nativecountry Guatemala -1.293e+01 3.345e+02 -0.039 0.969180
nativecountry Haiti -9.221e-01 1.615e+00 -0.571 0.568105
nativecountry Holand-Netherlands -1.282e+01 2.400e+03 -0.005 0.995736
nativecountry Honduras -9.584e-01 3.412e+00 -0.281 0.778775
nativecountry Hong -2.362e-01 1.492e+00 -0.158 0.874155
nativecountry Hungary 1.412e-01 1.555e+00 0.091 0.927653
nativecountry India -8.218e-01 1.314e+00 -0.625 0.531661
nativecountry Iran -3.299e-02 1.366e+00 -0.024 0.980736
nativecountry Ireland 1.579e-01 1.473e+00 0.107 0.914628
nativecountry Italy 6.100e-01 1.333e+00 0.458 0.647194
nativecountry Jamaica -2.279e-01 1.387e+00 -0.164 0.869467
nativecountry Japan 5.072e-01 1.375e+00 0.369 0.712179
nativecountry Laos -6.831e-01 1.661e+00 -0.411 0.680866
nativecountry Mexico -9.182e-01 1.303e+00 -0.705 0.481103
nativecountry Nicaragua -1.987e-01 1.507e+00 -0.132 0.895132
nativecountry Outlying-US(Guam-USVI-etc) -1.373e+01 8.502e+02 -0.016 0.987115
nativecountry Peru -9.660e-01 1.678e+00 -0.576 0.564797
nativecountry Philippines 4.393e-02 1.281e+00 0.034 0.972640
nativecountry Poland 2.410e-01 1.383e+00 0.174 0.861624
nativecountry Portugal 7.276e-01 1.477e+00 0.493 0.622327
nativecountry Puerto-Rico -5.769e-01 1.357e+00 -0.425 0.670837
nativecountry Scotland -1.188e+00 1.719e+00 -0.691 0.489616
nativecountry South -8.183e-01 1.341e+00 -0.610 0.541809
nativecountry Taiwan -2.590e-01 1.350e+00 -0.192 0.847878
nativecountry Thailand -1.693e+00 1.737e+00 -0.975 0.329678
nativecountry Trinadad&Tobago -1.346e+00 1.721e+00 -0.782 0.434105
nativecountry United-States -8.594e-02 1.269e+00 -0.068 0.946020
nativecountry Vietnam -1.008e+00 1.523e+00 -0.662 0.507799
nativecountry Yugoslavia 1.402e+00 1.648e+00 0.851 0.394874
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 21175 on 19186 degrees of freedom
Residual deviance: 12104 on 19090 degrees of freedom
AIC: 12298
Number of Fisher Scoring iterations: 15print("age , workclass , education , maritalstatus , occupation , relationship , sex , capitalgain , capitalloss , hoursperweek" , quote = FALSE)[1] age , workclass , education , maritalstatus , occupation , relationship , sex , capitalgain , capitalloss , hoursperweekProblem 1.2 - A Logistic Regression Model
What is the accuracy of the model on the testing set? Use a threshold of 0.5. (You might see a warning message when you make predictions on the test set - you can safely ignore it.)
predCensus <- predict(modelCensus, newdata = test, type = "response")prediction from a rank-deficient fit may be misleadingtemp <- table(test$over50k, predCensus >= 0.5)
( temp[1,1] + temp[2,2] ) / sum(temp)[1] 0.8552107Problem 1.3 - A Logistic Regression Model
What is the baseline accuracy for the testing set?
temp <- table(test$over50k)
max(temp) / sum(temp)[1] 0.7593621Problem 1.4 - A Logistic Regression Model
What is the area-under-the-curve (AUC) for this model on the test set?
library(ROCR)
ROCpred <- prediction(predCensus, test$over50k)
as.numeric(performance(ROCpred, "auc")@y.values)[1] 0.9061598Problem 2.1 - A CART Model
We have just seen how the logistic regression model for this data achieves a high accuracy. Moreover, the significances of the variables give us a way to gauge which variables are relevant for this prediction task. However, it is not immediately clear which variables are more important than the others, especially due to the large number of factor variables in this problem.
Let us now build a classification tree to predict “over50k”. Use the training set to build the model, and all of the other variables as independent variables. Use the default parameters, so don’t set a value for minbucket or cp. Remember to specify method=“class” as an argument to rpart, since this is a classification problem. After you are done building the model, plot the resulting tree.
How many splits does the tree have in total?
library(rpart)
library(rpart.plot)
censustree <- rpart( over50k ~ . , method = "class", data = train)
prp(censustree)
4[1] 4Problem 2.2 - A CART Model
Which variable does the tree split on at the first level (the very first split of the tree)?
print("relationship" , quote = FALSE)[1] relationshipProblem 2.3 - A CART Model
Which variables does the tree split on at the second level (immediately after the first split of the tree)? Select all that apply.
print("education , capitalgain" , quote = FALSE)[1] education , capitalgainProblem 2.4 - A CART Model
What is the accuracy of the model on the testing set? Use a threshold of 0.5. (You can either add the argument type=“class”, or generate probabilities and use a threshold of 0.5 like in logistic regression.)
predTree <- predict(censustree, newdata = test, type = "class")
temp <- table(test$over50k, predTree)
( temp[1,1] + temp[2,2] ) / sum(temp)[1] 0.8473927Problem 2.5 - A CART Model
Let us now consider the ROC curve and AUC for the CART model on the test set. You will need to get predicted probabilities for the observations in the test set to build the ROC curve and compute the AUC. Remember that you can do this by removing the type=“class” argument when making predictions, and taking the second column of the resulting object.
Plot the ROC curve for the CART model you have estimated. Observe that compared to the logistic regression ROC curve, the CART ROC curve is less smooth than the logistic regression ROC curve. Which of the following explanations for this behavior is most correct? (HINT: Think about what the ROC curve is plotting and what changing the threshold does.)
print("The CART model uses fewer continuous variables than the logistic regression model (capitalgain for CART versus age, capitalgain, capitallosses, hoursperweek), which is why the CART ROC curve is less smooth than the logistic regression one." , quote = FALSE)[1] The CART model uses fewer continuous variables than the logistic regression model (capitalgain for CART versus age, capitalgain, capitallosses, hoursperweek), which is why the CART ROC curve is less smooth than the logistic regression one.# 邏輯斯回歸是將整個結果以機率表現出來，可以連續性的去篩選其結果，因此 ROC 曲線可以較為平滑Problem 2.6 - A CART Model
What is the AUC of the CART model on the test set?
predTree <- predict(censustree, newdata = test)
predTree <- predTree[,2]
ROCpred = prediction(predTree, test$over50k)
as.numeric(performance(ROCpred, "auc")@y.values)[1] 0.8470256Problem 3.1 - A Random Forest Model
Before building a random forest model, we’ll down-sample our training set. While some modern personal computers can build a random forest model on the entire training set, others might run out of memory when trying to train the model since random forests is much more computationally intensive than CART or Logistic Regression. For this reason, before continuing we will define a new training set to be used when building our random forest model, that contains 2000 randomly selected obervations from the original training set. Do this by running the following commands in your R console (assuming your training set is called “train”):
set.seed(1)
trainSmall = train[sample(nrow(train), 2000), ]
Let us now build a random forest model to predict “over50k”, using the dataset “trainSmall” as the data used to build the model. Set the seed to 1 again right before building the model, and use all of the other variables in the dataset as independent variables. (If you get an error that random forest “can not handle categorical predictors with more than 32 categories”, re-build the model without the nativecountry variable as one of the independent variables.)
Then, make predictions using this model on the entire test set. What is the accuracy of the model on the test set, using a threshold of 0.5? (Remember that you don’t need a “type” argument when making predictions with a random forest model if you want to use a threshold of 0.5. Also, note that your accuracy might be different from the one reported here, since random forest models can still differ depending on your operating system, even when the random seed is set. )
set.seed(1)
trainSmall <- train[sample(nrow(train), 2000), ]
set.seed(1)
modelRF <- randomForest(over50k ~ . , data = trainSmall)
predRF <- predict(modelRF, newdata = test)
temp <- table(test$over50k, predRF)
( temp[1,1] + temp[2,2] ) / sum(temp)[1] 0.8348839Problem 3.2 - A Random Forest Model
As we discussed in lecture, random forest models work by building a large collection of trees. As a result, we lose some of the interpretability that comes with CART in terms of seeing how predictions are made and which variables are important. However, we can still compute metrics that give us insight into which variables are important.
One metric that we can look at is the number of times, aggregated over all of the trees in the random forest model, that a certain variable is selected for a split. To view this metric, run the following lines of R code (replace “MODEL” with the name of your random forest model):
vu = varUsed(MODEL, count=TRUE)
vusorted = sort(vu, decreasing = FALSE, index.return = TRUE)
dotchart(vusorted\(x, names(MODEL\)forest\(xlevels[vusorted\)ix]))
This code produces a chart that for each variable measures the number of times that variable was selected for splitting (the value on the x-axis). Which of the following variables is the most important in terms of the number of splits?
vu <- varUsed(modelRF, count = TRUE)
vusorted <- sort(vu, decreasing = FALSE, index.return = TRUE)
dotchart(vusorted$x, names(modelRF$forest$xlevels[vusorted$ix]))
print("age" , quote = FALSE)[1] ageProblem 3.3 - A Random Forest Model
A different metric we can look at is related to “impurity”, which measures how homogenous each bucket or leaf of the tree is. In each tree in the forest, whenever we select a variable and perform a split, the impurity is decreased. Therefore, one way to measure the importance of a variable is to average the reduction in impurity, taken over all the times that variable is selected for splitting in all of the trees in the forest. To compute this metric, run the following command in R (replace “MODEL” with the name of your random forest model):
varImpPlot(MODEL)
Which one of the following variables is the most important in terms of mean reduction in impurity?
varImpPlot(modelRF)
print("occupation" , quote = FALSE)[1] occupationProblem 4.1 - Selecting cp by Cross-Validation
We now conclude our study of this data set by looking at how CART behaves with different choices of its parameters.
Let us select the cp parameter for our CART model using k-fold cross validation, with k = 10 folds. Do this by using the train function. Set the seed beforehand to 2. Test cp values from 0.002 to 0.1 in 0.002 increments, by using the following command:
cartGrid = expand.grid( .cp = seq(0.002,0.1,0.002))
Also, remember to use the entire training set “train” when building this model. The train function might take some time to run.
Which value of cp does the train function recommend?
library(caret)Loading required package: lattice
Loading required package: ggplot2
Attaching package: ‘ggplot2’
The following object is masked from ‘package:randomForest’:
marginset.seed(2)
fitControl <- trainControl( method = "cv", number = 10 )
cartGrid <- expand.grid( .cp = seq(0.002,0.1,0.002))
train( over50k ~ . , data = train, method = "rpart", trControl = fitControl, tuneGrid = cartGrid )CART
19187 samples
12 predictor
2 classes: ' <=50K', ' >50K'
No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 17268, 17268, 17269, 17269, 17269, 17268, ...
Resampling results across tuning parameters:
cp Accuracy Kappa
0.002 0.8510972 0.55404931
0.004 0.8482829 0.55537475
0.006 0.8452078 0.53914084
0.008 0.8442176 0.53817486
0.010 0.8433317 0.53305978
0.012 0.8433317 0.53305978
0.014 0.8433317 0.53305978
0.016 0.8413510 0.52349296
0.018 0.8400480 0.51528594
0.020 0.8381193 0.50351272
0.022 0.8381193 0.50351272
0.024 0.8381193 0.50351272
0.026 0.8381193 0.50351272
0.028 0.8381193 0.50351272
0.030 0.8381193 0.50351272
0.032 0.8381193 0.50351272
0.034 0.8352011 0.48749911
0.036 0.8326470 0.47340390
0.038 0.8267570 0.44688035
0.040 0.8248289 0.43893150
0.042 0.8248289 0.43893150
0.044 0.8248289 0.43893150
0.046 0.8248289 0.43893150
0.048 0.8248289 0.43893150
0.050 0.8231084 0.42467058
0.052 0.8174798 0.37478096
0.054 0.8138837 0.33679015
0.056 0.8118514 0.30751485
0.058 0.8118514 0.30751485
0.060 0.8118514 0.30751485
0.062 0.8118514 0.30751485
0.064 0.8118514 0.30751485
0.066 0.8099233 0.29697206
0.068 0.7971025 0.22226318
0.070 0.7958512 0.21465656
0.072 0.7958512 0.21465656
0.074 0.7958512 0.21465656
0.076 0.7689601 0.05701508
0.078 0.7593684 0.00000000
0.080 0.7593684 0.00000000
0.082 0.7593684 0.00000000
0.084 0.7593684 0.00000000
0.086 0.7593684 0.00000000
0.088 0.7593684 0.00000000
0.090 0.7593684 0.00000000
0.092 0.7593684 0.00000000
0.094 0.7593684 0.00000000
0.096 0.7593684 0.00000000
0.098 0.7593684 0.00000000
0.100 0.7593684 0.00000000
Accuracy was used to select the optimal model using the largest value.
The final value used for the model was cp = 0.002.0.002[1] 0.002Problem 4.2 - Selecting cp by Cross-Validation
Fit a CART model to the training data using this value of cp. What is the prediction accuracy on the test set?
modelCP002 <- rpart(over50k~., data = train, method = "class", cp = 0.002)
predCP002 <- predict(modelCP002 , newdata = test, type = "class")
temp <- table(test$over50k, predCP002)
( temp[1,1] + temp[2,2] ) / sum(temp)[1] 0.8612306Problem 4.3 - Selecting cp by Cross-Validation
Compared to the original accuracy using the default value of cp, this new CART model is an improvement, and so we should clearly favor this new model over the old one – or should we? Plot the CART tree for this model. How many splits are there?
prp(modelCP002)
18[1] 18