ISLR Chapter 5 Problem 5

suppressPackageStartupMessages(library('ISLR'))

package 㤼㸱ISLR㤼㸲 was built under R version 3.6.3

summary(Default)

 default    student       balance           income     
 No :9667   No :7056   Min.   :   0.0   Min.   :  772  
 Yes: 333   Yes:2944   1st Qu.: 481.7   1st Qu.:21340  
                       Median : 823.6   Median :34553  
                       Mean   : 835.4   Mean   :33517  
                       3rd Qu.:1166.3   3rd Qu.:43808  
                       Max.   :2654.3   Max.   :73554  

attach(Default)

In Chapter 4, we used logistic regression to predict the probability of default using income and balance on the Default data set. We will now estimate the test error of this logistic regression model using the validation set approach. Do not forget to set a random seed before beginning your analysis.

1. Fit a logistic regression model that uses income and balance to predict default.

set.seed(1)
glm.fit = glm(default ~ income + balance, data = Default, family = binomial)

2.Using the validation set approach, estimate the test error of this model. In order to do this, you must perform the following steps:

• Split the sample set into a training set and a validation set.
• Fit a multiple logistic regression model using only the training observations.
• Obtain a prediction of default status for each individual in the validation set by computing the posterior probability of default for that individual, and classifying the individual to the default category if the posterior probability is greater than 0.5.
• Compute the validation set error, which is the fraction of the observations in the validation set that are misclassified.

train = sample(dim(Default)[1], dim(Default)[1]/2) ## i
glm.fit = glm(default ~ income + balance, data = Default, family = binomial, subset = train) ## ii
## iii
glm.pred = rep("No", dim(Default)[1]/2)
glm.probs = predict(glm.fit, Default[-train, ], type = "response")
glm.pred[glm.probs > 0.5] = "Yes"
mean(glm.pred != Default[-train, ]$default) ##iv

[1] 0.0254

I get a test error rate of 0.0254.

3. Repeat the process in (b) three times, using three different splits of the observations into a training set and a validation set. Comment on the results obtained.

train = sample(dim(Default)[1], dim(Default)[1]/2) ## i
glm.fit = glm(default ~ income + balance, data = Default, family = binomial, subset = train) ## ii
## iii
glm.pred = rep("No", dim(Default)[1]/2)
glm.probs = predict(glm.fit, Default[-train, ], type = "response")
glm.pred[glm.probs > 0.5] = "Yes"
mean(glm.pred != Default[-train, ]$default) ##iv

[1] 0.0274

train = sample(dim(Default)[1], dim(Default)[1]/2) ## i
glm.fit = glm(default ~ income + balance, data = Default, family = binomial, subset = train) ## ii
## iii
glm.pred = rep("No", dim(Default)[1]/2)
glm.probs = predict(glm.fit, Default[-train, ], type = "response")
glm.pred[glm.probs > 0.5] = "Yes"
mean(glm.pred != Default[-train, ]$default) ##iv

[1] 0.0244

train = sample(dim(Default)[1], dim(Default)[1]/2) ## i
glm.fit = glm(default ~ income + balance, data = Default, family = binomial, subset = train) ## ii
## iii
glm.pred = rep("No", dim(Default)[1]/2)
glm.probs = predict(glm.fit, Default[-train, ], type = "response")
glm.pred[glm.probs > 0.5] = "Yes"
mean(glm.pred != Default[-train, ]$default) ##iv

[1] 0.0244

Looks like the test error rate hovers around 2.5 percent.

4. Now consider a logistic regression model that predicts the probability of default using income, balance, and a dummy variable for student. Estimate the test error for this model using the validation set approach. Comment on whether or not including a dummy variable for student leads to a reduction in the test error rate.

train = sample(dim(Default)[1], dim(Default)[1]/2)
glm.fit = glm(default ~ income + balance + student, data = Default, family = binomial, 
    subset = train)
glm.pred = rep("No", dim(Default)[1]/2)
glm.probs = predict(glm.fit, Default[-train, ], type = "response")
glm.pred[glm.probs > 0.5] = "Yes"
mean(glm.pred != Default[-train, ]$default)

[1] 0.0278

It doesn’t appear adding the student dummy variable leads to a reduction in the test error rate.