Question 3

(a) Explain how k-fold cross-validation is implemented.

K-fold CV is implemented by randomly dividing subsets of observations into groups, called k folds. The two groups have relatively the same number of observations, but the two subsets are treated differently. The first ‘fold’ is a validation set, and the test is fit on the second set, which contains k-1, or the observations not in the first fold. The MSE is computed on the second fold and this is repreated “k” times, hence why the code can contain k=1, k=2, and so on.

(b) What are the advantages and disadvantages of k-fold crossvalidation relative to: i. The validation set approach? ii. LOOCV?

the advantages of the validation set approach are that it takes a simple approach and randomly selects subsets to use as validation and training sets, similar to other models. The disadvantages of the validation set approach is that is that the small subset of observations can cause the model to overestimate the test error rate.
The advantages of LOOCV are that it is less bias when choosing the subset of observations, since it only subtracts one. Because it has a higher sample size, the model does not overestimate the test error as much as the validation set approach. However, a disadvantage of LOOCV is that the model has to refit n number of times, making it more complicated.

Question 5

(a) Fit a logistic regression model that uses income and balance to predict default

library(ISLR2)
default <- Default
glm <- glm(default ~ balance + income, family = binomial, data = default)
summary(glm)

## 
## Call:
## glm(formula = default ~ balance + income, family = binomial, 
##     data = default)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.4725  -0.1444  -0.0574  -0.0211   3.7245  
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -1.154e+01  4.348e-01 -26.545  < 2e-16 ***
## balance      5.647e-03  2.274e-04  24.836  < 2e-16 ***
## income       2.081e-05  4.985e-06   4.174 2.99e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1579.0  on 9997  degrees of freedom
## AIC: 1585
## 
## Number of Fisher Scoring iterations: 8

(b) Using the validation set approach, estimate the test error of this model.

i. split the sample set into a training set and a validation set.

set.seed(1)
train <- sample(10000, 5000)
test = default[-train, ]
train <- default[train, ]

ii. Fit a multiple logistic regression model using only the training observations.

glmfit <- glm(default ~ income + balance, data = train, family = "binomial")
summary(glmfit)

## 
## Call:
## glm(formula = default ~ income + balance, family = "binomial", 
##     data = train)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.5830  -0.1428  -0.0573  -0.0213   3.3395  
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -1.194e+01  6.178e-01 -19.333  < 2e-16 ***
## income       3.262e-05  7.024e-06   4.644 3.41e-06 ***
## balance      5.689e-03  3.158e-04  18.014  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1523.8  on 4999  degrees of freedom
## Residual deviance:  803.3  on 4997  degrees of freedom
## AIC: 809.3
## 
## Number of Fisher Scoring iterations: 8

#iii. Obtain a prediction of default status for each individual in the validation set by computing the posterior probability of default for that individual, and classifying the individual to the default category if the posterior probability is greater than 0.5.

glmprob <- predict(glmfit, data = test, type = "response")
glmpred <- rep("No", length(glmprob))
glmpred[glmprob > 0.5] <- "Yes"

iv. Compute the validation set error, which is the fraction of the observations in the validation set that are misclassified.

(mean(glmpred!= test$default))*100

## [1] 4.76

(c) Repeat the process in (b) three times, using three different splits of the observations into a training set and a validation set. Comment on the results obtained.

set.seed(2)
train <- sample(10000, 7000)
test = default[-train, ]
train <- default[train, ]
glmfit <- glm(default ~ income + balance, data = train, family = "binomial")
glmprob <- predict(glmfit, data = test, type = "response")
glmpred <- rep("No", length(glmprob))
glmpred[glmprob > 0.5] <- "Yes"
(mean(glmpred!= test$default))*100

## [1] 4.328571

set.seed(3)
train <- sample(10000, 8000)
test = default[-train, ]
train <- default[train, ]
glmfit <- glm(default ~ income + balance, data = train, family = "binomial")
glmprob <- predict(glmfit, data = test, type = "response")
glmpred <- rep("No", length(glmprob))
glmpred[glmprob > 0.5] <- "Yes"
(mean(glmpred!= test$default))*100

## [1] 4.2125

set.seed(4)
train <- sample(10000, 9000)
test = default[-train, ]
train <- default[train, ]
glmfit <- glm(default ~ income + balance, data = train, family = "binomial")
glmprob <- predict(glmfit, data = test, type = "response")
glmpred <- rep("No", length(glmprob))
glmpred[glmprob > 0.5] <- "Yes"
(mean(glmpred!= test$default))*100

## [1] 4.111111

# All the testing errors are different, but still very close to one another. The testing error gets slightly smaller as the ratio gets larger.

##(d) Now consider a logistic regression model that predicts the probability of default using income, balance, and a dummy variable for student. Estimate the test error for this model using the validation set approach.

set.seed(1)
train <- sample(10000, 5000)
test = default[-train, ]
train <- default[train, ]
glmfit <- glm(default ~ income + balance + student, data = train, family = "binomial")
summary(glmfit)

## 
## Call:
## glm(formula = default ~ income + balance + student, family = "binomial", 
##     data = train)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.5823  -0.1419  -0.0554  -0.0210   3.3961  
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -1.134e+01  6.937e-01 -16.346   <2e-16 ***
## income       1.686e-05  1.122e-05   1.502   0.1331    
## balance      5.767e-03  3.213e-04  17.947   <2e-16 ***
## studentYes  -5.992e-01  3.324e-01  -1.803   0.0715 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1523.77  on 4999  degrees of freedom
## Residual deviance:  800.07  on 4996  degrees of freedom
## AIC: 808.07
## 
## Number of Fisher Scoring iterations: 8

glmprob <- predict(glmfit, data = test, type = "response")
glmpred <- rep("No", length(glmprob))
glmpred[glmprob > 0.5] <- "Yes"
(mean(glmpred!= test$default))*100

## [1] 4.84

## including the dummy variable does not really change the test error rate.

Number 6

(a) Using the summary() and glm() functions, determine the estimated standard errors for the coefficients associated with income and balance in a multiple logistic regression model that uses both predictors.

glmfit <- glm(default ~ income + balance, data = default, family = "binomial")
summary(glmfit)

## 
## Call:
## glm(formula = default ~ income + balance, family = "binomial", 
##     data = default)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.4725  -0.1444  -0.0574  -0.0211   3.7245  
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -1.154e+01  4.348e-01 -26.545  < 2e-16 ***
## income       2.081e-05  4.985e-06   4.174 2.99e-05 ***
## balance      5.647e-03  2.274e-04  24.836  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1579.0  on 9997  degrees of freedom
## AIC: 1585
## 
## Number of Fisher Scoring iterations: 8

(b) Write a function, boot.fn(), that takes as input the Default data set as well as an index of the observations, and that outputs the coefficient estimates for income and balance in the multiple logistic regression model.

(c) Use the boot() function together with your boot.fn() function to estimate the standard errors of the logistic regression coefficients for income and balance.

MS_4373_Assignment_4

Mia Brito

2022-11-28