homework04

library(ISLR)
library(ISLR2)

## 
## Attaching package: 'ISLR2'

## The following objects are masked from 'package:ISLR':
## 
##     Auto, Credit

Question 3

(a) Explain how k-fold cross-validation is implemented.

basically, k-fold Cross-Validation is a resampling procedure used to evaluate machine learning models on a limited data sample. This procedure has has a single parameter k that refers to the number of groups that a give data sample is to be split into.

(b) What are the advantages and disadvantages of k-fold crossvalidation relative to:

1. Compared to the Validation Set Approach

Advantages:

1. It uses more of the data for both training and testing, leading to more reliable estimate model perfromance

2. It uses all the data for both training and validation

Disadvantages:

1. It can be more computationally expensive than the validation set approach, especially if k is large of the dataset is very big.

2. Compared to the Leave-One-Out Cross-Validation (LOOCV)

Advantages: 1. For large datasets, k-fold cross-validation is less computationally intensive than LOOCV, it requires fewer model fits.

1. less variance in model performance estimate

Disadvantages:

1. Bias variance trade-off
2. More arbitrary

Question 5

5. In Chapter 4, we used logistic regression to predict the probability of default using income and balance on the Default data set. We will now estimate the test error of this logistic regression model using the validation set approach. Do not forget to set a random seed before beginning your analysis.

head(Default)

##   default student   balance    income
## 1      No      No  729.5265 44361.625
## 2      No     Yes  817.1804 12106.135
## 3      No      No 1073.5492 31767.139
## 4      No      No  529.2506 35704.494
## 5      No      No  785.6559 38463.496
## 6      No     Yes  919.5885  7491.559

data("Default")

(a) Fit a logistic regression model that uses income and balance to predict default.

set.seed(123)

# Fit a logistic regression model
model <- glm(default ~ income + balance, data = Default, family = "binomial")

i. Split the sample set into a training set and a validation set.

set.seed(123)

train_indices <- sample(1:nrow(Default), nrow(Default) * 0.7)

train_set <- Default[train_indices, ]
validation_set <- Default[-train_indices, ]

ii. Fit a multiple logistic regression model using only the training observations.

model <- glm(default ~ income + balance, data = train_set, family = "binomial")

iii. Obtain a prediction of default status for each individual in the validation set by computing the posterior probability of default for that individual, and classifying the individual to the default category if the posterior probability is greater than 0.5.

predictions <- predict(model, validation_set, type = "response")

predicted_class <- ifelse(predictions > 0.5, "Yes", "No")

iv. Compute the validation set error, which is the fraction of the observations in the validation set that are misclassified.

misclassified <- predicted_class != validation_set$default
validation_set_error <- sum(misclassified) / length(misclassified)


print(validation_set_error)

## [1] 0.02633333

9. We will now consider the Boston housing data set, from the ISLR2 library.

data(Boston)

(a) Based on this data set, provide an estimate for the population mean of medv. Call this estimate μˆ.

mu_hat <- mean(Boston$medv)
print(mu_hat)

## [1] 22.53281

(b) Provide an estimate of the standard error of μˆ. Interpret this result.

Hint: We can compute the standard error of the sample mean by dividing the sample standard deviation by the square root of the number of observations.

s <- sd(Boston$medv)

n <- length(Boston$medv)

se_mu_hat <- s / sqrt(n)

print(se_mu_hat)

## [1] 0.4088611

(c) Now estimate the standard error of μˆ using the bootstrap. How does this compare to your answer from (b)?

Bootstrap estimated the standard error of μ^ as 0.4118401, where as standard error of sample mean is 0.4088611.

bootstrap_replicate <- function(data) {
  bootstrap_sample <- sample(data, length(data), replace = TRUE)
  return(mean(bootstrap_sample))
}


n_replicates <- 1000


bootstrap_means <- replicate(n_replicates, bootstrap_replicate(Boston$medv))


se_mu_hat_bootstrap <- sd(bootstrap_means)


print(se_mu_hat_bootstrap)

## [1] 0.4118401

(d) Based on your bootstrap estimate from (c), provide a 95 % con- fidence interval for the mean of medv. Compare it to the results obtained using t.test(Boston$medv).

Based on my test, the CE is 21.72953 - 23.33608 which is very similar to the bootstrp estimate 21.70913 - 23.35649.

ci_bootstrap_formula <- c(mu_hat - 2 * se_mu_hat_bootstrap, mu_hat + 2 * se_mu_hat_bootstrap)
print(ci_bootstrap_formula)

## [1] 21.70913 23.35649

ci_bootstrap_percentile <- quantile(bootstrap_means, c(0.025, 0.975))
print(ci_bootstrap_percentile)

##     2.5%    97.5% 
## 21.75349 23.34298

t_test_result <- t.test(Boston$medv)
print(t_test_result$conf.int)

## [1] 21.72953 23.33608
## attr(,"conf.level")
## [1] 0.95

(e) Based on this data set, provide an estimate, μˆmed, for the median value of medv in the population.

mu_hat_med <- median(Boston$medv)

print(mu_hat_med)

## [1] 21.2

(f) We now would like to estimate the standard error ofμˆmed. Unfortunately, there is no simple formula for computing the standard error of the median. Instead, estimate the standard error of the median using the bootstrap. Comment on your findings.

According to my results, I see that my standard error is 0.3723012 which means if I wanted to repeatedly sample from the population and calculate the median medv for each sample, the distribution of those medians would have a standard deviation of around $372.30.

bootstrap_median <- function(data) {
  bootstrap_sample <- sample(data, length(data), replace = TRUE)
  return(median(bootstrap_sample))
}

n_replicates <- 1000

bootstrap_medians <- replicate(n_replicates, bootstrap_median(Boston$medv))

se_mu_hat_med_bootstrap <- sd(bootstrap_medians)


print(se_mu_hat_med_bootstrap)

## [1] 0.3723012

(g) Based on this data set, provide an estimate for the tenth per- centile of medv in Boston census tracts. Call this quantity μˆ0.1. (You can use the quantile() function.)

mu_hat_0_1 <- quantile(Boston$medv, 0.1)

print(mu_hat_0_1)

##   10% 
## 12.75

Question 9(h)

My estimated standard error of the tench percentile is 0.4969654. This value shows us the uncertainty around the estimate of the 10th percentile of the median value of owner-occupied homes due to sampling variability.

bootstrap_tenth_percentile <- function(data) {
  bootstrap_sample <- sample(data, length(data), replace = TRUE)
  return(quantile(bootstrap_sample, 0.1))
}

n_replicates <- 1000

bootstrap_tenth_percentiles <- replicate(n_replicates, bootstrap_tenth_percentile(Boston$medv))

se_mu_hat_0_1_bootstrap <- sd(bootstrap_tenth_percentiles)

print(se_mu_hat_0_1_bootstrap)

## [1] 0.4969654