Assignment 5

Chapter 6

library(ISLR2)
library(glmnet)

## Warning: package 'glmnet' was built under R version 4.6.1

## Loading required package: Matrix

## Loaded glmnet 5.0

library(leaps)

## Warning: package 'leaps' was built under R version 4.6.1

library(pls)

## Warning: package 'pls' was built under R version 4.6.1

## 
## Attaching package: 'pls'

## The following object is masked from 'package:stats':
## 
##     loadings

Question 2

For parts (a) through (c), indicate which of i. through iv. is correct. Justify your answer.

More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.
Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.

The lasso, relative to least squares, is:

iii, The lasso is less flexible than OLS because the L1 penalty shrinks coefficients and can set some to zero. This increases bias but reduces variance. Therefore, prediction accuracy improves when the decrease in variance outweighs the increase in bias.

Ridge regression, relative to least squares, is:

iii, Ridge regression is also less flexible than OLS. The L2 penalty shrinks coefficients toward zero, increasing bias but reducing variance. Prediction accuracy improves when the variance reduction exceeds the added bias.

Non-linear methods, relative to least squares, is:

ii, Nonlinear methods are more flexible because they can model complex relationships. Flexibility reduces bias but increases variance. Prediction accuracy improves when the bias reduction outweighs the variance increase.

Question 9

In this exercise, we will predict the number of applications received using the other variables in the College data set.

Split the data set into a training set and a test set.

data(College)

set.seed(123)
train_idx <- sample(1:nrow(College), nrow(College) * 0.7) 
train_set <- College[train_idx, ]
test_set  <- College[-train_idx, ]

Fit a linear model using least squares on the training set, and report the test error obtained.

lm.fit <- lm(Apps ~ ., data = train_set)
lm.pred <- predict(lm.fit, test_set)
ols_mse <- mean((test_set$Apps - lm.pred)^2)
print(paste("OLS Test MSE:", ols_mse))

## [1] "OLS Test MSE: 1734840.97722585"

Fit a ridge regression model on the training set, with λ chosen by cross-validation. Report the test error obtained.

x_train <- model.matrix(Apps ~ ., data = train_set)[, -1]
y_train <- train_set$Apps
x_test  <- model.matrix(Apps ~ ., data = test_set)[, -1]
y_test  <- test_set$Apps


set.seed(123)
cv.ridge <- cv.glmnet(x_train, y_train, alpha = 0)
ridge.pred <- predict(cv.ridge, newx = x_test, s = cv.ridge$lambda.min)
ridge_mse <- mean((y_test - ridge.pred)^2)
print(paste("Ridge Test MSE:", ridge_mse))

## [1] "Ridge Test MSE: 2976365.04883116"

Cross‑validated ridge regression resulted in a test MSE of 2,976,365, which is substantially worse than OLS. Because the College data contain relatively few predictors and limited multicollinearity, shrinking the coefficients introduced additional bias without enough reduction in variance

Fit a lasso model on the training set, with λ chosen by cross validation. Report the test error obtained, along with the number of non-zero coefficient estimates.

set.seed(123)
cv.lasso <- cv.glmnet(x_train, y_train, alpha = 1)
lasso.pred <- predict(cv.lasso, newx = x_test, s = cv.lasso$lambda.min)
lasso_mse <- mean((y_test - lasso.pred)^2)
print(paste("Lasso Test MSE:", lasso_mse))

## [1] "Lasso Test MSE: 1730913.43621665"

lasso_coefs <- predict(cv.lasso, type = "coefficients", s = cv.lasso$lambda.min)
num_nonzero <- sum(lasso_coefs != 0) - 1 
print(paste("Number of active features:", num_nonzero))

## [1] "Number of active features: 16"

The lasso achieved a test MSE of 1,730,913, slightly outperforming OLS. It selected 16 non‑zero predictors, indicating that nearly every variable contributed useful information. Its test MSE was slightly lower than ordinary least squares.

Fit a PCR model on the training set, with M chosen by cross validation. Report the test error obtained, along with the value of M selected by cross-validation.

set.seed(123)
pcr.fit <- pcr(Apps ~ ., data = train_set, scale = TRUE, validation = "CV")
validationplot(pcr.fit, val.type = "MSEP")

best_m_pcr <- which.min(MSEP(pcr.fit)$val[1,1,]) -1

pcr.pred <- predict(pcr.fit, test_set, ncomp = best_m_pcr)
pcr_mse <- mean((test_set$Apps - pcr.pred)^2)
print(paste("PCR Test MSE:", pcr_mse))

## [1] "PCR Test MSE: 1853634.78296277"

print(best_m_pcr)

## 16 comps 
##       16

Fit a PLS model on the training set, with M chosen by cross validation. Report the test error obtained, along with the value of M selected by cross-validation.

set.seed(123)
pls.fit <- plsr(Apps ~ ., data = train_set, scale = TRUE, validation = "CV")
validationplot(pls.fit, val.type = "MSEP")

best_m_pls <- which.min(MSEP(pls.fit)$val[1,1,]) - 1

pls.pred <- predict(pls.fit, test_set, ncomp = best_m_pls)
pls_mse <- mean((test_set$Apps - pls.pred)^2)
print(paste("PLS Test MSE:", pls_mse))

## [1] "PLS Test MSE: 1756685.22839907"

print(paste("Selected M for PLS:", best_m_pls))

## [1] "Selected M for PLS: 11"

Comment on the results obtained. How accurately can we predict the number of college applications received? Is there much difference among the test errors resulting from these five approaches?

OLS, lasso, PCR, and PLS all produced similar test errors, while ridge regression performed noticeably worse. The dataset contains only 17 predictors and does not exhibit severe multicollinearity, so heavy regularization is unnecessary. The lasso achieved the lowest test MSE (1,730,913), although its improvement over OLS and PLS was relatively small. PCR and PLS achieved comparable performance after selecting the optimal number of components using cross-validation. Overall, all methods except ridge produced similar prediction accuracy, suggesting that the College data set does not require heavy regularization.

Question 11

We will now try to predict per capita crime rate in the Boston data set.

data(Boston)
Boston <- na.omit(Boston)

set.seed(42)
train_idx <- sample(1:nrow(Boston), nrow(Boston) * 0.7)
train_set <- Boston[train_idx, ]
test_set  <- Boston[-train_idx, ]

x_train <- model.matrix(crim ~ ., data = train_set)[, -1]
y_train <- train_set$crim
x_test  <- model.matrix(crim ~ ., data = test_set)[, -1]
y_test  <- test_set$crim

Try out some of the regression methods explored in this chapter, such as best subset selection, the lasso, ridge regression, and PCR. Present and discuss results for the approaches that you consider.

###Method 1

regfit.best <- regsubsets(crim ~ ., data = train_set, nvmax=ncol(Boston)-1)

predict.regsubsets <- function(object, newdata, id, ...) {
  form <- as.formula(object$call[[2]])
  mat <- model.matrix(form, newdata)
  coefi <- coef(object, id = id)
  xvars <- names(coefi)
  mat[, xvars] %*% coefi
}

num_predictors <- ncol(Boston) - 1
val.errors <- rep(NA, num_predictors)

for (i in 1:num_predictors) {
  pred <- predict.regsubsets(regfit.best, test_set, id = i)
  val.errors[i] <- mean((test_set$crim - pred)^2)
}

best_size <- which.min(val.errors)
best_subset_pred <- predict.regsubsets(regfit.best, test_set, id = best_size)
best_subset_mse  <- mean((test_set$crim - best_subset_pred)^2)

print(paste("Best Subset Chosen Size:", best_size))

## [1] "Best Subset Chosen Size: 4"

print(paste("Best Subset Test MSE:", best_subset_mse))

## [1] "Best Subset Test MSE: 23.3272631827274"

###Method 2

set.seed(42)
cv.ridge <- cv.glmnet(x_train, y_train, alpha = 0)
bestlam_ridge <- cv.ridge$lambda.min

ridge.pred <- predict(cv.ridge, s = bestlam_ridge, newx = x_test)
ridge_mse  <- mean((y_test - ridge.pred)^2)

print(paste("Ridge Optimal Lambda:", bestlam_ridge))

## [1] "Ridge Optimal Lambda: 0.553307675536775"

print(paste("Ridge Test MSE:", ridge_mse))

## [1] "Ridge Test MSE: 23.6476103355337"

###Method 3

set.seed(42)
cv.lasso <- cv.glmnet(x_train, y_train, alpha = 1)
bestlam_lasso <- cv.lasso$lambda.min

lasso.pred <- predict(cv.lasso, s = bestlam_lasso, newx = x_test)
lasso_mse  <- mean((y_test - lasso.pred)^2)

print(paste("Lasso Optimal Lambda:", bestlam_lasso))

## [1] "Lasso Optimal Lambda: 0.00516016590061771"

print(paste("Lasso Test MSE:", lasso_mse))

## [1] "Lasso Test MSE: 25.1665593871221"

###Method 4

set.seed(42)
pcr.fit <- pcr(crim ~ ., data = train_set, scale = TRUE, validation = "CV")

best_m_pcr <- which.min(MSEP(pcr.fit)$val[1,1,]) -1

pcr.pred <- predict(pcr.fit, test_set, ncomp = best_m_pcr)
pcr_mse  <- mean((test_set$crim - pcr.pred)^2)

print(paste("PCR Chosen Components (M):", best_m_pcr))

## [1] "PCR Chosen Components (M): 12"

print(paste("PCR Test MSE:", pcr_mse))

## [1] "PCR Test MSE: 25.3271898666761"

Propose a model (or set of models) that seem to perform well on this data set, and justify your answer. Make sure that you are evaluating model performance using validation set error, cross validation, or some other reasonable alternative, as opposed to using training error.

results_matrix <- data.frame(
  Model = c("Best Subset Selection", "Ridge Regression", "The Lasso", "PCR"),
  Test_MSE = c(best_subset_mse, ridge_mse, lasso_mse, pcr_mse)
)
print(results_matrix)

##                   Model Test_MSE
## 1 Best Subset Selection 23.32726
## 2      Ridge Regression 23.64761
## 3             The Lasso 25.16656
## 4                   PCR 25.32719

Based on validation error, best subset selection and ridge regression performed best. The lasso underperformed because the optimal penalty parameter was too small to meaningfully shrink coefficients. PCR required many components and did not outperform simpler methods.

Does your chosen model involve all of the features in the data set? Why or why not?

final_lasso <- glmnet(
    model.matrix(crim~.,Boston)[,-1],
    Boston$crim,
    alpha=1
)
predict(final_lasso, type = "coefficients", s = bestlam_lasso)

## 13 x 1 sparse Matrix of class "dgCMatrix"
##             s=0.005160166
## (Intercept) 13.2737863198
## zn           0.0447386828
## indus       -0.0600840707
## chas        -0.8100651455
## nox         -9.6471609099
## rm           0.6014038817
## age         -0.0001213135
## dis         -0.9860722532
## rad          0.6025191142
## tax         -0.0032315089
## ptratio     -0.2953459240
## lstat        0.1378907247
## medv        -0.2152328068

We compared five regression approaches—ordinary least squares, ridge regression, the lasso, principal components regression (PCR), and partial least squares (PLS)—for predicting college applications. OLS, lasso, PCR, and PLS all produced similar test errors, while ridge regression performed noticeably worse due to minimal benefit from coefficient shrinkage. The lasso achieved slightly lower test error than OLS while retaining nearly all predictors, indicating limited sparsity in this dataset. PCR and PLS required cross‑validated component selection and achieved comparable accuracy.

For the Boston crime data, best subset selection and ridge regression achieved the lowest test MSE values. The lasso performed worse, likely due to selecting a very small penalty parameter that failed to meaningfully shrink coefficients. PCR also underperformed relative to best subset and ridge. Based on validation error, best subset selection and ridge regression appear to be the strongest models for this dataset.

Assignment 5

Alexandra Holen

2026-07-10

Chapter 6

Question 2

Question 9

Question 11