Assignment #5

QUESTION 2

For parts (a) through (c), indicate which of i. through iv. is correct. Justify your answer.

1. The lasso, relative to least squares, is:

1. More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
2. More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.
3. Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
4. Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.

Statement iii is correct - lasso applies a penalty restricting model flexibility. Variance decreases and bias increases as a result.

2. Repeat (a) for ridge regression relative to least squares.

Statement iii is correct - lasso applies a penalty restricting model flexibility. Variance decreases and bias increases as a result.

3. Repeat (a) for non-linear methods relative to least squares.

Statement ii is correct - nonlinear models are usually more flexible than linear models. Variance increases and bias decreases as a result.

QUESTION 9

In this exercise, we will predict the number of applications received using the other variables in the College data set.

1. Split the data set into a training set and a test set.

library(ISLR)

## 
## Attaching package: 'ISLR'

## The following objects are masked from 'package:ISLR2':
## 
##     Auto, Credit

library(MASS)
library(leaps)
library(glmnet)

## Loading required package: Matrix

## 
## Attaching package: 'Matrix'

## The following objects are masked from 'package:tidyr':
## 
##     expand, pack, unpack

## Loaded glmnet 4.1-8

library(pls)

## 
## Attaching package: 'pls'

## The following object is masked from 'package:stats':
## 
##     loadings

library(caTools)
set.seed(15)
college_data = sample(777, 500, replace=FALSE)
train = College[college_data,]
test = College[-college_data,]

2. Fit a linear model using least squares on the training set, and report the test error obtained.

lm.college = lm(Apps~., train)
predict.college = predict(lm.college, test)
error.lm.college = mean((predict.college-test$Apps)^2)
mean((predict.college-test$Apps)^2)

## [1] 1593075

3. Fit a ridge regression model on the training set, with λ chosen by cross-validation. Report the test error obtained.

grid = 10^seq(10,-2, length=100)
train = sample(777, 500, replace=FALSE)
test = -train
x = model.matrix(Apps~., College)[,-1]
y = College$Apps

ridge.mod = glmnet(x[train,],y[train], alpha=0, lambda=grid)
cv.out = cv.glmnet(x[train,],y[train], alpha=0)
bestlam = cv.out$lambda.min
ridge.pred = predict(ridge.mod, s=bestlam, newx=x[test,])
err.ridge = mean((ridge.pred-y[test])^2)
mean((ridge.pred-y[test])^2)

## [1] 2727719

4. Fit a lasso model on the training set, with λ chosen by cross- validation. Report the test error obtained, along with the num- ber of non-zero coefficient estimates.

lasso.mod = glmnet(x[train,],y[train], alpha=1, lambda=grid)
cv.out.2 = cv.glmnet(x[train,],y[train], alpha=1)
bestlam.2 = cv.out.2$lambda.min
lasso.pred = predict(lasso.mod,s=bestlam.2,newx=x[test,])
err.lasso = mean((lasso.pred-y[test])^2)
mean((lasso.pred-y[test])^2)

## [1] 1665872

coef.lasso = predict(lasso.mod, type="coefficients", s=bestlam.2, newx=x[test,])
length(coef.lasso[coef.lasso!=0])

## [1] 14

7. Comment on the results obtained. How accurately can we predict the number of college applications received? Is there much difference among the test errors resulting from these five approaches?

The ridge regression model’s error was the largest at 2727719. The other two models’ errors were similar, but linear model’s error was the smallest at 1593075.