library(ISLR)
library(glmnet)
## Loading required package: Matrix
## Loaded glmnet 4.1-8

Question 2

##2. For parts (a) through (c), indicate which of i. through iv. is correct. Justify your answer.

  1. The lasso, relative to least squares, is:
  1. More flexible and hence will give improved prediction ac- curacy when its increase in bias is less than its decrease in variance.
  2. More flexible and hence will give improved prediction accu- racy when its increase in variance is less than its decrease in bias.
  3. Less flexible and hence will give improved prediction accu- racy when its increase in bias is less than its decrease in variance.
  4. Less flexible and hence will give improved prediction accu- racy when its increase in variance is less than its decrease in bias.

Question 2(a)

The correct answer is iii. The lasso method introduces a penalty on the size of the coefficients. The reduction in model complexity will lead to an increase in bias and a decrease in the variance.

Question 2(b)

The correct answer is iii. similar to lasso, it also introduces a penalty to the coefficients of the predictors.

Question 2(c)

The correct answer is i. Non-linear methods are more flexible than linear methods by nature.

Question 9

In this exercise, we will predict the number of applications received using the other variables in the College data set.

data(College)

Question 9(a)

  1. Split the data set into a training set and a test set.
set.seed(123)

training_size <- round(nrow(College) * 0.75)

training_indices <- sample(seq_len(nrow(College)), size = training_size)


College_train <- College[training_indices, ]


College_test <- College[-training_indices, ]

Question 9(b)

(b) Fit a linear model using least squares on the training set, and report the test error obtained.

According to my results, I got Test Mean Squared Error of 1790402.7

college_lm <- lm(Apps ~ ., data = College_train)

college_test_predictions <- predict(college_lm, College_test)

test_mse <- mean((College_test$Apps - college_test_predictions)^2)
print(paste("Test Mean Squared Error:", test_mse))
## [1] "Test Mean Squared Error: 1790402.69639585"

(c) Fit a ridge regression model on the training set, with λ chosen by cross-validation. Report the test error obtained.

My Ridge Regression Test Mean Squared Error (MSE) of approximately 3,266,590.62, which is very high, It suggests that the performant might not be as effective in predicting the number of applications.

x_train <- model.matrix(Apps ~ . - 1, data = College_train)  # -1 to omit intercept
y_train <- College_train$Apps

x_test <- model.matrix(Apps ~ . - 1, data = College_test)
y_test <- College_test$Apps


set.seed(123) 
cv_ridge <- cv.glmnet(x_train, y_train, alpha = 0, nfolds = 10)  # alpha = 0 for ridge

plot(cv_ridge)

ridge_pred <- predict(cv_ridge, s = "lambda.min", newx = x_test)


ridge_test_mse <- mean((y_test - ridge_pred)^2)
print(paste("Ridge Regression Test Mean Squared Error:", ridge_test_mse))
## [1] "Ridge Regression Test Mean Squared Error: 3266590.61524968"

(d) Fit a lasso model on the training set, with λ chosen by cross- validation. Report the test error obtained, along with the number of non-zero coefficient estimates.

According to my result, my Lasso Regression Test Mean Squared Error is 1881128.3 and Number of Non-Zero Coefficients is 15

x_train <- model.matrix(Apps ~ . - 1, data = College_train)  # -1 to omit intercept
y_train <- College_train$Apps

x_test <- model.matrix(Apps ~ . - 1, data = College_test)
y_test <- College_test$Apps


set.seed(123)  
cv_lasso <- cv.glmnet(x_train, y_train, alpha = 1, nfolds = 10)

plot(cv_lasso)

lasso_pred <- predict(cv_lasso, s = "lambda.min", newx = x_test)

lasso_test_mse <- mean((y_test - lasso_pred)^2)
print(paste("Lasso Regression Test Mean Squared Error:", lasso_test_mse))
## [1] "Lasso Regression Test Mean Squared Error: 1881128.30942324"
lasso_coef <- coef(cv_lasso, s = "lambda.min")
non_zero_coefs <- sum(lasso_coef != 0)
print(paste("Number of Non-Zero Coefficients:", non_zero_coefs))
## [1] "Number of Non-Zero Coefficients: 15"

(g) Comment on the results obtained. How accurately can we predict the number of college applications received? Is there much difference among the test errors resulting from these five ap- proaches?

according to my results here is what I found:

  1. Linear Model: 1,790,402.7
  2. Ridge Regression: 3,266,590.62
  3. Lasso Regression: 1,881,128.31

I can see that my linear model worked the best, then Lasso took the second place and finally ridge was the worst one.