hw5
Bella Lin
2024-03-23
2. For parts (a) through (c), indicate which of i. through iv. is correct. Justify your answer.
(a) The lasso, relative to least squares, is:
i. More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
ii. More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.
iii. Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
iv. Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.
iii is the correct statement for the Lasso technique. The Lasso can shrink some coefficients to exactly zero, makes the model less flexible compared to least squares. The reduction in flexibility (increase in bias) can lead to improved prediction accuracy when the decrease in variance outweighs the increase in bias.
(b) Repeat (a) for ridge regression relative to least squares.
iii is the correct statement. Ridge regression improves over least squares by addressing the bias-variance trade-off, which is similar to Lasso. However, instead of enforcing exact zero coefficients, it shrinks the coefficients toward zero by adding a penalty term to the least squares objective function, which increase in bias and it is typically less than the decrease in variance (due to the shrinking of coefficients). Therefore, the overall prediction accuracy improves when the increase in bias is less than the decrease in variance.
(c) Repeat (a) for non-linear methods relative to least squares.
ii is the correct statement. Non-linear methods are more flexible than linear models. When using non-linear methods, the increase in variance may be less than the decrease in bias, therefore the overall prediction accuracy improves as the decrease in bias outweighs the increase in variance.
9. In this exercise, we will predict the number of applications received using the other variables in the College data set.
(a) Split the data set into a training set and a test set.
set.seed(123)
split <- sample(nrow(College), 0.7 * nrow(College), replace = F) # 70/30 split
train <- College[split, ]
test <- College[split, ](b) Fit a linear model using least squares on the training set, and report the test error obtained.
b1 <- lm(Apps ~ ., data = train)
summary(b1)##
## Call:
## lm(formula = Apps ~ ., data = train)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3097.8 -455.8 -46.5 343.8 6452.5
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -310.17331 481.30075 -0.644 0.519566
## PrivateYes -681.96465 164.08211 -4.156 3.78e-05 ***
## Accept 1.22130 0.05921 20.626 < 2e-16 ***
## Enroll 0.08046 0.21794 0.369 0.712155
## Top10perc 49.33503 6.18296 7.979 9.31e-15 ***
## Top25perc -16.11744 5.02717 -3.206 0.001428 **
## F.Undergrad 0.02284 0.03985 0.573 0.566831
## P.Undergrad 0.03541 0.03529 1.003 0.316139
## Outstate -0.05446 0.02132 -2.555 0.010910 *
## Room.Board 0.18967 0.05275 3.596 0.000354 ***
## Books 0.21366 0.28099 0.760 0.447381
## Personal -0.03685 0.07279 -0.506 0.612876
## PhD -6.00401 5.34580 -1.123 0.261897
## Terminal -5.01712 5.77787 -0.868 0.385609
## S.F.Ratio -2.18927 14.83898 -0.148 0.882766
## perc.alumni -8.01836 4.67330 -1.716 0.086792 .
## Expend 0.07614 0.01340 5.681 2.23e-08 ***
## Grad.Rate 10.63461 3.38228 3.144 0.001760 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 992.3 on 525 degrees of freedom
## Multiple R-squared: 0.9175, Adjusted R-squared: 0.9148
## F-statistic: 343.2 on 17 and 525 DF, p-value: < 2.2e-16b_pred1 <- predict(b1, newdata = test)
test_error <- mean((test$Apps - b_pred1)^2)
test_error## [1] 952058.5(c) Fit a ridge regression model on the training set, with λ chosen by cross-validation. Report the test error obtained.
set.seed(123)
x_train <- model.matrix(Apps ~ ., data = train)[,-1]
y_train <- train$Apps
cv_ridge <- cv.glmnet(x_train, y_train, alpha = 0)
c1 <- glmnet(x_train, y_train, alpha = 0, lambda = cv_ridge$lambda.min)
x_test <- model.matrix(Apps ~ ., data = test)[,-1]
c_pred1 <- predict(c1, s = cv_ridge$lambda.min, newx = x_test)
ridge_error <- mean((test$Apps - c_pred1)^2)
ridge_error## [1] 1075434(d) Fit a lasso model on the training set, with λ chosen by cross- validation. Report the test error obtained, along with the number of non-zero coefficient estimates.
set.seed(123)
cv_lasso <- cv.glmnet(x_train, y_train, alpha = 1)
d1 <- glmnet(x_train, y_train, alpha = 1, lambda = cv_lasso$lambda.min)
d_pred1 <- predict(d1, s = cv_lasso$lambda.min, newx = x_test)
lasso_error <- mean((test$Apps - d_pred1)^2)
lasso_error## [1] 953582.4lasso_coeff <- sum(coef(c1, s = cv_lasso$lambda.min) != 0)
lasso_coeff## [1] 18(g) Comment on the results obtained. How accurately can we predict the number of college applications received? Is there much difference among the test errors resulting from these five approaches?
barplot(c(test_error, ridge_error, lasso_error), col="#CCCCFF", xlab="Regression Methods", ylab="Test Error", main="Test Errors for All Methods", names.arg=c("Linear", "Ridge", "Lasso"))
The linear model has a test error of 952058.5. Ridge regression has the highest test error among the three methods, with a value of 1075434. Lasso regression has a slightly higher test error compared to the linear model, with a value of 953582.4.
linear_r <- summary(b1)$r.squared
linear_r## [1] 0.9174532ridge_r <- 1 - mean((test$Apps - c_pred1)^2) / var(test$Apps)
ridge_r## [1] 0.9069279lasso_r <- 1 - mean((test$Apps - d_pred1)^2) / var(test$Apps)
lasso_r## [1] 0.9174733barplot(c(linear_r, ridge_r, lasso_r), col="#CCCCFF", xlab="Regression Methods", ylab="Test R-Squared", main="Test R-Squared for All Methods", names.arg=c("Linear", "Ridge", "Lasso"))
From the results above, the linear and lasso regression models seem to have very similar R-squared values, indicating comparable predictive performance, while the ridge regression model has a slightly lower R-squared value. Compare both test error and r square, linear model perform slightly better than the lasso regression model in predicting the number of college applications received.