Wrong: More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance. Since Lasso is more restrictive than OLS since it is based on two optimization problems it is much less flexible than OLS.
Wrong: More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias. Since Lasso is more restrictive than OLS since it is based on two optimization problems it is much less flexible than OLS.
Correct: Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
Wrong: Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias. Since Lasso is more restrictive than OLS since it is based on two optimization problems it is much more likely to have lower variance compared to OLS.
Ridge relative to the least squares model
Wrong: More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance. Since Ridge is more restrictive than OLS since it is based on two optimization problems it is much less flexible than OLS.
Wrong: More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias. Since Ridge is more restrictive than OLS since it is based on two optimization problems it is much less flexible than OLS.
Correct: Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance
Wrong: Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias. Since Ridge is more restrictive than OLS since it is based on two optimization problems it is much more likely to have lower variance compared to OLS.
Nonlinear models relative to the least squares model
Wrong: More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance. Since non-linear models are more flexible than OLS and are prone to over fit the data they are more likely to have higher variance.
Correct: More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias
Wrong: Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance. Nonlinear models are more flexible.
Wrong: Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias. Nonlinear models are more flexible.
9
Using the rsample package in the tinymodels we can achieve a stratified split, but in this situation since the response is numerical a random split works just as fine.
library(ISLR)library(rsample)library(caret)
Warning: package 'caret' was built under R version 4.5.2
Loading required package: ggplot2
Loading required package: lattice
Attaching package: 'caret'
The following object is masked from 'package:rsample':
calibration
data_split <-initial_split(College, prop=.75, strata=Apps)train_data <-training(data_split)test_data <-testing(data_split)#cross validation set up cross_validation<-trainControl(method='cv', number=10)#Least squares model_glm <-train( Apps ~., data=train_data,method ="glm", family ='gaussian',trControl = cross_validation,preProcess =c("center","scale"))#Ridge Regressionridge_params <-expand.grid(alpha =0, #for lasso it is 1lambda =c(.5,1,3, 5,10,50,100,500,1000))model_ridge <-train( Apps ~.,data=train_data,method='glmnet',tuneGrid = ridge_params,trControl = cross_validation,preProcess =c("center","scale"))#Lasso Regressionlasso_params <-expand.grid(alpha =1, #for lasso it is 1lambda =c(.5,1,3, 5,10,50,100,500,1000))model_lasso <-train( Apps ~.,data=train_data,method='glmnet',tuneGrid = lasso_params,trControl = cross_validation,preProcess =c("center","scale"))#PCRpcr_params <-expand.grid(ncomp=1:15)model_pcr<-train( Apps~., data=train_data, method="pcr",tuneGrid=pcr_params,trControl =cross_validation,#needed sine the predictors are compared for variance, cannot have year dsitance overashadow smaller metrics-Unsupervised so does not use the response variablepreProcess =c("center","scale"))#PLS pls_params<-expand.grid(ncomp=1:15)model_pls<-train( Apps~.,data=train_data, method="pls",tuneGrid=pls_params,trControl=cross_validation,#same as pcr it uses distanc eso needs to be standardized-Supervised so also uses the responsepreProcess =c("center","scale"))
With all of the models performing relatively similarly, I would chose the OLS model, since interpreting the coefficients is easier and more predictable.
crim zn indus nox
Min. : 0.00632 Min. : 0.00 Min. : 0.46 Min. :0.3850
1st Qu.: 0.08205 1st Qu.: 0.00 1st Qu.: 5.19 1st Qu.:0.4490
Median : 0.25651 Median : 0.00 Median : 9.69 Median :0.5380
Mean : 3.61352 Mean : 11.36 Mean :11.14 Mean :0.5547
3rd Qu.: 3.67708 3rd Qu.: 12.50 3rd Qu.:18.10 3rd Qu.:0.6240
Max. :88.97620 Max. :100.00 Max. :27.74 Max. :0.8710
rm age dis tax
Min. :3.561 Min. : 2.90 Min. : 1.130 Min. :187.0
1st Qu.:5.886 1st Qu.: 45.02 1st Qu.: 2.100 1st Qu.:279.0
Median :6.208 Median : 77.50 Median : 3.207 Median :330.0
Mean :6.285 Mean : 68.57 Mean : 3.795 Mean :408.2
3rd Qu.:6.623 3rd Qu.: 94.08 3rd Qu.: 5.188 3rd Qu.:666.0
Max. :8.780 Max. :100.00 Max. :12.127 Max. :711.0
ptratio black lstat medv chas_factor
Min. :12.60 Min. : 0.32 Min. : 1.73 Min. : 5.00 0:471
1st Qu.:17.40 1st Qu.:375.38 1st Qu.: 6.95 1st Qu.:17.02 1: 35
Median :19.05 Median :391.44 Median :11.36 Median :21.20
Mean :18.46 Mean :356.67 Mean :12.65 Mean :22.53
3rd Qu.:20.20 3rd Qu.:396.23 3rd Qu.:16.95 3rd Qu.:25.00
Max. :22.00 Max. :396.90 Max. :37.97 Max. :50.00
#spit the data into test and train setdata_split_boston <-initial_split(clean_boston, prop=.75, strata=crim)train_data_boston <-training(data_split_boston)test_data_boston <-testing(data_split_boston)#cross validation set up cross_validation<-trainControl(method='cv', number=10)
best subset selection
Best subset selection highlighted an interesting quirk with the full model. The subset_pred value given was negative though crim cannot be negative. This quirk highlights the high skew faced by the crim column. From the ggpairs of the before and after of the crim column we can see that applying a logarithmic transformation better normalizes the response.
library(leaps)library(plotly)
Warning: package 'plotly' was built under R version 4.5.2
Attaching package: 'plotly'
The following object is masked from 'package:MASS':
select
The following object is masked from 'package:ggplot2':
last_plot
The following object is masked from 'package:stats':
filter
The following object is masked from 'package:graphics':
layout
library(GGally)predict_regsubsets <-function(object, newdata, id) { form <-as.formula(object$call[[2]]) mat <-model.matrix(form, newdata) coefi <-coef(object, id = id) xvars <-names(coefi) mat[, xvars] %*% coefi }model_best_subset <-regsubsets( crim ~ ., data = train_data_boston,method ="exhaustive", nvmax =10)cat('Best Subset Selection \n')
Re-runing the model with the transformed crim column results in much more stable outputs and estimations for the response column. The best susbset model selected the top 6 variable model as the top performer.
The following models were run using the same logarithmic operation performed to be able to normalize the response variable and handle the skew.
#Ridge Regressionridge_params <-expand.grid(alpha =0, #for lasso it is 1lambda =c(.001,.01,.1,.5,1))model_ridge_boston <-train(log(crim) ~.,data=train_data_boston,method='glmnet',tuneGrid = ridge_params,trControl = cross_validation,preProcess =c("center","scale"))#Lasso Regressionlasso_params <-expand.grid(alpha =1, #for lasso it is 1lambda =c(.001,.01,.1,.5,1))model_lasso_boston <-train(log(crim) ~.,data=train_data_boston,method='glmnet',tuneGrid = lasso_params,trControl = cross_validation,preProcess =c("center","scale"))pcr_params <-expand.grid(ncomp=1:10)model_pcr_boston<-train(log(crim) ~.,data=train_data_boston,method="pcr",tuneGrid=pcr_params,trControl =cross_validation,preProcess =c("center","scale"))
summary
All models performed relatively well and the extra precaution was made to transform the models back using the inverse operation of \(exp\) due to the \(log\) transformation applied to crim. Keeping with the law parsimony the model selected would be Best Subset Selection since it uses only 6 variables, and performed similarly to the Ridge,PCR and Lasso, who kept all variables.
