Playing with Variable Selection and LASSO

source("Smart LASSO.R")
cores = detectCores()

First, simulating on any dataset containing failures (naive)

Last week, we discussed how model performance is affected when we allow our simulation process to throw out training sets (and resample) when the training set didn’t include enough bank failures that we wanted it to. The baseline for model performance is the situation where we simluate a model n times (100 is quite stable, so I’ve been going with that) and collect the test stats, but where we use all training sets which contain at least 1 failure into the model. From previous weeks, we have a pretty good idea of what that looks like.

(Note the choice of 2008Q2, sample size of 100, and number of simulations = 100, which will remain the case throughout.

run_models(100, "2008-06-30", 100, "logreg")

     in_sample_mse  out_of_sample_mse     in_sample_rmse out_of_sample_rmse      in_sample_auc  out_of_sample_auc 
        0.06320000         0.07238642         0.24471943         0.26838221         0.89054422         0.80045269 

run_models(100, "2008-06-30", 100, "lasso")

     in_sample_mse  out_of_sample_mse     in_sample_rmse out_of_sample_rmse      in_sample_auc  out_of_sample_auc 
        0.07360000         0.06226483         0.26871622         0.24940062         0.85637157         0.79371096 

run_models_each_variable(100, "2009-06-30", 100)

As we can see – based on the training set size, in-sample MSE on the full model is low, out-of-sample MSE on the full model is about the same as the one-variable MSEs.

Second, simulating on any dataset containing at least m failures (less naive; m = 5)

Some improvement can hopefully be made by discriminating our training sets a bit. The code below adds two new parameters to my modeling function that outputs the graph of MSE: (1) the minimum number of failures a training set is allowed to contain (otherwise, it’s resampled), and (2) the ratio we apply between the number of failures in a training set and the number of predictors in the model. The ratio is what decides the lambda used for our LASSO model.

Let’s try m >= 5 for different training set sizes:

run_models_each_variable_lasso(100, "2008-06-30", 100, 5, 1)

run_models_each_variable_lasso(100, "2008-06-30", 500, 5, 1)

run_models_each_variable_lasso(100, "2008-06-30", 500, 20, 1)

A little disappointing, I have to say!

Nothing seems to lower MSE below this ~0.07 threshold.

Third, simulating on any dataset containing at least m failures divided by a certain number of predictors, m

run_models_each_variable_lasso(100, "2008-06-30", 100, 5, 2)

run_models_each_variable_lasso(100, "2008-06-30", 100, 5, 4)

run_models_each_variable_lasso(100, "2008-06-30", 100, 10, 3)

run_models_each_variable_lasso(100, "2008-06-30", 100, 10, 4)

This was a somewhat suprising result until I understood what was going on. These models run into a situation where many of the training sets have only ten failures, and the LASSO is being forced to drop 2-3 variables and predict badly on the training set. The remaining variables don’t explain much about the outcome than each one would individually. What I can’t figure out – and I double-checked the method I’m building – is why the test data universally outperforms training data. “Underfitting?”