More LASSO Experiments

As with last week, this is a deeper dive into seeing how hyperparamater changes in LASSO models can affect the test-statistics for predicting bank failures.

source("Each Variable Models.R")
source("Smart LASSO 2.R")
cores = detectCores()

Lambda tuning, given a training set contains at least one failure (MSE)

First, I’m trying out lambda-tuning by selecting the ratio between the number of failures in a selected training set (at least 1) n and the number of predictors used by the LASSO model. The model used must have that ratio be whatever is specified. Holding some other parameters constant, we see how changing the ratio changes our outcome:

• Date considered: Q4 2006
• Training set size: 100
• Number of simulations: 100 (quite stable)
• (Ignore the “first” parameter for now – it relates to how our optimal lambda is chosen. Right now, were using the same criterion we used last week – where the first lambda to satisfy the ratio constraint is chosen.)

run_models("1.1 Ratio = 1", "MSE", 100, "2006-12-31", 100, 1, 1, "first")

run_models("1.2 Ratio = 2", "MSE", 100, "2006-12-31", 100, 1, 2, "first")

run_models("1.3 Ratio = 3", "MSE", 100, "2006-12-31", 100, 1, 3, "first")

Here, we see the (unsurprising) result that (1) the lambda selected by the model (on average) gets larger as our declared failures/predictors ratio increases, and (2) at this training set size, the full model and individual models don’t look that different from an MSE standpoint. This begs the question, how do the same reports look for other test statistics?

Lambda tuning, given a training set contains at least one failure (AUC)

Here’s AUC (with the same series of hyperparameters):

run_models("2.1 Ratio = 1", "AUC", 100, "2006-12-31", 100, 1, 1, "first")

run_models("2.2 Ratio = 2", "AUC", 100, "2006-12-31", 100, 1, 2, "first")

run_models("2.3 Ratio = 3", "AUC", 100, "2006-12-31", 100, 1, 3, "first")

Hmm, some less than impressive out-of-sample performance for AUC on the full model. As a general pattern, raising the failures/predictors ratio seems to improve the bias between in-sample and out-of-sample stats.

More generalized AUC tuning

While we’re looking at AUC, some changes in the other hyperparamaters look like this:

run_models("3.1 Ratio = 1, 2007", "AUC", 10, "2007-12-31", 100, 1, 1, "first")

run_models("3.2 Ratio = 1, 2008", "AUC", 10, "2008-12-31", 100, 1, 1, "first")

run_models("3.3 Ratio = 1, 2007, larger sample", "AUC", 50, "2007-12-31", 500, 1, 1, "first")

run_models("3.4 Ratio = 1, 2008, larger sample", "AUC", 50, "2008-12-31", 500, 1, 1, "first")

run_models("3.5 Ratio = 3, 2008, smaller sample", "MSE", 50, "2008-06-30", 100, 10, 3, "first")

run_models("3.6 Ratio = 4, 2008, smaller sample", "MSE", 50, "2008-06-30", 100, 10, 4, "first")

run_models("3.7 Ratio = 3, 2008, smaller sample", "AUC", 50, "2008-06-30", 100, 10, 3, "first")

run_models("3.8 Ratio = 4, 2008, smaller sample", "AUC", 50, "2008-06-30", 100, 10, 4, "first")

My interpretations of the above graphs would be:

• AUC seems to offer more visibility into a model’s “correctness” than MSE does, and maybe part of the reason for that is that it varies more often than MSE does (for full vs. individual models). Even in relatively small training set sizes, it becomes apparent that full-variable models produce higher AUCs than most of the individual variables (3.1, 3.2). It becomes esp. apparent when sample sizes are larger (3.3, 3.4), at which point bias between in-sample and out-of-sample disappears too.
• The goofy charts from last week remain goofy (at least to my intuition). 3.6 and 3.7 have identical parameters to what we saw last week: MSE, for training sets of 100 where > 10 failures are selected, all seem to overfit and perform inaccurately. The AUC version of the same parameters, though, don’t show the same story – they look similar to the other AUC charts. I would have expected a discrepency for AUC and MSE, so I’m unsure how to interpret?

Lambda tuning by selecting the optimal lambda given our ratio constraint

This is where the last parameter I’m calling comes in. In every instance thus far, I’ve selected (as with last week) the very first lambda that satifsies the ratio constraint. In practice, though, glmnet provides us with a series of lambda tests that satisfy the ratio constraint, and which in some cases knocks out even more variables than it “should”. The below does lambda selection by (1) selecting all candidate lambdas as the inputs for a cross-validation, and selecting for prediction the one lambda that minimizes training error.

run_models("4.1 Ratio = 2", "MSE", 50, "2006-12-31", 100, 5, 2, "best")

run_models("4.2 Ratio = 2", "AUC", 50, "2006-12-31", 100, 5, 2, "best")

run_models("4.3 Ratio = 4", "MSE", 50, "2006-12-31", 100, 5, 4, "best")

run_models("4.4 Ratio = 4", "AUC", 50, "2006-12-31", 100, 5, 4, "best")

• For our MSE graphs, there is a similar “underfitting effect” that we saw in our chart (3…) above, but we also see that AUC for the full model becomes noticably worse with this lambda selection.
• Different ratios seem to produce negligible differences.

What’s next?

• Elastic-net applied to the first/second sections above
• Cox / proportional-hazards modeling