Week 10: Elastic Net, Cox and More

source("Pre-Processing.R")

Tuning Full Model with Elastic Net Hyperparameters

Given a training set size for a specific date on all CAMEL variables, I first fit an elastic net model to that training set while validating the lambda and alpha parameters on the validation set. Recall that for a specified date, our dataset contains about n = 7731 financial institutions, so the typical example for a small training set size is:

• training_set = 100
• validation_set = 3816
• test_set = 3816

I’m of the opinion that the validation/test set sizes are overkill, but that’s OK. A word about the hyperparamater selection. Our series of parameters are accepted by the model: date of significance, training set size, minimum # of failures in a training set (simulations is nominally a parameter, but steadies at 100 so I’ll keep it there; the ‘best’ vs. ‘first’ logic for lambda selection from last week I’ve done away with, since there didn’t seem to be much of a difference between the two conventions.)

First, I loop over values of alpha surveying the whole range between 0.0, 0.1, … , 0.9, 1.0. For each alpha, I capture a sequence of lambdas using cv.glmnet (but don’t actually use the CV functionality – just gather the sequence). That sequence is used to fit an elastic-net model with the given alpha, and the best lambda which enforces our ratio rule (failures / predictors) for that alpha. For each alpha, that process occurs until we have a best-performing model in validation, with a specific alpha and lambda. Then we test.

Here’s how the whole thing performs against the single-variable models, given toggling some of the hyperparameters:

run_models("1.1: Elastic Net, training set = 100", "AUC", "elastic_net", 100, "2007-06-30", 100, 10)

run_models("1.2: Elastic Net, training set = 1000", "AUC", "elastic_net", 100, "2007-06-30", 1000, 10)

run_models("1.3: Elastic Net, training set = 100", "AUC", "elastic_net", 100, "2007-12-31", 100, 10)

run_models("1.4: Elastic Net, training set = 1000", "AUC", "elastic_net", 100, "2007-12-31", 1000, 10)

Our results here are generally unsurprising, notably:

• For a farther back time period (Q2 2007), the full model’s test AUC isn’t notably better than individual models, and is in fact worse than some of the individual predictors. This indicates to me some noise in the multiple-variable model during earlier time periods, even when 2-3 predictor variables are being removed by the elastic net. At this size/timeframe, elastic net performs slightly worse than pure lasso did as we saw last week (~.63, Chart 3.1).
• An increase in our training set size decreases bias across the board, and the full model outperforms all but one single variable handily (Tier 1 capital ratio is consistent throughout this week’s models for reliably measuring failure risk).
• Both “sides” of the elastic net seem to be utilized well – the lambda not close to zero in any cases, 2-3 variables being dropped in most cases, and alpha lingering between .4-.6.
• 1.4 is a promising result – big differential between test AUC and the nearest individual model – but that’s a bit close to the actual crisis years.

Also, I tested gathering the lambda from the same banks in the training set, but with the oldest data points in the dataset I’ve pulled (2004). While not visualized here (and a little cumbersome to code), the model performance wasn’t reaching an AUC of above .55 or an MSE of < .12. Suffice to say, the manual validation of alpha / lambda performs this task pretty well.

#### How does AUC change as sample size changes?

This was another intuitive result to recreate, where the pattern of “more complex models are generally overperformant when more data exists” can be seen in multiple examples. I declined to visualize this, since it becomes tricky when different individual models knock one another off for best performer across different sample sizes.

Here’s a display of a bunch of examples of these across different full model types (lasso, ridge, etc.) and time periods. The function is hard-code to the same range of minimum failures, simulations, etc.:

sample_size_to_AUC_differential

sample_size_to_AUC_differential("lasso", "2007-06-30")

sample_size_to_AUC_differential("ridge", "2007-06-30")

sample_size_to_AUC_differential("logit", "2007-06-30")

And similarly for early 2007…

sample_size_to_AUC_differential

sample_size_to_AUC_differential("lasso", "2007-03-31")

sample_size_to_AUC_differential("ridge", "2007-03-31")

sample_size_to_AUC_differential("logit", "2007-03-31")

And finally, in late 2007 – presumably more accurate across the board…

sample_size_to_AUC_differential("lasso", "2007-12-31")

sample_size_to_AUC_differential("ridge", "2007-12-31")

sample_size_to_AUC_differential("logit", "2007-12-31")