Please navigate through the section tabs on the left and click on plots to see them better
I started this project in January 2023 for a Statistics class at Portland State University. It represents my coding abilities in R and my understanding of statistical concepts and methodologies. I will be working with German Credit Data which can be found at:
https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data
Most of the methods I will employ come from:
James, G., Hastie, T., Witten, D., & Tibshirani, R. (2023). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer.
The data set contains 1000 observations where people are classified as either a good (“0”) or bad (“1”) credit risk for the variable “Default” that I will be using as the response. I will try to predict the response variable using the following classification methods: Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), Logistic Regression, K Nearest Neighbors (KNN) and Logistic Ridge Regression (Ridge). Each method will use the same transformed dataset and k-fold cross-validation with k=5 to allow for better comparison. I will evaluate the models based on the misclassification rate (total incorrect predictions / total predictions), sensitivity (true positives / positive predictions) and specificity (true negatives / negative predictions). Where a true positive is correctly predicting a defaulter, a true negative is correctly predicting a good loan. Sensitivity and specificity are important in this context because the data set also came with a cost matrix showing that the bank values avoiding false negatives at a rate five times higher than false positives. This is because a false negative represents someone getting a loan when they will default which loses more money than they stand to make from a fully repaid loan. So I want to favor specificity over sensitivity for tuning and setting thresholds. Plotting the Remote Operator Curves (ROC) will show this trade-off and help with model selection.
I used the dplyr package to manipulate the data into more acceptable forms to run the models. Most of the variables used are from categorical data with 4 to 5 different classes that were turned into binary dummy variables and given “bv_” as a prefix. Some of the quantitative variables were given log transformations to help with normality. Some of the categorical variables were kept as ordinal. For example, one variable had 4 classes that each represented monetary ranges and were assigned numeric values from 1 to 4 to retain the meaning of the ordered ranks. The data manipulations were performed based on the data exploration in the following section and reading the documentation that can be found on the UCI website.
library(readr)
library(knitr)
library(dplyr)
library(glmnet)
library(ISLR)
library(ISLR2)
library(MASS)
library(pROC)
library(ggplot2)
library(reshape2)
library(scales)
library(car)
library(caret)
library(boot)
library(class)
germancredit <- germancredit %>%
mutate(across(c(history, employ, job), as.factor)) %>%
mutate(across(c(history, employ, job), ~as.integer(.))) %>%
mutate(bv_no_checking_account = ifelse(checkingstatus1 == "A14", 1, 0),
bv_200_plus_checking = ifelse(checkingstatus1 == "A13", 1, 0),
log_duration = log(duration),
bv_car_loan = ifelse(purpose %in% c("A40", "A41"), 1, 0),
bv_personal_loan = ifelse(purpose %in% c("A42", "A43", "A44", "A45", "A410"), 1, 0),
bv_ed_loan = ifelse(purpose == "A46", 1, 0),
bv_business_loan = ifelse(purpose == "A49", 1, 0),
log_amount = log(amount),
bv_no_savings = ifelse(savings == "A65", 1, 0),
bv_500_plus_savings = ifelse(savings %in% c("A63", "A64"), 1, 0),
bv_is_male = ifelse(status %in% c("A91", "A93", "A94"), 1, 0),
bv_has_cosigner = ifelse(others %in% c("A102", "A103"), 1, 0),
bv_property_A121_real_estate = ifelse(property == "A121", 1, 0),
bv_property_A122_buildingsociety_life = ifelse(property == "A122", 1, 0),
bv_property_A123_car = ifelse(property == "A123", 1, 0),
log_age = log(age),
bv_other_bank = ifelse(otherplans == "A141", 1, 0),
bv_other_stores = ifelse(otherplans == "A142", 1, 0),
bv_own_house = ifelse(housing == "A152", 1, 0),
bv_free_housing = ifelse(housing == "A153", 1, 0),
bv_has_tele = ifelse(tele == "A192", 1, 0),
bv_is_foreign = ifelse(foreign == "A201", 1, 0)
)
selected_cols <- c("Default", "bv_no_checking_account", "bv_200_plus_checking", "log_duration", "bv_car_loan", "bv_ed_loan", "bv_business_loan", "log_amount", "bv_no_savings", "bv_500_plus_savings", "history", "employ", "installment", "bv_is_male", "bv_has_cosigner", "bv_property_A121_real_estate", "bv_property_A122_buildingsociety_life", "bv_property_A123_car", "log_age", "bv_other_bank", "bv_other_stores", "bv_own_house", "bv_free_housing", "cards", "job", "liable", "residence")
df3 <- germancredit[selected_cols]Print function used to look at the frequencies of categorical variables. “foreign” was the only variable discarded in the models due to an unbalanced distribution. The class labels corresponding to “A11”, “A12”…ect can be found on the UCI website.
cat_variable_freq <- function(variable_name) {
variable_sums <- table(germancredit[[variable_name]])
cat(variable_name, "\n")
print(variable_sums)
cat("\n")
}
cat_variable_freq("checkingstatus1")## checkingstatus1
##
## A11 A12 A13 A14
## 274 269 63 394
## foreign
##
## A201 A202
## 963 37
Plot function used to look at distributions factored by the response variable “Default”
create_bar_plot <- function(data, variable) {
ggplot(data, aes(x = !!as.symbol(variable), fill = factor(Default))) +
geom_bar(position = "dodge") +
stat_count(geom = "text", aes(label = ..count..), position = position_dodge(width = 0.9), vjust = -0.5) +
labs(title = paste("Distribution of", variable),
x = variable, y = "Count") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
}
create_bar_plot(germancredit, "age")QQ plot function used to determine log transformations: We can see that the transformation on “amount” successfully addresses the issue.
qq_print <- function(data, col) {
par(mfrow = c(1, 2), mar = c(3, 3, 1, 1))
qqnorm(data[[col]], main = paste("QQ Plot of", col))
qqline(data[[col]], col = "red")
boxplot(data[[col]], main = paste("Boxplot of", col), col = "lightblue", border = "black", notch = TRUE)
}
qq_print(germancredit, "amount")Correlation Matrix and Variance Inflation Factor plots that show the assumptions of colinearity and variance are met. We are looking for values above ~10 on the VIF plot to raise concerns.
correlation_matrix <- cor(df3)
colnames(correlation_matrix) <- substr(colnames(correlation_matrix), 1, 12)
rownames(correlation_matrix) <- substr(rownames(correlation_matrix), 1, 12)
melted_cor <- melt(correlation_matrix)
ggplot(melted_cor, aes(x = Var1, y = Var2, fill = value)) +
geom_tile() +
scale_fill_gradientn(colours = c("blue", "green", "white", "orange", "red"),
values = scales::rescale(c(-1, 1)),
limits = c(-1, 1),
breaks = c(-1, -0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8, 1),
labels = c("-1.0", "-0.8", "-0.6", "-0.4", "-0.2", "0.0", "0.2", "0.4", "0.6", "0.8", "1.0")) +
theme_minimal() +
labs(title = "Correlation Matrix", x = "", y = "") +
theme(axis.text.x = element_text(angle = 90, hjust = 1),
axis.text.y = element_text(size = 10)) +
coord_fixed()#############################
lm_model <- lm(Default ~ ., data = df3)
vif_result <- vif(lm_model)
vif_df <- as.data.frame(vif_result)
vif_df$Variable <- rownames(vif_df)
names(vif_df) <- c("VIF", "Variable")
ggplot(vif_df, aes(x = Variable, y = VIF)) +
geom_bar(stat = "identity", fill = "skyblue", width = 0.7) +
labs(title = "Variance Inflation Factor (VIF)",
x = "Predictors",
y = "VIF") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))While there are packages that directly implement k-fold cross validation for glm, I decided to run it manually in a for loop to extract the specific metrics I want. The k-fold cross-validation set at k=5 splits the data into k-1 folds for training, while leaving the last fold as the test set for validation. The process is repeated k number of times so that all of the data can be used for training while lowering the overall bias. With k=n in the cross-validation I would be running Leave One Out Cross-Validation (LOOCV).
This function is used to average the sensitivities and specificities at each threshold outside the loop to help plot the mean ROC curve after k-fold cross validation. it can be used for sensitivity or specificity.
mean_list <- function(sensitivities_list) {
fun_sensitivity_mean <- numeric(length = length(sensitivities_list[[1]]))
for (i in seq_along(fun_sensitivity_mean)) {
sensitivity_sum <- 0
for (j in seq_along(sensitivities_list)) {
sensitivity_value <- sensitivities_list[[j]][[i]]
sensitivity_sum <- sensitivity_sum + sensitivity_value
}
fun_sensitivity_mean[i] <- sensitivity_sum / length(sensitivities_list)
}
return(fun_sensitivity_mean)
}The null classifier is a concept that sets a baseline for the model’s success. With a binary response variable, you can select one class for all predictions, and the model will be successful solely based on the distribution. For the response variable Default, the distribution was 300 defaults and 700 non-defaults, so selecting a model that always predicts no default will have a misclassification rate of 0.300 and a 70% accuracy.
LDA QDA were both ran using functions from the MASS package. An initial look at the results shows poor results slightly better than the null classifier. LDA can be seen to perform slightly better than QDA.
set.seed(123)
# number of folds for cross validation
k <- 5
mean_misclassification <- numeric(k)
sensitivity <- numeric(k)
specificity <- numeric(k)
colors <- colorRampPalette(c("lightblue", "darkblue"))(k)
sensitivities_list <- list()
specificities_list <- list()
thresholds_list <- list()
for (i in 1:k) {
folds <- createFolds(df3$Default, k = k, list = TRUE, returnTrain = FALSE)
# Extract train and test data for this fold (createFold returns indices)
train_indices <- unlist(folds[-i])
test_indices <- unlist(folds[i])
train_data <- df3[train_indices, ]
test_data <- df3[test_indices, ]
lda_model <- lda(Default ~ ., data = train_data)
lda_predictions <- predict(lda_model, newdata = test_data)
lda_conf_matrix <- table(lda_predictions$class, test_data$Default)
sensitivity[i] <- lda_conf_matrix[2, 2] / sum(lda_conf_matrix[2, ])
specificity[i] <- lda_conf_matrix[1, 1] / sum(lda_conf_matrix[1, ])
misclassification_rate <- (lda_conf_matrix[1, 2] + lda_conf_matrix[2, 1]) / sum(lda_conf_matrix)
mean_misclassification[i] <- misclassification_rate
lda_roc_curve <- roc(test_data$Default, as.numeric(lda_predictions$posterior[, 2]))
sensitivities_list[[i]] <- lda_roc_curve$sensitivities
specificities_list[[i]] <- lda_roc_curve$specificities
thresholds_list[[i]] <- lda_roc_curve$thresholds
plot.roc(lda_roc_curve, col = colors[i], lwd = 2, add = ifelse(i > 1, TRUE, FALSE))
}
lda_sensitivity_mean <- mean_list(sensitivities_list)
lda_specificity_mean <- mean_list(specificities_list)
lines(lda_specificity_mean, lda_sensitivity_mean, col = "red", lwd = 2)
lda_sensitivity <- mean(sensitivity)
lda_specificity <- mean(specificity)
cat("LDA Sensitivity:", lda_sensitivity, "\n")## LDA Sensitivity: 0.61429
## LDA Specificity: 0.7750438
lda_misclassification_rate <- mean(mean_misclassification)
cat("LDA Misclassification Rate:", lda_misclassification_rate, "\n")## LDA Misclassification Rate: 0.261
legend("bottomright", legend = c(paste("Fold", 1:k), "Mean"), col = c(colors, "red"), lwd = 2)
main_title <- "ROC Curve for k-fold iterations of LDA"
title(main = main_title, line = 2.5)set.seed(123)
# number of folds for cross validation
k <- 5
mean_misclassification <- numeric(k)
sensitivity <- numeric(k)
specificity <- numeric(k)
colors <- colorRampPalette(c("lightblue", "darkblue"))(k)
sensitivities_list <- list()
specificities_list <- list()
thresholds_list <- list()
for (i in 1:k) {
folds <- createFolds(df3$Default, k = k, list = TRUE, returnTrain = FALSE)
train_indices <- unlist(folds[-i])
test_indices <- unlist(folds[i])
train_data <- df3[train_indices, ]
test_data <- df3[test_indices, ]
qda_model <- qda(Default ~ ., data = train_data)
qda_predictions <- predict(qda_model, newdata = test_data)
qda_conf_matrix <- table(qda_predictions$class, test_data$Default)
sensitivity[i] <- qda_conf_matrix[2, 2] / sum(qda_conf_matrix[2, ])
specificity[i] <- qda_conf_matrix[1, 1] / sum(qda_conf_matrix[1, ])
misclassification_rate <- (qda_conf_matrix[1, 2] + qda_conf_matrix[2, 1]) / sum(qda_conf_matrix)
mean_misclassification[i] <- misclassification_rate
qda_roc_curve <- roc(test_data$Default, as.numeric(qda_predictions$posterior[, 2]))
sensitivities_list[[i]] <- qda_roc_curve$sensitivities
specificities_list[[i]] <- qda_roc_curve$specificities
thresholds_list[[i]] <- qda_roc_curve$thresholds
plot.roc(qda_roc_curve, col = colors[i], lwd = 2, add = ifelse(i > 1, TRUE, FALSE))
}
qda_sensitivity_mean <- mean_list(sensitivities_list)
qda_specificity_mean <- mean_list(specificities_list)
lines(qda_specificity_mean, qda_sensitivity_mean, col = "red", lwd = 2)
qda_sensitivity <- mean(sensitivity)
qda_specificity <- mean(specificity)
cat("QDA Sensitivity:", qda_sensitivity, "\n")## QDA Sensitivity: 0.5184411
## QDA Specificity: 0.7769005
qda_misclassification_rate <- mean(mean_misclassification)
cat("QDA Misclassification Rate:", qda_misclassification_rate, "\n")## QDA Misclassification Rate: 0.299
legend("bottomright", legend = c(paste("Fold", 1:k), "Mean"), col = c(colors, "red"), lwd = 2)
main_title <- "ROC Curve for k-fold iterations of QDA"
title(main = main_title, line = 2.5)The linear regression models will be ran using the generalized linear model (glm function) with family set to “binomial”.
Stepwise Regression was run to see which predictors are indicated to be significant (P<.05). Forward and multi-directional methods both yielded similar results. This was done primarily for inference as the reduced model will not be used in any of the further analysis in favor of Ridge regression.
stepwise_model <- step(glm(Default ~ ., data = df3, family = binomial), direction = "both", trace = 0)
summary(stepwise_model)##
## Call:
## glm(formula = Default ~ bv_no_checking_account + bv_200_plus_checking +
## log_duration + bv_car_loan + bv_ed_loan + bv_no_savings +
## bv_500_plus_savings + history + employ + installment + bv_is_male +
## bv_has_cosigner + bv_property_A121_real_estate + bv_property_A122_buildingsociety_life +
## bv_property_A123_car + bv_other_bank + bv_own_house + bv_free_housing +
## cards, family = binomial, data = df3)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.76576 0.70208 -1.091 0.27540
## bv_no_checking_account -1.51756 0.19397 -7.824 5.13e-15 ***
## bv_200_plus_checking -0.76147 0.34062 -2.236 0.02538 *
## log_duration 0.84917 0.15499 5.479 4.28e-08 ***
## bv_car_loan 0.35665 0.17704 2.014 0.04396 *
## bv_ed_loan 0.88107 0.36904 2.387 0.01696 *
## bv_no_savings -0.94047 0.23986 -3.921 8.82e-05 ***
## bv_500_plus_savings -0.76119 0.30727 -2.477 0.01324 *
## history -0.41896 0.08599 -4.872 1.10e-06 ***
## employ -0.15948 0.06780 -2.352 0.01866 *
## installment 0.22486 0.07419 3.031 0.00244 **
## bv_is_male -0.33232 0.17802 -1.867 0.06193 .
## bv_has_cosigner -0.40049 0.27844 -1.438 0.15034
## bv_property_A121_real_estate -0.75610 0.37331 -2.025 0.04282 *
## bv_property_A122_buildingsociety_life -0.53230 0.37054 -1.437 0.15085
## bv_property_A123_car -0.62718 0.36159 -1.734 0.08283 .
## bv_other_bank 0.46101 0.22079 2.088 0.03679 *
## bv_own_house -0.35649 0.21014 -1.696 0.08981 .
## bv_free_housing -0.80931 0.42435 -1.907 0.05650 .
## cards 0.25164 0.15584 1.615 0.10637
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1221.73 on 999 degrees of freedom
## Residual deviance: 963.12 on 980 degrees of freedom
## AIC: 1003.1
##
## Number of Fisher Scoring iterations: 5
Logistic Regression was initially run with a threshold set at >.5 for a positive prediction. We can see a misclassification rate slightly over .3 which is concerning as it is not performing better than the null classifier.
set.seed(123)
k <- 5
mean_misclassification <- numeric(k)
sensitivity <- numeric(k)
specificity <- numeric(k)
colors <- colorRampPalette(c("lightblue", "darkblue"))(k)
sensitivities_list <- list()
specificities_list <- list()
roc_curve_list <- list()
for (i in 1:k) {
folds <- createFolds(df3$Default, k = k, list = TRUE, returnTrain = FALSE)
train_indices <- unlist(folds[-i])
test_indices <- unlist(folds[i])
train_data <- df3[train_indices, ]
test_data <- df3[test_indices, ]
log_model <- glm(Default ~ ., data = train_data, family = binomial)
log_predictions <- predict(log_model, newdata = test_data, type = "response")
log_conf_matrix <- table(ifelse(log_predictions > 0.5, 1, 0), test_data$Default)
sensitivity[i] <- log_conf_matrix[2, 2] / sum(log_conf_matrix[2, ])
specificity[i] <- log_conf_matrix[1, 1] / sum(log_conf_matrix[1, ])
log_roc_curve <- roc(test_data$Default, log_predictions)
roc_curve_list[[i]] <- log_roc_curve
sensitivities_list[[i]] <- log_roc_curve$sensitivities
specificities_list[[i]] <- log_roc_curve$specificities
thresholds_list[[i]] <- log_roc_curve$thresholds
plot.roc(log_roc_curve, col = colors[i], lwd = 2, add = ifelse(i > 1, TRUE, FALSE))
}
mean_sensitivity <- mean(sensitivity)
mean_specificity <- mean(specificity)
mean_misclassification_rate <- 1 - (mean_sensitivity + mean_specificity) / 2
log_sensitivity_mean <- mean(sapply(sensitivities_list, mean))
log_specificity_mean <- mean(sapply(specificities_list, mean))
log_sensitivity_mean <- mean_list(sensitivities_list)
log_specificity_mean <- mean_list(specificities_list)
lines(log_specificity_mean, log_sensitivity_mean, col = "red", lwd = 2)
legend("bottomright", legend = c(paste("Fold", 1:k), "Mean"), col = c(colors, "red"), lwd = 2)
main_title <- "ROC Curve for k-fold iterations of Log Regression"
title(main = main_title, line = 2.5)## Logistic Regression Mean Sensitivity: 0.6125741
## Logistic Regression Mean Specificity: 0.775401
## Logistic Regression Mean Misclassification Rate: 0.3060125
The dataset was scaled to a mean of 0 and standard deviation of 1 prior to running the analysis. KNN uses distance algorithms to find the nearest k points from any observation and determines classification based on those calculations. So the first step of KNN is to select K. The code loops through different values of k from 0 to 30 and plot the misclassification rates. The process uses the same methodology as k-fold cross validation, but will now iterate through 30 folds knn k values. Ideally we want to select a k value where the overall trend shows a dip indicating improved accuracy. There isn’t a clear choice for k, but I will still select based on minimizing the the error rate at k=10
set.seed(123)
scaled_df3 <- preProcess(df3[, -1], method = c("scale", "center"))
scaled_data <- predict(scaled_df3, df3[, -1])
# number of k-folds and range of knn k's to test for
k <- 30
misclassification_rate <- numeric(k)
for (i in 1:k) {
folds <- createFolds(df3$Default, k = k, list = TRUE, returnTrain = FALSE)
train_indices <- unlist(folds[-i])
test_indices <- unlist(folds[i])
train_data <- scaled_data[train_indices, ]
test_data <- scaled_data[test_indices, ]
knn_predictions <- knn(train = train_data, test = test_data, cl = df3$Default[train_indices], k = i)
knn_conf_matrix <- table(knn_predictions, df3$Default[test_indices])
misclassification_rate[i] <- (knn_conf_matrix[1, 2] + knn_conf_matrix[2, 1]) / sum(knn_conf_matrix)
}
plot(1:k, misclassification_rate, type = "b", pch = 16, col = "blue",
xlab = "k", ylab = "Misclassification Rate",
main = "K Selection for KNN")# Find the index of the minimum misclassification rate
min_index <- which.min(misclassification_rate)
# Print the value of k corresponding to the minimum misclassification rate
cat("Value of k when misclassification rate is at minimum:", min_index, "\n")## Value of k when misclassification rate is at minimum: 10
## Misclassification Rate at minimum: 0.1176471
When running the model the k is now once again equal to 5 for the number of folds. The KNN k value is input directly into the model. The results show another disappointing result for misclassification
set.seed(123)
# Define the number of folds
k <- 5
sensitivity <- numeric(k)
specificity <- numeric(k)
misclassification_rate <- numeric(k)
sensitivities_list <- list()
specificities_list <- list()
roc_curve_list <- list()
for (i in 1:k) {
folds <- createFolds(df3$Default, k = k, list = TRUE, returnTrain = FALSE)
train_indices <- unlist(folds[-i])
test_indices <- unlist(folds[i])
train_data <- df3[train_indices, ]
test_data <- df3[test_indices, ]
knn_predictions <- knn(train = train_data[, -1], test = test_data[, -1], cl = train_data$Default, k = 10)
knn_conf_matrix <- table(knn_predictions, test_data$Default)
sensitivity[i] <- knn_conf_matrix[2, 2] / sum(knn_conf_matrix[2, ])
specificity[i] <- knn_conf_matrix[1, 1] / sum(knn_conf_matrix[1, ])
misclassification_rate[i] <- (knn_conf_matrix[1, 2] + knn_conf_matrix[2, 1]) / sum(knn_conf_matrix)
knn_roc_curve <- roc(test_data$Default, as.numeric(knn_predictions))
roc_curve_list[[i]] <- log_roc_curve
sensitivities_list[[i]] <- knn_roc_curve$sensitivities
specificities_list[[i]] <- knn_roc_curve$specificities
thresholds_list[[i]] <- knn_roc_curve$thresholds
plot.roc(knn_roc_curve, col = colors[i], lwd = 2, add = ifelse(i > 1, TRUE, FALSE))
}
avg_sensitivity <- mean(sensitivity)
avg_specificity <- mean(specificity)
avg_misclassification_rate <- mean(misclassification_rate)
knn_sensitivity_mean <- mean_list(sensitivities_list)
knn_specificity_mean <- mean_list(specificities_list)
lines(knn_specificity_mean, knn_sensitivity_mean, col = "red", lwd = 2)## Average Sensitivity: 0.5677364
## Average Specificity: 0.7263142
## Average Misclassification Rate: 0.298
Ridge regression incorporated into the glm model was the primary way that I decided to address dimensionality reduction. Instead of selecting significant predictors to reduce the model we can incorporate the full set of predictors with penalties for shrinking the coefficients. Here is one fold of the selection for lambda where we can see all of the predictors going from their initial state and converging at zero. The log range from -4 to 0 shows that the expected lambda value should be between 0 and 1. At the left hand side of the plot there is no shrinkage penalties and all of the coefficients are the same for standard logistic regression. At the right hand side of the plot there is too much shrinkage and all of the coefficients are completely reduced to zero.
The second plot shows the binomial deviance plotted over the log(lambda) where we can see that the model is improved with ridge regression as there is a minimum dip before the curve starts increasing again. We will select the lambda that minimizes the deviance allowing for some shrinking of the predictor coefficients which is .036. The first vertical line shows the minimization.
I did use the cv.glmnet function for the k fold cross validation in this case
k <- 5
for (i in 1:k) {
folds <- createFolds(df3$Default, k = k, list = TRUE, returnTrain = FALSE)
train_indices <- unlist(folds[-i])
test_indices <- unlist(folds[i])
train_data <- df3[train_indices, ]
test_data <- df3[test_indices, ]
y <- as.factor(train_data$Default)
X <- model.matrix(Default ~ . - 1, data = train_data)
log_ridge_model <- glmnet(X, y, family = "binomial", alpha = 0, standardize = TRUE)
#only plot for one fold
if (i == 1) {
plot(log_ridge_model, xvar = "lambda")
}
}## Best lambda: 0.03600849
ROC Curve for Ridge
# inputs best lambda from previous chunk
ridge_model <- glmnet(X, y, family = "binomial", alpha = 0, lambda = best_lambda)
y_prob <- predict(ridge_model, newx = X, type = "response")
ridge_roc_curve <- roc(y, y_prob)
plot(ridge_roc_curve, main = "ROC Curve", col = "blue")
legend("bottomright", legend = paste("AUC =", round(auc(ridge_roc_curve), 3)), col = "blue", lty = 1, cex = 0.8)The first plot shows each model’s ROCs on the same plot for comparison. The ROC shows sensitivity and specificity for each model’s threshold. Ridge regression appears to have the best sensitivity and specificity at any threshold, making it the best choice for continuing analysis. while KNN has a very low Area Under the Curve (AUC) it does have one specific spot in our target area where it appears to perform well.
The second plot shows a closer look at the lower left portion of the ROC curve with high values for specificity (or low for 1 - specificity). We are looking for these thresholds where our model would be less likely to predict a false negative than a false positive.
# convert ridge data into same size for other rocs
ridge_sensitivities_200 <- ridge_roc_curve$sensitivities[seq(1, length(ridge_roc_curve$sensitivities), by = 4)]
ridge_specificities_200 <- ridge_roc_curve$specificities[seq(1, length(ridge_roc_curve$specificities), by = 4)]
par(pty = "s")
plot(1 - lda_specificity_mean, lda_sensitivity_mean, type = "l", col = "red", lwd = 2,
xlab = "1 - Specificity", ylab = "Sensitivity",
main = "Comparing ROCs", xlim = c(0, 1), ylim = c(0, 1), asp = 1)
lines(1 - qda_specificity_mean, qda_sensitivity_mean, col = "orange", lwd = 2)
lines(1 - log_specificity_mean, log_sensitivity_mean, col = "blue", lwd = 2)
lines(1 - knn_specificity_mean, knn_sensitivity_mean, col = "green", lwd = 2)
lines(1 - ridge_specificities_200, ridge_sensitivities_200, col = "purple", lwd = 2)
legend("bottomright", legend = c("LDA", "QDA", "Logistic Regression", "KNN", "Ridge"),
col = c("red", "orange", "blue", "green", "purple"), lwd = 2, cex = 0.8)plot(1 - lda_specificity_mean, lda_sensitivity_mean, type = "l", col = "red", lwd = 2,
xlab = "1 - Specificity", ylab = "Sensitivity",
main = "Comparing ROCs (area of interest)", xlim = c(0, .3), ylim = c(0, .6))
lines(1 - qda_specificity_mean, qda_sensitivity_mean, col = "orange", lwd = 2)
lines(1 - log_specificity_mean, log_sensitivity_mean, col = "blue", lwd = 2)
lines(1 - knn_specificity_mean, knn_sensitivity_mean, col = "green", lwd = 2)
lines(1 - ridge_specificities_200, ridge_sensitivities_200, col = "purple", lwd = 2)
legend("bottomright", legend = c("LDA", "QDA", "Logistic Regression", "KNN", "Ridge"),
col = c("red", "orange", "blue", "green", "purple"), lwd = 2, cex = 0.8)Since Ridge regression is the best fit and the most straightforward model to modify the thresholds, I turned the model into a function then I ran it at various thresholds from .1 to .9. Evaluating the confusion matrix at each threshold allows us to find the highest specificity model while maintaining an adequate overall misclassification rate. These outputs show a confusion matrix with the predicted classes (0 or 1) on the left and the actual classes on the top. We can see that the best specificity has a trade-off with sensitivity and the overall misclassification rate. At a threshold of .2, there are over ten times more false positives than false negatives, which is close to the rate of five times more than we were looking for. At .3, there are approximately three times more false positives than false negatives, so our ideal model based solely on the cost of false negatives would be .25, which is right at the goal of five times less false negatives than false positives. However, we also see that the misclassification rate is worse at this value and slightly greater than the null classifier at .3. The overall misclassification rate improves dramatically around .4 to .5. making those potential considerations. I have ended the project here with ridge regression around a .25 to .4 threshold being the best model I could produce, which still leaves a lot of room for improvement.
perform_ridge_regression <- function(log_threshold, X, y, best_lambda) {
ridge_model <- glmnet(X, y, family = "binomial", alpha = 0, lambda = best_lambda)
y_prob <- predict(ridge_model, newx = X, type = "response")
y_pred <- ifelse(y_prob > log_threshold, 1, 0)
conf_matrix <- table(factor(y_pred), factor(y))
cat("threshold:", log_threshold, "\n")
print(conf_matrix)
sensitivity <- conf_matrix[2, 2] / sum(conf_matrix[2, ])
specificity <- conf_matrix[1, 1] / sum(conf_matrix[1, ])
misclassification_rate <- (conf_matrix[1, 2] + conf_matrix[2, 1]) / sum(conf_matrix)
cat("Sensitivity:", sensitivity, "\n")
cat("Specificity:", specificity, "\n")
cat("Misclassification Rate:", misclassification_rate, "\n")
cat("\n")
}
perform_ridge_regression(.1, X, y, best_lambda)## threshold: 0.1
##
## 0 1
## 0 138 2
## 1 417 243
## Sensitivity: 0.3681818
## Specificity: 0.9857143
## Misclassification Rate: 0.52375
## threshold: 0.2
##
## 0 1
## 0 294 25
## 1 261 220
## Sensitivity: 0.4573805
## Specificity: 0.9216301
## Misclassification Rate: 0.3575
## threshold: 0.25
##
## 0 1
## 0 349 40
## 1 206 205
## Sensitivity: 0.4987835
## Specificity: 0.8971722
## Misclassification Rate: 0.3075
## threshold: 0.3
##
## 0 1
## 0 390 56
## 1 165 189
## Sensitivity: 0.5338983
## Specificity: 0.8744395
## Misclassification Rate: 0.27625
## threshold: 0.4
##
## 0 1
## 0 454 90
## 1 101 155
## Sensitivity: 0.6054688
## Specificity: 0.8345588
## Misclassification Rate: 0.23875
## threshold: 0.5
##
## 0 1
## 0 497 128
## 1 58 117
## Sensitivity: 0.6685714
## Specificity: 0.7952
## Misclassification Rate: 0.2325
## threshold: 0.6
##
## 0 1
## 0 530 180
## 1 25 65
## Sensitivity: 0.7222222
## Specificity: 0.7464789
## Misclassification Rate: 0.25625
## threshold: 0.7
##
## 0 1
## 0 547 221
## 1 8 24
## Sensitivity: 0.75
## Specificity: 0.7122396
## Misclassification Rate: 0.28625
## threshold: 0.8
##
## 0 1
## 0 555 239
## 1 0 6
## Sensitivity: 1
## Specificity: 0.6989924
## Misclassification Rate: 0.29875
As I became engrossed in this project I had a strong desire to continue testing different models and continue to refine them. It became apparent to me that model selection is an ongoing process and I don’t want to call this a completed project. It would have also been possible to modify the thresholds in LDA, QDA, and KNN. However, I don’t think any of those methods showed enough merit. If I were to continue the project, I might have focused on incorporating LASSO into the regression or elastic net. I could have also focused on entirely different models like decision trees.
I hope this project demonstrates my current understanding of the subject matter but also leaves room open for me to continue learning different methods and apply them to exciting new data sets to conduct meaningful analysis.