Predicting Loan Applicant Default: A Model Building Approach
Adam Moffitt
2023-07-31
1. Introduction
Credit card companies use models to predict the likelihood that an applicant will default on their Bank Loan. With that in mind, the purpose of this project is to create a model that will predict whether or not an applicant will default on their Bank Loan. A data set with 1000 observations and 16 variables from Applied Analytics through Case Studies Using SAS and R by Deepti Gupta will be used for the project. The data set includes 8 categorical variables, including the outcome variable of interest called “Default,” and 8 numerical variables.
2. Exploratory Data Analysis
After reviewing the summary statistics, it is apparent that the variables Default , Car_Loan , Personal_loan , Home_loan , and Education_Loan are categorized as numeric variables when they are in fact Categorical variables. They will be converted to Factor variables in addition to Emp_Status , Marital_Status , and Gender for ease of analysis. The summary statistic function was rerun after converting these variables and that output is displayed below.
Figure 1. Summary Statistics of Variables
--------------------------------
Default Checking_amount Term Credit_score
Min. :0.0 Min. :-665.0 Min. : 9.00 Min. : 376.0
1st Qu.:0.0 1st Qu.: 164.8 1st Qu.:16.00 1st Qu.: 725.8
Median :0.0 Median : 351.5 Median :18.00 Median : 770.5
Mean :0.3 Mean : 362.4 Mean :17.82 Mean : 760.5
3rd Qu.:1.0 3rd Qu.: 553.5 3rd Qu.:20.00 3rd Qu.: 812.0
Max. :1.0 Max. :1319.0 Max. :27.00 Max. :1029.0
Gender Marital_status Car_loan Personal_loan
Length:1000 Length:1000 Min. :0.000 Min. :0.000
Class :character Class :character 1st Qu.:0.000 1st Qu.:0.000
Mode :character Mode :character Median :0.000 Median :0.000
Mean :0.353 Mean :0.474
3rd Qu.:1.000 3rd Qu.:1.000
Max. :1.000 Max. :1.000
Home_loan Education_loan Emp_status Amount
Min. :0.000 Min. :0.000 Length:1000 Min. : 244
1st Qu.:0.000 1st Qu.:0.000 Class :character 1st Qu.:1016
Median :0.000 Median :0.000 Mode :character Median :1226
Mean :0.056 Mean :0.112 Mean :1219
3rd Qu.:0.000 3rd Qu.:0.000 3rd Qu.:1420
Max. :1.000 Max. :1.000 Max. :2362
Saving_amount Emp_duration Age No_of_credit_acc
Min. :2082 Min. : 0.00 Min. :18.00 Min. :1.000
1st Qu.:2951 1st Qu.: 15.00 1st Qu.:29.00 1st Qu.:1.000
Median :3203 Median : 41.00 Median :32.00 Median :2.000
Mean :3179 Mean : 49.39 Mean :31.21 Mean :2.546
3rd Qu.:3402 3rd Qu.: 85.00 3rd Qu.:34.00 3rd Qu.:3.000
Max. :4108 Max. :120.00 Max. :42.00 Max. :9.000 When looking at No_of_credit_acc, it can be seen that the number of credit accounts declines after five, however, for the time being, No_of_credit_acc will remain numeric and un collapsed. It may prove more advantageous in later stages to convert No_of_credit_acc to a factor variable and collapse 6-9 into a 6+ category.
| Credit Accounts | Freq |
|---|---|
| 1 | 308 |
| 2 | 325 |
| 3 | 119 |
| 4 | 105 |
| 5 | 109 |
| 6 | 6 |
| 7 | 8 |
| 8 | 6 |
| 9 | 14 |
Univariate ChiSquare Tests were performed with categorical variables by the default variable to discover any significant association.
| Variable | Chi_Square | DF | P_Value |
|---|---|---|---|
| Car_loan | 5.074005 | 1 | 0.0242872 |
| Gender | 5.348516 | 1 | 0.0207399 |
| Marital_status | 6.159816 | 1 | 0.0130685 |
| Personal_loan | 45.297532 | 1 | 0.0000000 |
| Home_loan | 7.790881 | 1 | 0.0052511 |
| Education_loan | 80.093327 | 1 | 0.0000000 |
| Emp_status | 0.080655 | 1 | 0.7764118 |
All categorical variables with the exception of Emp_status yielded a statistically significant association with default. The most significant association occurred between default and Education_Loan where those without student loans did not default. The results are shown in the bar chart below
Fig 4. Bar Chart of Education by Default
A correlation matrix was used to assess for any significant correlation between numeric variables. From the Matrix it can be seen that the most significant positive correlation is between Saving_amount and age. As age increases, savings amount increases. In terms of negative correlation, the most significant is between Age and Term. As age increases, term decreases. The squares with X’s indicate that there is no significant correlation between the two varibles at the .05 significance level. All squares without X’s demonstrated significant correlation at the .05 significance level.
Fig 5. Correlation Matrix of Numeric variables
The density and scatter plots are coded where the Orange color reflects not defaulting and the aqua color represents defaulting.
In Review of density plots that are the numerical variables crossed
by default, a number of things stick out visually. Those who default
have a lower Checking amount, greater Terms, lower Credit Scores,
greater Amount, lower Savings amount, and a younger age.
Scatter plots were prepared for the following numerical variables with the variable Default creating two different levels. These plots indicate visually there may be some potential co- linear relationships between the variables. For example, there appears to be a colinear relationship between Amount and Checking_Amount, Amount and Term, and Employment Duration and savings amount. The scatter plots also visualize the relationships shown in the density plots but in a different way.
3 The Candidate Models
The Original data set was split in to a training data set to train the candidate models and a test data set was used to assess the performance of the models. From the Orignal Datset of 1,000 observations, a 70/30 split was Incorporated with 700 observations in the training set and 300 observations in the testing set.
3.A Logistic Regression Model Building
To investigate whether or not someone will default on their loan, three logistic regression models were fit to the data to determine which model performs best in predicting whether or not someone will default. The logistic regression model will also indicate which variables are the most significant in terms of association between defaulting on a loan.
A Full Logistic Regression Model is fit to the training data set with “Default” as the response variable and the other fifteen variables as predictor variables.
| Estimate | Std. Error | z value | Pr(>|z|) | |
|---|---|---|---|---|
| (Intercept) | 39.9025787 | 5.9240627 | 6.7356780 | 0.0000000 |
| Checking_amount | -0.0049317 | 0.0008038 | -6.1354975 | 0.0000000 |
| Term | 0.2342254 | 0.0684103 | 3.4238316 | 0.0006174 |
| Credit_score | -0.0105484 | 0.0024878 | -4.2401139 | 0.0000223 |
| GenderMale | -0.0468614 | 0.6260959 | -0.0748471 | 0.9403364 |
| Marital_statusSingle | 0.3845520 | 0.6244803 | 0.6157953 | 0.5380297 |
| Car_loan1 | -1.3779309 | 3.5405050 | -0.3891905 | 0.6971352 |
| Personal_loan1 | -2.4773232 | 3.5407698 | -0.6996567 | 0.4841417 |
| Home_loan1 | -4.4749376 | 3.6213878 | -1.2356969 | 0.2165712 |
| Education_loan1 | 0.4208259 | 3.5738732 | 0.1177506 | 0.9062652 |
| Emp_statusunemployed | 0.2463111 | 0.4065920 | 0.6057943 | 0.5446513 |
| Amount | 0.0005281 | 0.0006678 | 0.7907602 | 0.4290840 |
| Saving_amount | -0.0048328 | 0.0007323 | -6.5990257 | 0.0000000 |
| Emp_duration | 0.0056039 | 0.0056208 | 0.9969792 | 0.3187746 |
| Age | -0.6596863 | 0.0788019 | -8.3714480 | 0.0000000 |
| No_of_credit_acc | -0.0799863 | 0.1207618 | -0.6623480 | 0.5077482 |
From the table above it can be observed that the variables Checking_amount , Term , Credit_score , Saving_Amount , and Emp_duration have a highly significant assocation regarding whether or not someone will default. These five variables are also the only statistically significant variables at a .05 significance level in terms of association between whether or not someone will default.
To construct a reduced model, a step wise selection algorithm was incorporated to select features for the reduced model. Forward and Backward selection yielded the same features for the model, whereas forward selection chose all but one feature to include in the model. Based on this, the features chosen from backward and forward-backward (both) selection were incorporated in to the reduced, Step AIC model.
| Estimate | Std. Error | z value | Pr(>|z|) | |
|---|---|---|---|---|
| (Intercept) | 40.8381151 | 4.6231540 | 8.833388 | 0.0000000 |
| Checking_amount | -0.0050678 | 0.0008035 | -6.307110 | 0.0000000 |
| Term | 0.2370735 | 0.0677178 | 3.500905 | 0.0004637 |
| Credit_score | -0.0101541 | 0.0024044 | -4.223202 | 0.0000241 |
| Car_loan1 | -1.8782406 | 0.7065856 | -2.658193 | 0.0078561 |
| Personal_loan1 | -2.9301812 | 0.7097100 | -4.128702 | 0.0000365 |
| Home_loan1 | -4.9573357 | 1.0644451 | -4.657202 | 0.0000032 |
| Saving_amount | -0.0047697 | 0.0007217 | -6.608820 | 0.0000000 |
| Age | -0.6575394 | 0.0765847 | -8.585783 | 0.0000000 |
After feature selection was performed using the Step AIC method within the prior section, the selected features in the reduced model were tested for pairwise interactions one at a time. The analysis determined the following terms have a significant interaction at the .05 significance level: Term:Credit_score: 0.00696 , Credit_score:Saving_amount: 0.028013 , Car_loan:Saving_Amount: 0.01270 , Personal_loan:Saving_amount: 0.02896. These interaction terms were then included into a new stepwise AIC algorithm using backward selection. The results are found below.
| Estimate | Std. Error | z value | Pr(>|z|) | |
|---|---|---|---|---|
| (Intercept) | -38.3831883 | 26.6421948 | -1.440692 | 0.1496718 |
| Checking_amount | -0.0054646 | 0.0008927 | -6.121128 | 0.0000000 |
| Term | 2.1847681 | 0.8051805 | 2.713389 | 0.0066599 |
| Credit_score | 0.0914907 | 0.0356429 | 2.566874 | 0.0102620 |
| Car_loan1 | 9.9227909 | 4.9684127 | 1.997175 | 0.0458061 |
| Personal_loan1 | -3.3529294 | 0.7408941 | -4.525518 | 0.0000060 |
| Home_loan1 | -4.9800002 | 1.0525285 | -4.731464 | 0.0000022 |
| Saving_amount | 0.0102921 | 0.0079472 | 1.295060 | 0.1952997 |
| Age | -0.6925115 | 0.0811738 | -8.531225 | 0.0000000 |
| Term:Credit_score | -0.0025649 | 0.0010494 | -2.444056 | 0.0145232 |
| Credit_score:Saving_amount | -0.0000184 | 0.0000105 | -1.743303 | 0.0812807 |
| Car_loan1:Saving_amount | -0.0039166 | 0.0015985 | -2.450170 | 0.0142789 |
3.B Creating a Neural Network for Prediction of Default
The initial step in developing a Neural network is developing a model matrix so the names of all feature variables including implicitly defined dummy variables are defined and extracted. This was performed on both the test and training dataset. Feature engineering in the form of normalization was also performed on all numerical variables to prepare them for building a neural network.
There are some naming issues in the above dummy feature variables for network modeling (although they are good for regular linear and generalized linear regression models). These feature variables were renamed by excluding special characters in order to build the neural network model.
Next we build the neural network model. The neuralnet function was used for this task. A single layer neural network was developed.
| error | 14.9974768 |
| reached.threshold | 0.0094634 |
| steps | 3617.0000000 |
| Intercept.to.1layhid1 | 16.6141178 |
| checkingAmount.to.1layhid1 | -8.1722715 |
| Term.to.1layhid1 | 4.2904723 |
| creditScore.to.1layhid1 | -6.5354810 |
| GenderMale.to.1layhid1 | 0.3248997 |
| maritalStatusSingle.to.1layhid1 | 0.7001326 |
| carLoan1.to.1layhid1 | -0.4712334 |
| personalLoan1.to.1layhid1 | -1.7527155 |
| homeLoan1.to.1layhid1 | -3.3455897 |
| educationLoan1.to.1layhid1 | 0.8317540 |
| empStatusUnemployed.to.1layhid1 | 0.3219977 |
| Amount.to.1layhid1 | -0.1122575 |
| savingAmount.to.1layhid1 | -9.1276846 |
| empDuration.to.1layhid1 | 0.7902562 |
| Age.to.1layhid1 | -14.1673891 |
| noOfCreditAcc.to.1layhid1 | -0.7896480 |
| Intercept.to.Default1 | -0.0046067 |
| 1layhid1.to.Default1 | 1.0169859 |
A diagram of the neural network can be viewed below.
Fig 11. Single-layer backpropagation Neural network model for Default
3.C Decision Tree Models for the Prediction of Default Status.
Four Decision Tree models were fit to the data set to see which variables perform the best in terms of predicting whether someone will default on there loan. The models are:
- Tree with Gini Index: Non-Penalization
- Tree with Entropy Non-Penalization
- Tree with Gini Index: Penalization False Negatives
- Tree with Entropy: Penalization False Negatives.
- Tree with Gini Index: Penalization False Positives
- Tree with Entropy: Penalization False Positives.
The Trees themselves can be viewed below.
Fig 12. Non-penalized decision tree models using Gini index (left) and entropy (right).
Fig 13. penalized decision tree models using Gini index (left) and entropy (right).
Fig 14. penalized decision tree models using Gini index (left) and entropy (right).
All trees incorporated Age as the main branch with a cut off between 30-32 years old. From there Checking Amount, Credit Score, Savings Amount, Amount, and term were incorporated into to branches within each model.
4 Performance of Candidate Models
The Full model contained VIF values all below five for numeric variables, indicating there’s no multi-collinearity issues. Car_loan, Personal_loan, Home_loan, and Education_loan have VIF values above five, but they are categorical variables. VIF values are applicable to continuous numerical variables.
Fig 15. Variance Inflation Factor (VIF) - Full Model
--------------------------------
Checking_amount Term Credit_score Gender
1.178541 1.118359 1.144263 2.712260
Marital_status Car_loan Personal_loan Home_loan
2.931119 92.365892 92.220960 21.413613
Education_loan Emp_status Amount Saving_amount
31.926009 1.105981 1.075671 1.217359
Emp_duration Age No_of_credit_acc
1.255676 1.236782 1.096510 VIF values for the reduced model are all below five indicating multi-collinearity is not an issue with the model.
Fig 16. Variance Inflation Factor (VIF) - Step AIC Model
--------------------------------
Checking_amount Term Credit_score Car_loan Personal_loan
1.183323 1.085144 1.105634 3.756111 3.803263
Home_loan Saving_amount Age
1.831949 1.204715 1.190951 Because there are interaction terms within this model, the VIF values are well above the 5-10 range which indicates there is significant multi-collinearity issues. The Variables with multi-collinearity issues are Term, Credit Score, and Saving Amount.
Fig 17. Variance Inflation Factor (VIF) - Interaction Terms Model
--------------------------------
Checking_amount Term
1.262488 135.626848
Credit_score Car_loan
190.751796 161.014114
Personal_loan Home_loan
3.814218 1.964400
Saving_amount Age
143.601513 1.365835
Term:Credit_score Credit_score:Saving_amount
181.089179 267.646853
Car_loan:Saving_amount
163.384862 Outliers for each Logistic Regression Model can be observed in the Pearson Residual Plots below.
Significant outliers were discovered for observations 766, 391, and 225. None of these outliers appear to be the result of any type of error and thus remained in the model. The reduced model produced the same outliers as the full model and again were maintained within the reduced model. The interaction model yielded the same outliers as the other two models and the outliers were maintained within the model.
Fig 18. Pearson Residual Plots of Logistic Regression Models
In terms of general prediction, a cut off probability first needed to be established. Five fold cross-validation was performed to determine the optimal cut off probability. After performing Five Fold Cross Validation on the training dataset, it was determined that the optimal cut off probability was 0.33 for the full model. This means that any individual with a probability greater than 0.33 was predicted to default on their loan. In addition, using the training data set, the full model correctly predicted whether someone would default or not default on their loan 94.14% of the time.
For the Step AIC model, the results yielded an optimal cut off point of 0.29 for this model and an accuracy of 93.86% on the training data.
Cross validation performed on the interaction terms Model yielded an optimal cut off probability of 0.52 with an accuracy level on the training data of 94.86%.
5-fold cross validation performed on the Neural Network to determine the optimal cut off point for determining the model’s accuracy yielded an optimal cut of point of 0.48 with an accuracy of 94% for the training data.
Fig 19: Plot of optimal cut-off determination Logistic and Neural Network Models
Five Fold Cross-Validation was performed to determine optimal cut off points for each decision tree. the results of the Cross-Validation can be seen below. It can be observed that each tree yields several optimal cut off points. The average of the cut-off points was calculated to determine the final cut off point for each tree. The cut-off points and accuracy on the testing data are as follows: -Tree with Gini Index: Non-Penalization: 0.5, 91.71% -Tree with Entropy Non-Penalization: 0.473, 91.57% -Tree with Gini Index: Penalization False Negatives: 0 .42, 90.29% -Tree with Entropy: Penalization False Negatives: 0.605, 93.29% -Tree with Gini Index: Penalization False Positives: 0.632, 88.43% -Tree with Entropy: Penalization False Positives: 0.4483, 92%
Fig 20: Plot of optimal cut-off determination Decision Tree Models
The table below shows the accuracy of each candidate model using the optimal cut off points from cross validation on the test data set. This approach provides an indication of how each candidate model performs on a new set of data in terms of predicting whether someone will default on a loan. The full model is most accurate with an accuracy of 94%, which means the model accurately predicted whether someone will default or not default on their loan 94% of the time. Behind the full model, the Step AIC model and Interaction Terms Model were accurate 93.67% of the time. These results indicate that the addition of interaction terms to the Step AIC model does not add anything in terms of model accuracy.
| Model | Percent.Correct |
|---|---|
| Full Model | 94.00000 |
| StepAIC Model | 93.66667 |
| Interaction Terms Model | 93.66667 |
| Neural Network Model | 91.66667 |
| Tree with Gini Index: Non-Penalization | 92.00000 |
| Tree with Entropy Non-Penalization | 93.00000 |
| Tree with Gini Index: Penalization False Negatives | 88.00000 |
| Tree with Entropy: Penalization False Negatives | 91.66667 |
| Tree with Gini Index: Penalization False Positives | 88.33333 |
| Tree with Entropy: Penalization False Positives | 89.33333 |
Area Under the Curve and Receiver Operator curves are methods to determine how well a model performs in terms of discriminating between someone defaulting on their loan. Higher areas under the curve (AUC) indicate better discrimination. Much like accuracy, the full logistic regression model yielded the highest AUC on the test data set indicating the best discrimination among candidate models.
Fig 22. AUC curves of Logistic Regression Models and Neural Network Models
The Neural Network Model had the second best discrimination with an AUC of 0.9776, and the Step AIC model had an AUC of 0.9771. Both of these models are considered to have excellent discrimination in determining whether someone will default on their loan. The tree with the highest AUC was the Tree with Entropy: Penalization False Negatives.
Figure 23. Comparison of ROC curves Decision Tree Models
5. Discussion
After review of each candidate model and their performance on a new testing dataset, the Step AIC model is the most desirable to predict whether someone will default on their loan. The Step AIC Model performed like the full model in terms of accuracy and discrimination as shown through the method of cross-validation and the area under the receiver operator curve. Little performance was lost in the Reduced Stepwise Model from paring down from fifteen variables to eight variables. It may be tempting to implement a full model to predict whether someone will default on their loan, however the principle of parsimony should be adhered to in this instance. The simplicity of the Step AIC model will facilitate greater efficiency and will reduce any risk of over fitting the data; something that may arise if the full logistic regression model is implemented. Although the Reduced Stepwise Model with Interaction Terms performed as well as the Step AIC model in terms of accuracy, it did not perform as well in terms of discrimination. The Reduced Stepwise Model with Interaction terms does not improve upon the performance of the Reduced Stepwise Model by including interaction terms into the Reduced Stepwise Model. A likely explanation is that the introduction of interaction terms may have over-fitted the model to the training data.
The Neural Network had better discrimination than the Step AIC Model, however the difference was negligible at 0.0005. The neural network model underperformed the Step AIC in accuracy by 2%, a much greater difference.
The decision tree models provide nice visual interpretations of the factors that may lead to defaulting on a loan, however the decision tree models did not perform as well as the Step AIC Logistic Regression model in predicting whether someone will default on their loan.
The Step AIC logistic regression model can be used as a tool to predict whether someone will default on their loan. This model can be employed when an applicant applies for a loan. The applicant will enter their Checking Account Balance, age, Savings Account Balance, and requested loan term. Information such on Credit Score, car loans, existing personal loans, and a home loan will be pulled from credit reporting agencies. This information will be plugged in to the logistic regression model and will predict whether someone will default on the loan. If the model predicts an applicant will default, then that applicant should be denied the loan.
Another potential step on the back end is that when the information is inserted into the logistic regression model, an acceptable loan term could be determined that will yield a result of the applicant not defaulting on their loan. If no acceptable term is found, then the applicant will be denied the loan.
When incorporating a logistic regression model building approach to determine who will and will not default on their loan, a reduced model incorporating the variables Checking_amount, Term, Credit_score, Car_loan, Personal_loan, Home_loan, Saving_amount, and Age proves to be the most parsimonious and effective model in terms of prediction and association.