1.1 Background

Bankruptcy prediction is of great importance in economic decision making, which aims to assess the financial condition of a company and its future perspectives within the context of longterm operation on the market. Early attempts of bankruptcy prediction are based on traditional statistical methods such as Logistic Regression or Discriminant Analysis. Later on, artifical intelligence techniques including Support Vector Machine and Neural Network were utilized. Recently, ensemble methods such as Bagging or Boosting gain popularity and perform well on bankruptcy prediction. Nevertheless, the imbalance nature of bankruptcy prediction makes the task not easy. This report is an empirical study of bankruptcy prediction based on one real world data set, mainly focusing on tackling imbalance and the comparison of different methods.

1.2 Data Set

The data set is Polish companies bankruptcy data, which contains 5910 observations and 65 variables. The first 64 variables are the predictors, including all sorts of companies’ current financial status and properties such as profit, liabilities, and working capital against sales as well as total assets. The 65th variable is the response, with values “Yes” and “No” reflecting whether or not the company will be bankrupt the next year. Only 410 out of 5910 observations are bankrupt, indicating the data is highly imbalanced. Moreover, there exist 4666 missing values in the predictors, so an imputation technique should be performed.

1.3 Objective

First of all, some preprocessing techniques are applied on the original data set and then based on the preprocessed data set, we make some visualization. Next, we divide the data set into training set (70%) and testing set (30%). Different sampling methods are performed on the training set to tackle imbalance, which are then used to construct several different classifiers including Linear Discriminant Analysis, Logistic Regression, K-Nearest Neighbors, Support Vector Machine, Neural Network and some ensemble classifiers including Random Forest, Adaptive Boosting (AdaBoost), Gradient Boosting Machine (GBM), with Decision Tree being base classifier. Then, based on evaluation metrics such as accuracy, sensitivity, specificity and AUC, we compare the performance of different classifiers on testing set.

1.4 Organization of the report

The rest of this report will be presented as the following order:

• Preprocessing and Visualization. To deal with missing values in the original data set, we implement Random Forest Imputation. Then, some visualization of the preprocessed data set are applied.

• Classification without Tackling Imbalance. We construct several classifiers both with the original feature space and principal component space, and compare their performance based on accuracy, sensitivity and specificity.

• Classification with Tackling Imbalance. In addition to adding a new variable numNA recording the number of missing values, we implement three sampling techniques and compare the performance of different classifiers.

• Discussion. Based on the performance of different classifiers using different sampling methods, we first make a comparison of the three sampling techniques, and then select the classifier that is most suitable for bankruptcy prediction.

2.1 Preprocessing

In the original data set, there exist 4666 missing values, where Attr37 has the largest missing rate (43.11%), while other variables has much smaller missing rate (less than 7%). We implement simple deletion for Attr37 and multiple imputation for other variables. Compared with other popular multiple imputation methods including K-Nearest Neighbours Imputation and Multivariate Imputation using Chained Equations, Random Forest Imputation which uses a random forest trained on the observed values of a data matrix to predict the missing values, is more suitable for tackling high-dimensional data especially when there exist complex interactions among variables. Therefore, considering the high-dimensional nature of the data set, we decide to apply Random Forest Imputation. Specifically we impute missing values with all predictors excluding Attr37. Also the class information is not used in the imputation, as we assume that the true values of the missing data do not depend on class.

2.2 Visualization

The left panel of Figure 1 is a visualization of the correlation matrix of 63 predictors in the preprocessed data set, where different colors correspond to different correlation level. From this plot, we find that some of the predictors are highly correlated, indicating that some dimension reduction techniques should be implemented before classification.

The right panel of Figure 1 is a collection of boxplots for 63 predictors with respect to different class labels, where outliers are removed for better visualization. From this plot, we find that for different class labels, there does not exist obvious difference in distribution for most of the predictors, indicating that single predictor could not distinguish class label.

There are numerous dimension reduction techniques that could be applied for visualization, among which Principal Component Analysis and Multidimensional Scaling are most popular.

First, we visualize the observations in the preprocessed data set with Principal Component Analysis. From the left panel of Figure 2 which projects observations onto the first two principal components, we find that observations of different class labels are not well separated, indicating that classification cannot be implemented on low-dimensional space.

Second, we visualize the predictors in the preprocessed data set with Multidimensional Scaling. From the right panel of Figure 2 which projects predictors onto the two-dimensional space, we find that there exist some clusters, indicating that some of the predictors are highly correlated, which further suggests that some dimension reduction techniques should be implemented before classification.

3.1 Without PCA

The classification methods in this section include LDA, logistic regression, kNN, SVM, neural network, random forest, AdaBoost, and gradient boosting machine. The accuracy, sensitivity (true positive rate) and specificity (true negative rate) of predictions on training and test sets are reported to compare the performance. Sensitivity and specificity are calculated with bankrupt companies being the positive class.

training and test accuracy

	train.accu	train.sens	train.spec	test.accu	test.sens	test.spec
LDA	0.9340	0.1568	0.9919	0.9278	0.1545	0.9855
Logistic	0.8513	0.4355	0.8823	0.8421	0.4309	0.8727
kNN	0.9330	0.1254	0.9932	0.9261	0.0894	0.9885
SVM	0.9306	0.0000	1.0000	0.9306	0.0000	1.0000
NN	0.9306	0.0000	1.0000	0.9306	0.0000	1.0000
RF	1.0000	1.0000	1.0000	0.9447	0.3577	0.9885
AdaBoost	0.9562	0.4460	0.9943	0.9380	0.3008	0.9855
GBM	0.9575	0.4390	0.9961	0.9380	0.2602	0.9885

From this table we can see that all of these classifiers have decent accuracy on both training and test sets. However, a further inspection on the sensitivity and specificity could reveal that the high accuracy is usually achieved via classifying most of the cases as positive, the extreme cases being SVM and neural network, with 0% sensitivity and 100% specificity.

Among these classifiers, random forest performs fairly well on the original data. It fits the training set perfectly and achieves some degree of accuracy at identifying the positive cases (35.77% sensitivity). AdaBoost and GBM yield similar results on test set but have lower sensitivity on training set. Logistic regression has the best sensitivity on test set (43.09%), but it also has lower specificity compared to other models.

3.2 With PCA

The first 20 principal components already take up 90% of the total variance. Now we run the same classification methods on the dataset of the PC scores.

training and test accuracy

	train.accu	train.sens	train.spec	test.accu	test.sens	test.spec
LDA	0.9299	0.0662	0.9943	0.4343	0.8780	0.4012
Logistic	0.9314	0.0767	0.9951	0.2893	0.9024	0.2436
kNN	0.9372	0.1986	0.9922	0.9306	0.0000	1.0000
SVM	0.9497	0.2822	0.9995	0.9306	0.0000	1.0000
NN	0.9449	0.2927	0.9935	0.8472	0.1463	0.8994
RF	1.0000	1.0000	1.0000	0.9216	0.0325	0.9879
AdaBoost	0.9862	0.8014	1.0000	0.9312	0.0081	1.0000
GBM	0.9446	0.2404	0.9971	0.8951	0.1220	0.9527

From the results we can see that PCA doesn’t really improve the models. For many classifiers we actually get worse accuracy. For instance random forest now has much lower sensitivity. This might be due to the fact that random forest usually doesn’t suffer from high dimension. By reducing dimension with PCA, we lose some of the variability and as a result get a worse random forest model. LDA and logistic regression both obtain high sensitivity and low specificity and low accuracy on test set.

4.1 Strategies

4.2 Results with different sampling methods

We tried most combinations of the pre-processing methods, sampling methods and different metrics mentioned above. To avoid verbosity, only the results using ROC AUC metric with numNA added to the imputed dataset are shown, since this combination usually results in the best performance for each of the classifiers tested in this project. Also, the three sampling methods are applied to the training set during the model training in order to compare their differences. Note that we are using the train function in R package caret to do all the model training. The sampling process happens in the background during the subsetting of cross-validation. So by doing the a repeated cross-validation (one of train function’s preset resampling method), we can get fairly accurate error rate estimates without the CV test sets being affected by sampling.

4.3 Conclusion

Based on all three sampling methods, the SMOTE methods has a best resistance on the imbalanced data. And Random Forest, AdaBoost and GBM have the overall best performance in dealing with the imbalanced data. Among those three classification methods, GBM is the most stable method and remains high overall accuracy rate, sensitivity, and specificity in terms of training and testing set.

5.1 Comparison across the three sampling techniques

Comparing the results in section 3 and section 4, we can observe that the performance of classfiers receive massive improvements after implementing sampling techniques, ROC metric, and the addition of new variable numNA. But of course the results vary with different sampling techniques used in the model. Next we discuss the different behaviors of each of the sampling techniques in this classification problems.

The advantage of upsampling is that it retains all the information in the training set. Also since the positive cases are replicated around 15 times, one single misclassification of a positive case will result in 15 errors. So essentially oversampling assigns a higher cost on false negative errors in the training set. But bear in mind there’s no such replication in the test set. So the replication of minority cases could lead to overfitting. This can be seen in Table 5. For the RF and AdaBoost models, the training accuracy, sensitivity and specificity are almost perfect, yet the test sensitivity is far lower than that of using SMOTE sampling, despite some improvement on test accuracy and specificity. In addition, using oversampling in this problem almost doubles the sample size. So the increase in computational burden is also quite considerable.

In contrast, downsampling offers computational ease. And there is no replication so overfitting is usually not a problem. From Table 7 and Table 8, we can see that models with downsampling tend to have higher sensitivity and lower specificity compared to the other two sampling techniques. For AdaBoost and GBM models, both the CV estimates of sensitivity and specificity are around 85%. So they can be great candidates if the focus is to find models that have high sensitivity yet still retain a reasonable specificity. The disadvantage of downsampling is of course the loss of information. Although from the results in section 4.2.3, no significant drop in performance can be seen other than the slightly lower ROC and overall accuracy compared to the other two sampling methods.

SMOTE sampling is a combination of upsampling and downsampling. It uses the existing minority cases to synthesize new artificial samples, and at the same time randomly samples a subset of the majority cases. The resulting balanced dataset is usually smaller than the original dataset but larger than the output of downsampling. It combines many of the advantages of upsampling and downsampling, like computational ease and a limited loss of information. But because SMOTE synthesize new samples by doing interpolation with existing data, it can only creates samples within current minority regions. In our models, using SMOTE sampling usually results in high ROC and high overall accuracy. The sensitivity is usually not as high as that of using downsampling, but in return the specificity is not sacrificed as much, which means not too many false positives.

5.2 Model selection

Next we determine which models should be the optimal choices in application. For any binary classification problem, there is always a trade-off between the sensitivity and specificity. The choice of model can depend on the classification goal. In this case, misclassifying a will-be bankrupt company as not bankrupt is usually assumed to be much more costly than the other way around. Therefore the ideal model should be able to correctly identify as many positive cases as possible. That being said, specificity can’t be ignored either, as a low specificity will make the positive predictions less credible, an example being the logistic regression model in section 3.2, where the sensitivity on test set reaches 90% but only one twelfth of the positive predictions are true positive (confusion matrix see below). So here we are looking for classifiers that maximize sensitivity without losing too much specificity.

        pred
          Yes   No
     Yes  111   12
     No  1248  402

Based on the tables in section 4, the model with the highest cross validation average of sensitivity is the AdaBoost model using downsampling. Among all the models using downsampling, the AdaBoost model also has the highest ROC AUC average and the highest specificity. And its small cross validation SD suggests that the model’s prediction performance is quite stable. The confusion matrix indicates that about one third of the positive predictions are true positive. It’s not perfect, but still acceptable considering the imbalance of the two classes. With both sensitivity and specificity at around 85%, the AdaBoost model should be a decent classifier for this problem.

        pred
          Yes   No
     Yes  106   17
     No   239 1411

Moreover, we can take a look at the predicted class probabilities of the AdaBoost model to get a clearer idea about how well the classifier separates the two classes. Figure six is a visualization of the predicted class probabilities on training set and test set. We can see that the classfier separates the classes fairly well. For most positive cases, the AdaBoost classifier assigns very close to 1 probability. That is true for even the test set, though a few positive cases are assigned close to 0 probability. For the negative cases, there are quite a lot of false negatives. Although that is the price to pay in order to get the high sensitivity.

It is also viable to select the AdaBoost and GBM models with SMOTE sampling, since they both have very high ROC and reasonable sensitivity. In fact, their sensitivity still has room for improvement. Instead of naively using 0.5 as the decision threshold, we can choose thresholds that lead to better sensitivity based on the repeated cross-validation results. By lowering the threshold, more cases will be classified as positive, thus increasing the sensitivity. Inevitably, this would also mean a lower specificity. The task is to find the sweet spot between sensitivity and specificity.

Figure 8 shows us how sensitivity and specificity vary with the change of decision threshold. Based on this plot, one can selects the most suitable threshold according to the desired sensitivity and specificity levels. It is worth mentioning that threshold doesn’t affect the performance of AdaBoost model as drastically as it does GBM model, which implies that the AdaBoost model tend to get class probability close to 0 and 1, with not many cases in between. This is confirmed in Figure 9. We can observe that the class probability has a more coherent distribution in GBM model (notice the difference of the violin plots in these two models).

Table 9 illustrates a few instances of changing the threshold of AdaBoost model using SMOTE sampling. It is quite surprising that we can set the threshold as low as 0.05, and still be able to have sensitivity and specificity exceeding 86%, which is even higher than that of the AdaBoost model using downsampling. This shows the importance of not blindly using the default 0.05 threshold. Table 10 shows some examples for the GBM model. The performance is very close to the AdaBoost model, but slightly worse.

performance of AdaBoost under different thresholds

threshold	cv.sens	cv.spec	test.sens	test.spec
0.05	0.8647	0.8615	0.8780	0.8527
0.10	0.8374	0.8847	0.8374	0.8733
0.15	0.8186	0.8965	0.8211	0.8915
0.20	0.8040	0.9042	0.8211	0.9006

performance of GBM under different thresholds

threshold	cv.sens	cv.spec	test.sens	test.spec
0.25	0.8815	0.8394	0.8780	0.8315
0.30	0.8445	0.8620	0.8699	0.8594
0.35	0.8200	0.8807	0.8293	0.8800
0.40	0.7927	0.8959	0.8211	0.8982

In conclusion, if the goal is to find a classifier with high sensitivity and high specificity, the AdaBoost model using SMOTE sampling with a threshold at 0.05 can yield over 86% sensitivity and specificity. If more focus is placed on identifying positive cases, then a more fitting model can be derived by further tuning down the threshold. Similarly, we can also raise the threshold to get a model that gets fewer false positive predictions. In general, the model choice is not definitive. It usually requires comparison among multiple candidates to suit specific classification needs.