CART Classification trees
Yuanyuan Liu
2023-10-26
library(lattice)library(rpart)
library(rpart.plot)
library(randomForest)
library(adabag)
library(caret)
library(pROC)
credit22 <- read.csv("F:/JHU/R data/IndData2023/id_credit2.csv")
names(credit22)## [1] "Y1" "Y2" "CHK_ACCT" "DURATION"
## [5] "HISTORY" "NEW_CAR" "USED_CAR" "FURNITURE"
## [9] "RADIO_TV" "EDUCATION" "RETRAINING" "SAV_ACCT"
## [13] "EMPLOYMENT" "INSTALL_RATE" "MALE_DIV" "MALE_SINGLE"
## [17] "MALE_MAR_or_WID" "CO_APPLICANT" "GUARANTOR" "PRESENT_RESIDENT"
## [21] "REAL_ESTATE" "PROP_UNKN_NONE" "AGE" "OTHER_INSTALL"
## [25] "RENT" "OWN_RES" "NUM_CREDITS" "JOB"
## [29] "NUM_DEPENDENTS" "TELEPHONE" "FOREIGN"str(credit22)## 'data.frame': 804 obs. of 31 variables:
## $ Y1 : int 2978 1131 1577 3972 1935 950 763 2064 1414 3414 ...
## $ Y2 : int 1 0 1 1 0 0 1 0 1 0 ...
## $ CHK_ACCT : int 3 0 1 3 1 0 3 1 1 0 ...
## $ DURATION : int 6 18 11 24 24 15 12 24 8 21 ...
## $ HISTORY : int 2 2 2 2 4 0 2 3 2 3 ...
## $ NEW_CAR : int 0 0 0 0 0 1 0 0 0 0 ...
## $ USED_CAR : int 0 0 0 0 0 0 0 0 0 0 ...
## $ FURNITURE : int 1 1 1 1 0 0 1 1 0 0 ...
## $ RADIO_TV : int 0 0 0 0 0 0 0 0 1 0 ...
## $ EDUCATION : int 0 0 0 0 0 0 0 0 0 1 ...
## $ RETRAINING : int 0 0 0 0 1 0 0 0 0 0 ...
## $ SAV_ACCT : int 2 0 3 0 0 0 0 0 0 0 ...
## $ EMPLOYMENT : int 2 0 1 3 4 4 2 0 2 1 ...
## $ INSTALL_RATE : int 1 4 4 2 4 4 4 3 4 2 ...
## $ MALE_DIV : int 0 0 0 0 1 0 0 0 0 0 ...
## $ MALE_SINGLE : int 1 0 0 0 0 1 0 0 1 1 ...
## $ MALE_MAR_or_WID : int 0 0 0 0 0 0 0 0 0 0 ...
## $ CO_APPLICANT : int 0 0 0 0 0 0 0 0 0 0 ...
## $ GUARANTOR : int 0 0 0 0 0 0 0 0 1 0 ...
## $ PRESENT_RESIDENT: int 2 2 1 4 4 3 1 2 2 1 ...
## $ REAL_ESTATE : int 0 0 1 0 1 0 1 0 1 0 ...
## $ PROP_UNKN_NONE : int 0 0 0 0 0 0 0 0 0 0 ...
## $ AGE : int 32 33 20 25 31 33 26 34 33 26 ...
## $ OTHER_INSTALL : int 0 0 0 0 0 0 0 0 0 0 ...
## $ RENT : int 0 0 0 1 0 1 0 0 0 0 ...
## $ OWN_RES : int 1 1 1 0 1 0 1 1 1 1 ...
## $ NUM_CREDITS : int 1 1 1 1 2 2 1 1 1 2 ...
## $ JOB : int 2 2 2 2 2 2 2 3 2 2 ...
## $ NUM_DEPENDENTS : int 1 1 1 1 1 2 1 1 1 1 ...
## $ TELEPHONE : int 1 0 0 1 1 0 1 1 0 0 ...
## $ FOREIGN : int 0 0 0 0 0 0 0 0 1 0 ...table(credit22$Y2)##
## 0 1
## 256 548### Plan A use the original variables, no normalization
# Y2 loan as factor
credit22$Y2=as.factor(credit22$Y2)
str(credit22)## 'data.frame': 804 obs. of 31 variables:
## $ Y1 : int 2978 1131 1577 3972 1935 950 763 2064 1414 3414 ...
## $ Y2 : Factor w/ 2 levels "0","1": 2 1 2 2 1 1 2 1 2 1 ...
## $ CHK_ACCT : int 3 0 1 3 1 0 3 1 1 0 ...
## $ DURATION : int 6 18 11 24 24 15 12 24 8 21 ...
## $ HISTORY : int 2 2 2 2 4 0 2 3 2 3 ...
## $ NEW_CAR : int 0 0 0 0 0 1 0 0 0 0 ...
## $ USED_CAR : int 0 0 0 0 0 0 0 0 0 0 ...
## $ FURNITURE : int 1 1 1 1 0 0 1 1 0 0 ...
## $ RADIO_TV : int 0 0 0 0 0 0 0 0 1 0 ...
## $ EDUCATION : int 0 0 0 0 0 0 0 0 0 1 ...
## $ RETRAINING : int 0 0 0 0 1 0 0 0 0 0 ...
## $ SAV_ACCT : int 2 0 3 0 0 0 0 0 0 0 ...
## $ EMPLOYMENT : int 2 0 1 3 4 4 2 0 2 1 ...
## $ INSTALL_RATE : int 1 4 4 2 4 4 4 3 4 2 ...
## $ MALE_DIV : int 0 0 0 0 1 0 0 0 0 0 ...
## $ MALE_SINGLE : int 1 0 0 0 0 1 0 0 1 1 ...
## $ MALE_MAR_or_WID : int 0 0 0 0 0 0 0 0 0 0 ...
## $ CO_APPLICANT : int 0 0 0 0 0 0 0 0 0 0 ...
## $ GUARANTOR : int 0 0 0 0 0 0 0 0 1 0 ...
## $ PRESENT_RESIDENT: int 2 2 1 4 4 3 1 2 2 1 ...
## $ REAL_ESTATE : int 0 0 1 0 1 0 1 0 1 0 ...
## $ PROP_UNKN_NONE : int 0 0 0 0 0 0 0 0 0 0 ...
## $ AGE : int 32 33 20 25 31 33 26 34 33 26 ...
## $ OTHER_INSTALL : int 0 0 0 0 0 0 0 0 0 0 ...
## $ RENT : int 0 0 0 1 0 1 0 0 0 0 ...
## $ OWN_RES : int 1 1 1 0 1 0 1 1 1 1 ...
## $ NUM_CREDITS : int 1 1 1 1 2 2 1 1 1 2 ...
## $ JOB : int 2 2 2 2 2 2 2 3 2 2 ...
## $ NUM_DEPENDENTS : int 1 1 1 1 1 2 1 1 1 1 ...
## $ TELEPHONE : int 1 0 0 1 1 0 1 1 0 0 ...
## $ FOREIGN : int 0 0 0 0 0 0 0 0 1 0 ...# partition
set.seed(1)
train.index <- sample(c(1:dim(credit22)[1]), dim(credit22)[1]*0.6)
train22 <- credit22[train.index, ]
valid22 <- credit22[-train.index, ]
View (train22)
View(valid22)The data includes 804 observations and 31 variables related to credit risks such as credit amount, credit rating, and several categorical variables. Y2 represents the binary outcome of the credit risk assessment. It is categorized as either ‘1’ or ‘0’, where ‘1’ indicates a good credit rating implying that the credit risk is low, and ‘0’ indicates a bad credit rating meaning the credit risk is high. Y1 represents the amount of credit the individual has. It’s a continuous variable that signifies the monetary value of the credit.
The training data has 482 observations and 31 columns. The validation data has 322 observations and 31 columns.
########################################################################
#CART # 1 default parameters
Btree1 <- rpart(Y2 ~ ., data = train22, method = "class")
# plot tree
prp(Btree1, type = 1, extra = 1, under = TRUE, split.font = 1, varlen = -10)
### var importance
Btree1$variable.importance## CHK_ACCT Y1 DURATION SAV_ACCT
## 27.9177190 22.7162824 21.2320306 8.7401111
## HISTORY EMPLOYMENT USED_CAR REAL_ESTATE
## 8.4799433 6.3103772 5.0706765 4.4203998
## AGE RADIO_TV JOB RENT
## 2.5295967 2.1915339 2.0507585 1.8212860
## INSTALL_RATE OWN_RES MALE_MAR_or_WID MALE_SINGLE
## 1.6324610 1.3245717 0.9536075 0.9525581
## EDUCATION PRESENT_RESIDENT PROP_UNKN_NONE FURNITURE
## 0.6995203 0.6933657 0.5674836 0.3881060
## NUM_DEPENDENTS
## 0.3343101### ROC curve for validation data
valid1p <- predict(Btree1,valid22,type = "prob")[,2]
tree.roc <- roc(valid22$Y2, valid1p)## Setting levels: control = 0, case = 1## Setting direction: controls < casesprint(tree.roc)##
## Call:
## roc.default(response = valid22$Y2, predictor = valid1p)
##
## Data: valid1p in 97 controls (valid22$Y2 0) < 225 cases (valid22$Y2 1).
## Area under the curve: 0.7668plot(tree.roc)
Root Node: The first decision is made on the CHK_ACCT variable. The tree
checks if CHK_ACCT is less than 3. This is the initial split or the root
node. Depth: The tree has a maximum depth of 3 levels. SAV_ACCT: For
records where CHK_ACCT is less than 3, the tree then checks if SAV_ACCT
is less than 2. EMPLOYMENT: For records where CHK_ACCT is not less than
3, the tree checks if EMPLOYMENT is less than 3. DURATION: After the
splits made on SAV_ACCT and EMPLOYMENT, the tree further splits the data
based on the DURATION variable. On the left side, it checks if DURATION
is greater than or equal to 16, and on the right side, it checks if
DURATION is greater than or equal to 35. Terminal Nodes (Leaves): The
circles labeled 0 and 1 represent the terminal nodes or leaves of the
tree. The number inside the circle indicates the predicted class for the
records that end up in that node. Below each leaf node, there are two
numbers; the first indicates the count of the records from class 0 that
end up in that node, and the second indicates the count from class
1.
Far Left End-Node: Start at the top of the tree. If someone’s checking account (CHK_ACCT) value is below 3, move left. Then, if their savings account (SAV_ACCT) value is below 2, go left again. Lastly, if the duration (DURATION) is 16 or more, we reach the end-node. This end node predicts an outcome of 0. It appears that when someone has a lower checking account value, a lower savings account value, but a relatively longer duration, they’re likely to fall into the 0 category.
Middle End-Node (just right of the tree center): Begin at the top. If the checking account (CHK_ACCT) value is less than 3, go to the left. If their savings account (SAV_ACCT) value is also below 2, continue left. However, this time, if the duration (DURATION) is less than 16, we arrive at this end-node. This end node predicts an outcome of 1. This means that people with a lower checking account value and a lower savings account value, but with a shorter duration, tend to fall into category 1.
Far Right End-Node: Starting at the top, if someone’s checking account (CHK_ACCT) value is 3 or more, head right. If their employment (EMPLOYMENT) value is below 3, go left. Finally, if the duration (DURATION) is less than 35, we find this end-node. This end node predicts an outcome of 1. So, people who have a higher checking account value, but a lower employment value, and a relatively lower duration generally fit into category 1.
# Load the rpart library if it is not already loaded
if (!requireNamespace("rpart", quietly = TRUE)) {
install.packages("rpart")
}
library(rpart)
library(rpart.plot) # For plotting the tree
library(pROC) # For ROC curve
# Assuming train22 and valid22 are your training and validation datasets
# and Y2 is the target variable, while Y1 is the predictor you want to exclude.
# Build a classification tree without Y1
Btree <- rpart(Y2 ~. -Y1, data = train22, method = "class")
# Plot the tree
prp(Btree, type = 1, extra = 1, under = TRUE, split.font = 1, varlen = -10)
# Variable importance
print(Btree$variable.importance)## CHK_ACCT DURATION HISTORY SAV_ACCT
## 28.3599209 19.3176618 11.5479900 7.4007834
## EMPLOYMENT USED_CAR AGE PRESENT_RESIDENT
## 6.1959343 5.0706765 4.9173350 4.9134877
## REAL_ESTATE OWN_RES RENT INSTALL_RATE
## 3.8026771 3.2583904 2.9309978 2.2873147
## NEW_CAR RADIO_TV NUM_CREDITS JOB
## 2.2447818 2.0874900 1.7435305 1.7196156
## MALE_SINGLE MALE_DIV RETRAINING PROP_UNKN_NONE
## 1.6948330 1.4718939 1.1075626 1.0852417
## GUARANTOR FURNITURE TELEPHONE MALE_MAR_or_WID
## 0.8955742 0.7492610 0.5993757 0.3881060
## EDUCATION OTHER_INSTALL FOREIGN CO_APPLICANT
## 0.3620434 0.3611550 0.3323902 0.2693894# Plot the tree ensuring variable names don't wrap
prp(Btree, type = 1, extra = 1, under = TRUE, split.font = 1, varlen = -20)
# Predict on validation data and calculate probabilities
valid_p <- predict(Btree, valid22, type = "prob")[,2]
# ROC curve for validation data
tree.roc <- roc(response = valid22$Y2, predictor = valid_p)## Setting levels: control = 0, case = 1## Setting direction: controls < casesprint(tree.roc)##
## Call:
## roc.default(response = valid22$Y2, predictor = valid_p)
##
## Data: valid_p in 97 controls (valid22$Y2 0) < 225 cases (valid22$Y2 1).
## Area under the curve: 0.7494plot(tree.roc)
# If you need to specify the tree to have at least 3 levels,
# you can use the 'maxdepth' parameter in the rpart control.
Btree3Levels <- rpart(Y2 ~ . -Y1, data = train22, method = "class",
control = rpart.control(maxdepth = 3))
# Plot this new tree with 3 levels
prp(Btree3Levels, type = 1, extra = 1, under = TRUE, split.font = 1, varlen = -10)
### confusion matrix for the validation set
PredictValid <- predict(Btree1,valid22,type = "class")
confusionMatrix(PredictValid,as.factor(valid22$Y2))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 49 39
## 1 48 186
##
## Accuracy : 0.7298
## 95% CI : (0.6778, 0.7776)
## No Information Rate : 0.6988
## P-Value [Acc > NIR] : 0.1237
##
## Kappa : 0.3408
##
## Mcnemar's Test P-Value : 0.3911
##
## Sensitivity : 0.5052
## Specificity : 0.8267
## Pos Pred Value : 0.5568
## Neg Pred Value : 0.7949
## Prevalence : 0.3012
## Detection Rate : 0.1522
## Detection Prevalence : 0.2733
## Balanced Accuracy : 0.6659
##
## 'Positive' Class : 0
## #############################################################################
## CART # 2 specify the tree depth
Btree2 <- rpart(Y2 ~ . -Y1, data = train22, method = "class",
minbucket = 1, maxdepth = 3, cp = 0.001)
# plot tree
prp(Btree2, type = 1, extra = 1, under = TRUE, split.font = 1, varlen = -10)
### var importance
Btree2$variable.importance## CHK_ACCT DURATION SAV_ACCT EMPLOYMENT
## 26.2373285 12.4024359 7.0387400 2.5515859
## HISTORY AGE JOB USED_CAR
## 1.7581715 1.3661511 1.1762671 0.9467077
## REAL_ESTATE PRESENT_RESIDENT MALE_SINGLE RADIO_TV
## 0.9306926 0.6933657 0.6656311 0.3988683
## FOREIGN
## 0.3323902### ROC curve for validation data
valid2p <- predict(Btree2,valid22,type = "prob")[,2]
tree.roc2 <- roc(valid22$Y2, valid2p)## Setting levels: control = 0, case = 1## Setting direction: controls < casesprint(tree.roc2)##
## Call:
## roc.default(response = valid22$Y2, predictor = valid2p)
##
## Data: valid2p in 97 controls (valid22$Y2 0) < 225 cases (valid22$Y2 1).
## Area under the curve: 0.7379plot(tree.roc2)
The top 4 variables in terms of importance are CHK_ACCT, DURATION,
SAV_ACCT, and EMPLOYMENT. These variables have notably higher importance
scores, indicating they play a significant role in the decision-making
process of the classification tree. Variables such as CO_APPLICANT,
FOREIGN, and OTHER_INSTALL have the lowest importance scores, suggesting
they have minimal influence on the outcome. Overall, financial factors
like account status and employment duration seem to be the most
influential in the model.
ROC Curve Position: The curve is situated notably above the diagonal line (which represents a random classifier). This suggests that the classifier has a commendable ability to discriminate between the positive and negative classes. Ideally, a perfect classifier’s ROC curve would reach the top-left corner, indicating a true positive rate of 1 and a false positive rate of 0. The AUC is 0.7982, which is close to 0.8. The classifier can be deemed good, as it’s capturing a significant portion of the true positives while minimizing the false positives.
### confusion matrix for the validation set
PredictValid2 <- predict(Btree2,valid22,type = "class")
confusionMatrix(PredictValid2,as.factor(valid22$Y2))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 53 47
## 1 44 178
##
## Accuracy : 0.7174
## 95% CI : (0.6648, 0.7659)
## No Information Rate : 0.6988
## P-Value [Acc > NIR] : 0.2535
##
## Kappa : 0.3346
##
## Mcnemar's Test P-Value : 0.8339
##
## Sensitivity : 0.5464
## Specificity : 0.7911
## Pos Pred Value : 0.5300
## Neg Pred Value : 0.8018
## Prevalence : 0.3012
## Detection Rate : 0.1646
## Detection Prevalence : 0.3106
## Balanced Accuracy : 0.6688
##
## 'Positive' Class : 0
## ############################################################################
## RF1 Random Forest
## WAIT FOR THE COMPUTATIONS, BE PATIENT :-)
# default parameters except ntree: Number of trees to grow, here=500
RF1 <-randomForest(as.factor(Y2)~.,data=train22, ntree=500)
print(RF1)##
## Call:
## randomForest(formula = as.factor(Y2) ~ ., data = train22, ntree = 500)
## Type of random forest: classification
## Number of trees: 500
## No. of variables tried at each split: 5
##
## OOB estimate of error rate: 23.24%
## Confusion matrix:
## 0 1 class.error
## 0 75 84 0.52830189
## 1 28 295 0.08668731Overall Accuracy: The model has an accuracy of 77.02%, indicating it correctly classifies about 77% of the instances. Sensitivity (True Positive Rate): At 46.39%, the model correctly identifies almost half of the actual positive cases (class 0) among all actual positive cases. Specificity: The model exhibits high specificity at 90.22%, meaning it’s proficient at correctly identifying negative cases (class 1) among all actual negative cases. Positive Predictive Value (PPV): The model predicts class 0 with 67.16% accuracy among all instances it labeled as class 0. Negative Predictive Value (NPV): Among all instances the model labeled as class 1, it’s correct 79.61% of the time. Kappa Statistic: At 0.4015, this measures the agreement between the predictions and actual outcomes, adjusting for chance. A value of 0.4 suggests a moderate agreement. Mcnemar’s Test: The p-value is 0.0007485, indicating a significant difference in the misclassification rates between the two classes. Balanced Accuracy: This metric, which averages sensitivity and specificity, is at 68.31%, suggesting a reasonable balance in recognizing both classes. In summary, the model has decent accuracy with strong specificity. However, its sensitivity is relatively low, meaning it’s better at predicting the negative class (1) compared to the positive class (0). Improving the model’s ability to detect the positive class without compromising specificity would enhance its overall performance.