I. Executive Summary

Banking institutions can better leverage funds deposited if they are confident the funds will remain in the account longer than traditional deposit accounts. This study seeks to isolate the best performing model to predict whether a client will subscribe to a bank term deposit as a result of a telemarketing campaign. The insights from this study can also lead to an increase in success in future campaigns by indicating the most important predictors.

For this study, the SVM model provided the highest balanced accuracy when predicting whether a client will subscribe to a bank term deposit. To evaluate the success of our model, we considered various alternative metrics rather than accuracy alone, due to the unbalanced nature of the data. The alternative metrics considered included kappa score, precision, and balanced accuracy.

When comparing our models that employed balancing techniques, we saw an increase in performance for balanced accuracy, kappa scores, and specificity. The most significant predictors were number of employees, previous outcome, month, age, and contact. These predictors may indicate that the characteristics of the telemarketing campaign are important to consider and may explain a large portion of the overall success of a campaign.

II. The Problem

This case study includes data collected throughout a direct marketing campaign by a Portuguese banking institution for term deposits. Term deposits are a type of deposit where a customer’s money is locked within the institution, or cannot be withdrawn without penalty, for an agreed upon period that ranges from months to years. While banking institutions already lend funds to customers or businesses based on normal deposits from clients, term deposits (including certificates of deposit (CDs) and time deposits) allow banking institutions a degree of security on how much funds are available for the institution to lend out to other customers. In return, customers that participate in term deposits enjoy a higher interest rate than normal savings accounts. Ultimately, the more customers that subscribe to term deposits, the more security the banking institution has in lending those funds out to other customers as compared to accounts that can be withdrawn from at any time.

The purpose of this case study is to find the best model to predict whether a client will subscribe to a bank term deposit. This will be based on a variety of input variables, campaign characteristics, including number of contacts performed during the campaign and whether the previous marketing campaign was successful or not, and social/economic attributes. By predicting the likelihood of clients to subscribe to a term deposit, we can isolate the most relevant customer characteristics in order to improve the success of future marketing campaigns by targeting specific audiences. By improving future marketing campaigns, the banking institution can enjoy a higher success rate of the marketing campaign and allocate appropriate resources to the campaign to ultimately secure more subscriptions to term deposits. By securing more subscriptions to time deposits, the institution may lend out funds to other customers with less risk.

In summary, we seek to find the best model to predict whether a client will subscribe to a term deposit and isolate the most significant customer characteristics that affect successful prediction. Finally, we will test the model against our testing data to evaluate the chosen model. The report will contain a review of existing literature, describe our methodology and data processing steps, then evaluate findings and discuss conclusions and recommendations.

IV. Methodology

In this case study we will be comparing the results of a Support Vector Machine (SVM) model and logistic regression model in the form of a Generalized Linear Model (GLM). Based on the comparison, we will suggest the best model with the highest precision and balanced accuracy metrics. The data was obtained from the UCI machine learning repository website. We have used secondary data for this case study. The data is related to direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, to assess if the product (bank term deposit) would be (‘yes’) or not (‘no’) subscribed.

The data will be split with a train-test split ratio of 70/30. The training data set contains 2667 observations and 17 variables after data preprocessing. The test data set 1144 observations with the same variables as the training data set. Additionally, the data is unbalanced with significantly more “no” target variables (3668) compared to the “yes” target variables. Due to this imbalance, there may be a bias in the algorithm towards the “no” target variable if the model is trained on the imbalance data. In order to evaluate the significance of the imbalance data on model performance, we employ two balancing techniques, up-sampling or oversampling and down-sampling or undersampling on our models and compare the model performance to the models trained on the unbalanced data. These balancing techniques will either under- or over-sample the minority group, in this case, the “yes” target variable for a more balanced ratio with the majority group. The downsampling technique simply undersamples the majority group to meet the desired balanced ratio. The upsampling technique creates synthetic observations of the minority class in order to balance the ratio. Additionally a repeated cross validation was performed and repeated three times.

Support vector machines (SVM) are a supervised learning method used to perform binary classification on data. They are motivated by the principle of optimal separation, the idea that a good classifier finds the largest gap possible between data points of different classes. In machine learning, support vector machines are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. However, they are mostly used in classification problems. SVMs can be defined as linear classifiers under the following two assumptions: The margin should be as large as possible. And the support vectors are the most useful data points because they are the ones most likely to be incorrectly classified. SVM models also have limitations like long training time for large datasets. It is difficult to understand and interpret the final model, variable weights, and individual impact. Since the final model is not so easy to see, we cannot do small calibrations to the model hence it’s tough to incorporate our business logic.

Logistic regression method can be used to fit the regression model when the response variable is binary. In this case the response variable whether the customer subscribed for term deposit or not, is binary in yes or no. Logistic regression has some assumptions such as no two input variables can be highly correlated with each other. The logistic model operates under several fundamental assumptions that contribute to the accuracy of the results of the model. First, binary logistic regression requires the dependent variable to be binary. Second, logistic regression requires the observations to be independent of each other. In other words, the observations should not come from repeated measurements or matched data. Third, logistic regression requires there to be little or no multicollinearity among the independent variables. Fourth, logistic regression assumes linearity of independent variables and log odds. Finally, logistic regression typically requires a large sample size. The major limitation of Logistic Regression is the assumption of linearity between the dependent variable and the independent variables. It not only provides a measure of how appropriate a predictor (coefficient size) is, but also its direction of association (positive or negative).

We evaluate two main models, SVM and GLM with unbalanced and balanced data, comparing precision and balanced accuracy metrics. Feature importance was also evaluated and plotted to identify the optimal parameters contributing to the model. Feature selection was also performed but not utilized in our models.

V. Data

The provided bank marketing data set contained 4119 observations and 21 variables. There are 9 categorical variables (job, marital, education, default, housing, loan, contact, month, day of week and outcome) and 10 numerical variables (age, duration, campaign, pdays, previous, employment variation rate, consumer price index, consumer confidence index, euribor 3 month rate and number of employees). The desired target variable is binary with ‘yes’ or ‘no’ responses indicating whether the client has subscribed a term deposit.

Missing values were indicated as ‘unknown’ and were present in six variables including the variables job, marital, education, default, housing and loan. The default variable had 803 missing values and represents 19% of the observations. Therefore, for this variable, we included the ‘unknown’ values as a level when converting the variable as a factor. The remaining variables had missing values ranging from 11-167 and represented 0-4% of the observations. Due to the low number of missing values for these variables, we decided to omit these observations resulting in 3811 observations and representing 93% of the original dataset.

The categorical variables were converted to factors. For the numerical variables (list numerical variables), we evaluated for highly correlated variables, near zero variance and linear dependence. We identified three highly correlated variables with a cutoff value of 0.7 (euribor 3 month rate, employment variation rate and number of employees). Due to high correlation, euribor 3 month rate and employment variation rate were removed while keeping the number of employees variable. We also identified non zero variance in the pdays variable which indicates how many days since last contact. The majority of the observations (3959) were recorded as 999 which indicates that the customer has not been previously contacted. In addition to non zero variance, we identified some possible errors related to this variable. For example, there were several (436) observations that had recorded 999 in pdays but had values greater than zero recorded for the previous variable indicating the customer had been contacted previously. The previous and poutcome variable which indicates whether the last contact was a success or failure are concordant. Due to the non-zero variance and discordance, the pdays variable was removed. For linear dependency, we did not find definite linearly dependent variables. However, we suspect that there is some dependency between the variables consumer price index and consumer confidence index that may interfere with the models. We evaluated the best model with either or both of these models.

Additional pre-processing to standardize the data included scaling and centering for the numerical variables. The caret package was used for data preprocessing including data standardization and removing the correlated and non-zero variance variables. To address possible issues with unbalanced data, the models were also evaluated using two balancing techniques, upsampling and downsampling and compared to models using unbalanced data. These techniques were implemented with the train control function of the caret package.

VI. Findings

We evaluated two models, the generalized linear model (GLM) and the support vector machine (SVM) model with both balanced and unbalanced data. Overall, the SVM model tended to slightly outperform the GLM model in almost all metrics.

Comparing the performance for the balanced and unbalanced data, the accuracy with the unbalanced data was higher compared to the models that utilized balancing techniques. However, due to the unbalanced nature of the data, alternative metrics such as the kappa score, precision and balanced accuracy are more appropriate measures of performance. Using these metrics, we see that the balancing techniques increase performance of these models with significantly higher specificity as well as increased precision, balanced accuracy and kappa scores. Among the balancing techniques, upsampling resulted in higher balanced accuracies for both models compared to downsampling. The difference between the precision of the SVM and GLM models trained with upsampling was only 0.08%. However, the balanced accuracy for the SVM model was 72.7% compared to the 71.6% for the GLM model. Therefore, we chose SVM utilizing upsampling as our final model.

## Resample01 Resample02 Resample03 Resample04 Resample05 Resample06 Resample07 
##       2667       2667       2667       2667       2667       2667       2667 
## Resample08 Resample09 Resample10 
##       2667       2667       2667

Figure 1. Comparison of accuracy and kappa scores for GLM and SVM models trained with balanced and unbalanced data.

Table 1. Performance matrix for GLM and SVM models trained on unbalanced and balanced data.

##                 Sensitivity Specificity Precision    Recall        F1
## glm unbalanced    0.9843137   0.2338710 0.9135578 0.9843137 0.9476168
## glm upsampled     0.8147059   0.6048387 0.9443182 0.8147059 0.8747368
## glm downsampled   0.7950980   0.6129032 0.9441211 0.7950980 0.8632251
## svm unbalanced    0.9774510   0.2177419 0.9113346 0.9774510 0.9432356
## svm upsampled     0.8500000   0.6048387 0.9465066 0.8500000 0.8956612
## svm downsampled   0.8901961   0.5080645 0.9370485 0.8901961 0.9130216
##                 Balanced Accuracy
## glm unbalanced          0.6090923
## glm upsampled           0.7097723
## glm downsampled         0.7040006
## svm unbalanced          0.5975965
## svm upsampled           0.7274194
## svm downsampled         0.6991303

Table 2. Confusion matrix for final model (SVM trained on upsampled data).

as.table(svm.up.mat)
##           Reference
## Prediction  no yes
##        no  867  49
##        yes 153  75

Due to the suspicion that there may be some dependency associated with consumer price index and consumer confidence index, we re-evaluated the final model by removing each of these variables and comparing the performance. The precision essentially did not change and there was no significant change in the balanced accuracy for each of these models (performance metrics shown below).

Table 3. Performance metrics for final model with the removal of each of the consumer indexes (price and confidence).

##         Sensitivity Specificity Precision    Recall        F1 Balanced Accuracy
## model 1   0.8127451   0.1854839 0.8913978 0.8127451 0.8502564         0.4991145
## model 2   0.7024631   0.6976744 0.9481383 0.7024631 0.8070175         0.7000687
## model 3   0.8500000   0.6048387 0.9465066 0.8500000 0.8956612         0.7274194

Feature importance was evaluated on the logistic regression model and showed number of employees and successful outcome of previous contact to be the most important factors for this model. The top ten important features are shown below. Feature selection was also performed but not utilized in our models and showed similar results. Feature selection showed similar results with the first five predictors listed as number of employees, previous outcome, month, age, and contact.

Figure 2. Top 10 important features associated with the logistic regression model.

VII. Conclusions and Recommendations

Our study showed the SVM model to have superior performance compared to the GLM model which is consistent with previous research. We also showed that training the model with balancing techniques increased performance for both models, with the oversampling technique achieving the best model performance. We also found the number of employees and successful previous outcomes to be the most important predictors in our model. This was consistent with feature selection which also showed these predictors to be important.

Optimization of performance metrics may be achieved with additional work including evaluating models with feature selection. We saw that removing either consumer price index or consumer confidence index from the model did not significantly change the performance of the model. This is consistent with our feature selection and feature importance evaluations in which these predictors were not ranked as highly important predictors. Additional studies to evaluate the optimal number of predictors with the most important features can be performed to optimize model performance. The SVM model can also be fine tuned with a grid search of hyperparameters that may also increase model performance.

In addition to the oversampling and undersampling techniques for balancing data, there are techniques such as synthetic minority over-sampling technique (SMOTE) and random over-sampling examples (ROSE). These techniques could also be used to compare model performance.

References

Appendix

Data Preprocessing

Missing Data Evaluation 

## 
##  Variables sorted by number of missings: 
##        Variable       Count
##         default 0.194950231
##       education 0.040543821
##         housing 0.025491624
##            loan 0.025491624
##             job 0.009468318
##         marital 0.002670551
##             age 0.000000000
##         contact 0.000000000
##           month 0.000000000
##     day_of_week 0.000000000
##        duration 0.000000000
##        campaign 0.000000000
##           pdays 0.000000000
##        previous 0.000000000
##        poutcome 0.000000000
##    emp.var.rate 0.000000000
##  cons.price.idx 0.000000000
##   cons.conf.idx 0.000000000
##       euribor3m 0.000000000
##     nr.employed 0.000000000
##               y 0.000000000

Highly Correlated & Near Zero Variance Variables

#Highly correlated & near zero variance variables
correlationMatrix = cor(d[c(1,11:14, 16:20)])

highlyCorrelated = findCorrelation(correlationMatrix, cutoff=0.70, verbose = TRUE)

nearZeroVar(d2, names = TRUE, saveMetrics = TRUE)

Models used for comparison

#models
#SVM
set.seed(10)
svm.fit.down = train(y~., data = trainTransformed, method = "svmLinear", trControl = train_control_down)
svm.fit.up = train(y~., data = trainTransformed, method = "svmLinear", trControl = train_control_up)
svm.fit.none = train(y~., data = trainTransformed, method = "svmLinear", trControl = train_control_none)

pred.model1 = predict(svm.fit.up, testTransformed)
svm.up.mat = confusionMatrix(pred.model1,factor(testTransformed$y))

pred.model2 = predict(svm.fit.down, testTransformed)
svm.down.mat = confusionMatrix(pred.model2,factor(testTransformed$y))

pred.model3 = predict(svm.fit.none, testTransformed)
svm.none.mat = confusionMatrix(pred.model3,factor(testTransformed$y))



#glm
glm.fit.down = train(y~., data = trainTransformed, method = "glm", trControl = train_control_down)
glm.fit.up = train(y~., data = trainTransformed, method = "glm", trControl = train_control_up)
glm.fit.none = train(y~., data = trainTransformed, method = "glm", trControl = train_control_none)


pred.model4 = predict(glm.fit.up, testTransformed)
glm.up.mat = confusionMatrix(pred.model4,factor(testTransformed$y))

pred.model5 = predict(glm.fit.down, testTransformed)
glm.down.mat = confusionMatrix(pred.model5,factor(testTransformed$y))

pred.model6 = predict(glm.fit.none, testTransformed)
glm.none.mat = confusionMatrix(pred.model6,factor(testTransformed$y))

#SVM without consumer price index
svm.fit.down.no.price = train(y~., data = trainTransformed1, method = "svmLinear", trControl = train_control_down)
svm.fit.up.no.price = train(y~., data = trainTransformed1, method = "svmLinear", trControl = train_control_up)

pred.model7 = predict(svm.fit.up.no.price, testTransformed1)
svm.no.price.mat = confusionMatrix(pred.model7,factor(testTransformed$y))


#SVM without consumer confidence index
set.seed(10)
svm.fit.down.no.conf = train(y~., data = trainTransformed2, method = "svmLinear", trControl = train_control_down)
svm.fit.up.no.conf.mat = train(y~., data = trainTransformed2, method = "svmLinear", trControl = train_control_up)

pred.model8 = predict(svm.fit.up.no.conf, testTransformed2)
svm.no.conf.mat = confusionMatrix(pred.model8,factor(testTransformed2$y))

#Comparing models Accuracy & Kappa Scores with resampling
Resamples<-createResample(y=trainTransformed$y, time=10, list=T)
sapply(Resamples, length)

results <- resamples(list(glm=glm.fit.up, svm=svm.fit.up))
summary(results)