Telco Customer Churn
Chakkapong Burudpakdee
2024-09-06
Focused customer retention programs
Context
Predict behavior to retain customers. [IBM Sample Data Sets]
Content
Each row represents a customer, each column contains customer’s attributes described on the column Metadata.
The data set includes information about:
-Customers who left within the last month – the column is called Churn
-Services that each customer has signed up for – phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies.
-Customer account information – how long they’ve been a customer, contract, payment method, paperless billing, monthly charges, and total charges. -Demographic info about customers – gender, age range, and if they have partners and dependents.
##
## Attaching package: 'vip'## The following object is masked from 'package:utils':
##
## vi## Loading required package: lattice##
## ----------------------------------------------------------------------
##
## Your next step is to start H2O:
## > h2o.init()
##
## For H2O package documentation, ask for help:
## > ??h2o
##
## After starting H2O, you can use the Web UI at http://localhost:54321
## For more information visit https://docs.h2o.ai
##
## ----------------------------------------------------------------------##
## Attaching package: 'h2o'## The following objects are masked from 'package:stats':
##
## cor, sd, var## The following objects are masked from 'package:base':
##
## &&, %*%, %in%, ||, apply, as.factor, as.numeric, colnames,
## colnames<-, ifelse, is.character, is.factor, is.numeric, log,
## log10, log1p, log2, round, signif, trunc##
## Attaching package: 'dplyr'## The following object is masked from 'package:lime':
##
## explain## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union## Connection successful!
##
## R is connected to the H2O cluster:
## H2O cluster uptime: 2 hours 23 minutes
## H2O cluster timezone: America/New_York
## H2O data parsing timezone: UTC
## H2O cluster version: 3.44.0.3
## H2O cluster version age: 8 months and 17 days
## H2O cluster name: H2O_started_from_R_deviancedev01_sga668
## H2O cluster total nodes: 1
## H2O cluster total memory: 15.26 GB
## H2O cluster total cores: 24
## H2O cluster allowed cores: 24
## H2O cluster healthy: TRUE
## H2O Connection ip: localhost
## H2O Connection port: 54321
## H2O Connection proxy: NA
## H2O Internal Security: FALSE
## R Version: R version 4.4.1 (2024-06-14)## Warning in h2o.clusterInfo():
## Your H2O cluster version is (8 months and 17 days) old. There may be a newer version available.
## Please download and install the latest version from: https://h2o-release.s3.amazonaws.com/h2o/latest_stable.html## | | | 0% | |======================================================================| 100%## | | | 0% | |======================================================================| 100%Standard binomial regression machine learning for predicting customer churn:
## | | | 0% | |======================================================================| 100%## Model Details:
## ==============
##
## H2OBinomialModel: glm
## Model ID: GLM_model_R_1725646653168_7855
## GLM Model: summary
## family link regularization
## 1 binomial logit Elastic Net (alpha = 0.5, lambda = 3.105E-4 )
## number_of_predictors_total number_of_active_predictors number_of_iterations
## 1 44 27 5
## training_frame
## 1 train_obs_sid_94f6_1
##
## Coefficients: glm coefficients
## names coefficients
## 1 Intercept -1.536953
## 2 PaymentMethod.Bank transfer (automatic) 0.000000
## 3 PaymentMethod.Credit card (automatic) -0.060703
## 4 PaymentMethod.Electronic check 0.325176
## 5 PaymentMethod.Mailed check -0.034134
## standardized_coefficients
## 1 -2.681058
## 2 0.000000
## 3 -0.060703
## 4 0.325176
## 5 -0.034134
##
## ---
## names coefficients standardized_coefficients
## 40 PaperlessBilling.Yes 0.166336 0.166336
## 41 Partner.No 0.000000 0.000000
## 42 Partner.Yes 0.000000 0.000000
## 43 tenure -0.053649 -1.317123
## 44 MonthlyCharges 0.000000 0.000000
## 45 TotalCharges 0.000260 0.588327
##
## H2OBinomialMetrics: glm
## ** Reported on training data. **
##
## MSE: 0.1351985
## RMSE: 0.3676935
## LogLoss: 0.416414
## Mean Per-Class Error: 0.2368015
## AUC: 0.8463841
## AUCPR: 0.6597588
## Gini: 0.6927682
## R^2: 0.3074831
## Residual Deviance: 5699.042
## AIC: 5755.042
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## No Yes Error Rate
## No 3944 1079 0.214812 =1079/5023
## Yes 471 1349 0.258791 =471/1820
## Totals 4415 2428 0.226509 =1550/6843
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.330159 0.635122 205
## 2 max f2 0.155245 0.754427 294
## 3 max f0point5 0.545127 0.636725 112
## 4 max accuracy 0.500024 0.805641 131
## 5 max precision 0.828114 1.000000 0
## 6 max recall 0.004079 1.000000 396
## 7 max specificity 0.828114 1.000000 0
## 8 max absolute_mcc 0.340407 0.486131 200
## 9 max min_per_class_accuracy 0.303392 0.762692 218
## 10 max mean_per_class_accuracy 0.276009 0.767525 230
## 11 max tns 0.828114 5023.000000 0
## 12 max fns 0.828114 1818.000000 0
## 13 max fps 0.002040 5023.000000 399
## 14 max tps 0.004079 1820.000000 396
## 15 max tnr 0.828114 1.000000 0
## 16 max fnr 0.828114 0.998901 0
## 17 max fpr 0.002040 1.000000 399
## 18 max tpr 0.004079 1.000000 396
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`## | | | 0% | |======================================================================| 100%## Warning in plot.window(...): "medcol" is not a graphical parameter## Warning in plot.window(...): "medlty" is not a graphical parameter## Warning in plot.window(...): "staplelty" is not a graphical parameter## Warning in plot.window(...): "boxlty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "medcol" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "medlty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "staplelty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "boxlty" is not a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medcol" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "staplelty" is not
## a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "boxlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medcol" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "staplelty" is not
## a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "boxlty" is not a
## graphical parameter## Warning in box(...): "medcol" is not a graphical parameter## Warning in box(...): "medlty" is not a graphical parameter## Warning in box(...): "staplelty" is not a graphical parameter## Warning in box(...): "boxlty" is not a graphical parameter## Warning in title(...): "medcol" is not a graphical parameter## Warning in title(...): "medlty" is not a graphical parameter## Warning in title(...): "staplelty" is not a graphical parameter## Warning in title(...): "boxlty" is not a graphical parameter
## PartialDependence: Partial dependency plot for MonthlyCharges
## MonthlyCharges mean_response stddev_response std_error_mean_response
## 1 18.000000 0.265933 0.240974 0.002913
## 2 23.315789 0.265933 0.240974 0.002913
## 3 28.631579 0.265933 0.240974 0.002913
## 4 33.947368 0.265933 0.240974 0.002913
## 5 39.263158 0.265933 0.240974 0.002913
## 6 44.578947 0.265933 0.240974 0.002913
## 7 49.894737 0.265933 0.240974 0.002913
## 8 55.210526 0.265933 0.240974 0.002913
## 9 60.526316 0.265933 0.240974 0.002913
## 10 65.842105 0.265933 0.240974 0.002913
## 11 71.157895 0.265933 0.240974 0.002913
## 12 76.473684 0.265933 0.240974 0.002913
## 13 81.789474 0.265933 0.240974 0.002913
## 14 87.105263 0.265933 0.240974 0.002913
## 15 92.421053 0.265933 0.240974 0.002913
## 16 97.736842 0.265933 0.240974 0.002913
## 17 103.052632 0.265933 0.240974 0.002913
## 18 108.368421 0.265933 0.240974 0.002913
## 19 113.684211 0.265933 0.240974 0.002913
## 20 119.000000 0.265933 0.240974 0.002913## | | | 0% | |======================================================================| 100%
## | | | 0% | |======================================================================| 100%Confusion Matrix and feature explanations of regression model:
## Confusion Matrix (vertical: actual; across: predicted) for max f1 @ threshold = 0.487083290620306:
## No Yes Error Rate
## No 138 13 0.086093 =13/151
## Yes 16 33 0.326531 =16/49
## Totals 154 46 0.145000 =29/200
Random forest machine learning for predicting customer churn. 5 folds for k-fold cross-validations:
## | | | 0% | |============================================= | 64% | |======================================================================| 100%## Model Details:
## ==============
##
## H2OBinomialModel: drf
## Model ID: DRF_model_R_1725646653168_7858
## Model Summary:
## number_of_trees number_of_internal_trees model_size_in_bytes min_depth
## 1 50 50 712126 20
## max_depth mean_depth min_leaves max_leaves mean_leaves
## 1 20 20.00000 1009 1217 1125.36000
##
##
## H2OBinomialMetrics: drf
## ** Reported on training data. **
## ** Metrics reported on Out-Of-Bag training samples **
##
## MSE: 0.14816
## RMSE: 0.3849155
## LogLoss: 0.6152016
## Mean Per-Class Error: 0.2536289
## AUC: 0.8158894
## AUCPR: 0.6017352
## Gini: 0.6317788
## R^2: 0.2410916
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## No Yes Error Rate
## No 3579 1444 0.287478 =1444/5023
## Yes 400 1420 0.219780 =400/1820
## Totals 3979 2864 0.269472 =1844/6843
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.263104 0.606319 270
## 2 max f2 0.136930 0.730287 325
## 3 max f0point5 0.509307 0.586830 168
## 4 max accuracy 0.619937 0.785766 123
## 5 max precision 0.920454 0.797101 21
## 6 max recall 0.000001 1.000000 399
## 7 max specificity 0.999709 0.999204 0
## 8 max absolute_mcc 0.390921 0.443021 212
## 9 max min_per_class_accuracy 0.293902 0.738403 256
## 10 max mean_per_class_accuracy 0.263104 0.746371 270
## 11 max tns 0.999709 5019.000000 0
## 12 max fns 0.999709 1812.000000 0
## 13 max fps 0.000001 5023.000000 399
## 14 max tps 0.000001 1820.000000 399
## 15 max tnr 0.999709 0.999204 0
## 16 max fnr 0.999709 0.995604 0
## 17 max fpr 0.000001 1.000000 399
## 18 max tpr 0.000001 1.000000 399
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`
##
## H2OBinomialMetrics: drf
## ** Reported on cross-validation data. **
## ** 5-fold cross-validation on training data (Metrics computed for combined holdout predictions) **
##
## MSE: 0.1439709
## RMSE: 0.379435
## LogLoss: 0.4837232
## Mean Per-Class Error: 0.248329
## AUC: 0.8261902
## AUCPR: 0.6174212
## Gini: 0.6523805
## R^2: 0.262549
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## No Yes Error Rate
## No 3704 1319 0.262592 =1319/5023
## Yes 426 1394 0.234066 =426/1820
## Totals 4130 2713 0.255005 =1745/6843
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.281535 0.615045 258
## 2 max f2 0.172061 0.740334 310
## 3 max f0point5 0.533765 0.605780 148
## 4 max accuracy 0.533765 0.792635 148
## 5 max precision 0.952293 0.875000 9
## 6 max recall 0.000020 1.000000 399
## 7 max specificity 0.995686 0.999602 0
## 8 max absolute_mcc 0.281535 0.454653 258
## 9 max min_per_class_accuracy 0.294543 0.746765 253
## 10 max mean_per_class_accuracy 0.281535 0.751671 258
## 11 max tns 0.995686 5021.000000 0
## 12 max fns 0.995686 1817.000000 0
## 13 max fps 0.000020 5023.000000 399
## 14 max tps 0.000020 1820.000000 399
## 15 max tnr 0.995686 0.999602 0
## 16 max fnr 0.995686 0.998352 0
## 17 max fpr 0.000020 1.000000 399
## 18 max tpr 0.000020 1.000000 399
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`
## Cross-Validation Metrics Summary:
## mean sd cv_1_valid cv_2_valid cv_3_valid
## accuracy 0.746275 0.030514 0.751570 0.750376 0.738577
## auc 0.826137 0.012800 0.841292 0.824146 0.806521
## err 0.253725 0.030514 0.248430 0.249624 0.261423
## err_count 346.200000 32.321820 356.000000 332.000000 349.000000
## f0point5 0.554132 0.019297 0.567273 0.546672 0.550315
## f1 0.616688 0.012527 0.636735 0.603819 0.616062
## f2 0.697364 0.040027 0.725581 0.674307 0.699650
## lift_top_group 3.194370 0.454785 3.184444 3.843931 3.143642
## logloss 0.485089 0.073904 0.428463 0.460883 0.612494
## max_per_class_error 0.299591 0.042459 0.266539 0.268786 0.272915
## mcc 0.459087 0.016018 0.482434 0.443754 0.449684
## mean_per_class_accuracy 0.751656 0.008806 0.766731 0.744164 0.748158
## mean_per_class_error 0.248344 0.008806 0.233269 0.255836 0.251842
## mse 0.144146 0.006692 0.140243 0.143222 0.153428
## pr_auc 0.620087 0.018937 0.649848 0.619835 0.598263
## precision 0.519550 0.028085 0.528814 0.514228 0.513761
## r2 0.261579 0.023762 0.292015 0.255880 0.226346
## recall 0.766007 0.072291 0.800000 0.731214 0.769231
## rmse 0.379584 0.008794 0.374490 0.378447 0.391699
## specificity 0.737305 0.066552 0.733461 0.757114 0.727086
## cv_4_valid cv_5_valid
## accuracy 0.787964 0.702888
## auc 0.832203 0.826520
## err 0.212036 0.297112
## err_count 303.000000 391.000000
## f0point5 0.578148 0.528251
## f1 0.609032 0.617791
## f2 0.643402 0.743880
## lift_top_group 3.238527 2.561308
## logloss 0.442588 0.481016
## max_per_class_error 0.331445 0.358272
## mcc 0.468635 0.450927
## mean_per_class_accuracy 0.747846 0.751382
## mean_per_class_error 0.252154 0.248618
## mse 0.136133 0.147705
## pr_auc 0.611617 0.620874
## precision 0.559242 0.481707
## r2 0.268121 0.265531
## recall 0.668555 0.861035
## rmse 0.368961 0.384323
## specificity 0.827138 0.641728## | | | 0% | |======================================================================| 100%## Warning in plot.window(...): "medcol" is not a graphical parameter## Warning in plot.window(...): "medlty" is not a graphical parameter## Warning in plot.window(...): "staplelty" is not a graphical parameter## Warning in plot.window(...): "boxlty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "medcol" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "medlty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "staplelty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "boxlty" is not a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medcol" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "staplelty" is not
## a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "boxlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medcol" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "staplelty" is not
## a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "boxlty" is not a
## graphical parameter## Warning in box(...): "medcol" is not a graphical parameter## Warning in box(...): "medlty" is not a graphical parameter## Warning in box(...): "staplelty" is not a graphical parameter## Warning in box(...): "boxlty" is not a graphical parameter## Warning in title(...): "medcol" is not a graphical parameter## Warning in title(...): "medlty" is not a graphical parameter## Warning in title(...): "staplelty" is not a graphical parameter## Warning in title(...): "boxlty" is not a graphical parameter
## PartialDependence: Partial dependency plot for MonthlyCharges
## MonthlyCharges mean_response stddev_response std_error_mean_response
## 1 18.000000 0.256860 0.237082 0.002866
## 2 23.315789 0.257721 0.236621 0.002860
## 3 28.631579 0.258517 0.237692 0.002873
## 4 33.947368 0.260819 0.239997 0.002901
## 5 39.263158 0.261845 0.241621 0.002921
## 6 44.578947 0.257841 0.236694 0.002861
## 7 49.894737 0.253773 0.234265 0.002832
## 8 55.210526 0.253442 0.230110 0.002782
## 9 60.526316 0.249329 0.229835 0.002778
## 10 65.842105 0.248157 0.233193 0.002819
## 11 71.157895 0.260349 0.242765 0.002935
## 12 76.473684 0.273071 0.257050 0.003107
## 13 81.789474 0.278388 0.259563 0.003138
## 14 87.105263 0.281748 0.262248 0.003170
## 15 92.421053 0.298407 0.262688 0.003176
## 16 97.736842 0.320068 0.264685 0.003200
## 17 103.052632 0.326138 0.252539 0.003053
## 18 108.368421 0.331804 0.244832 0.002960
## 19 113.684211 0.334482 0.239255 0.002892
## 20 119.000000 0.346581 0.231383 0.002797## | | | 0% | |======================================================================| 100%
## | | | 0% | |======================================================================| 100%Confusion Matrix and feature explanations of random forest cross validation model:
## Confusion Matrix (vertical: actual; across: predicted) for max f1 @ threshold = 0.52335531026125:
## No Yes Error Rate
## No 141 10 0.066225 =10/151
## Yes 17 32 0.346939 =17/49
## Totals 158 42 0.135000 =27/200
Gradient boosting machine learning for predicting customer churn:
## | | | 0% | |================================================================== | 94% | |======================================================================| 100%##
## 1 function (x, y, training_frame, model_id = NULL, validation_frame = NULL,
## 2 nfolds = 0, keep_cross_validation_models = TRUE, keep_cross_validation_predictions = FALSE,
## 3 keep_cross_validation_fold_assignment = FALSE, score_each_iteration = FALSE,
## 4 score_tree_interval = 0, fold_assignment = c("AUTO", "Random",
## 5 "Modulo", "Stratified"), fold_column = NULL, ignore_const_cols = TRUE,
## 6 offset_column = NULL, weights_column = NULL, balance_classes = FALSE,## Warning in h2o.partialPlot(h2o_gbm, data = train_obs.h2o, cols =
## "MonthlyCharges"): argument 'data' is deprecated; please use 'newdata' instead.## | | | 0% | |======================================================================| 100%## Warning in plot.window(...): "medcol" is not a graphical parameter## Warning in plot.window(...): "medlty" is not a graphical parameter## Warning in plot.window(...): "staplelty" is not a graphical parameter## Warning in plot.window(...): "boxlty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "medcol" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "medlty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "staplelty" is not a graphical parameter## Warning in plot.xy(xy, type, ...): "boxlty" is not a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medcol" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "staplelty" is not
## a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "boxlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medcol" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "medlty" is not a
## graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "staplelty" is not
## a graphical parameter## Warning in axis(side = side, at = at, labels = labels, ...): "boxlty" is not a
## graphical parameter## Warning in box(...): "medcol" is not a graphical parameter## Warning in box(...): "medlty" is not a graphical parameter## Warning in box(...): "staplelty" is not a graphical parameter## Warning in box(...): "boxlty" is not a graphical parameter## Warning in title(...): "medcol" is not a graphical parameter## Warning in title(...): "medlty" is not a graphical parameter## Warning in title(...): "staplelty" is not a graphical parameter## Warning in title(...): "boxlty" is not a graphical parameter
## PartialDependence: Partial dependency plot for MonthlyCharges
## MonthlyCharges mean_response stddev_response std_error_mean_response
## 1 18.000000 0.229585 0.225148 0.002722
## 2 23.315789 0.229585 0.225148 0.002722
## 3 28.631579 0.208043 0.210135 0.002540
## 4 33.947368 0.234553 0.225506 0.002726
## 5 39.263158 0.234751 0.226492 0.002738
## 6 44.578947 0.232387 0.221859 0.002682
## 7 49.894737 0.231683 0.219880 0.002658
## 8 55.210526 0.234874 0.220069 0.002660
## 9 60.526316 0.228927 0.215844 0.002609
## 10 65.842105 0.231998 0.223316 0.002700
## 11 71.157895 0.231499 0.222260 0.002687
## 12 76.473684 0.249861 0.242216 0.002928
## 13 81.789474 0.261840 0.254410 0.003075
## 14 87.105263 0.270495 0.258223 0.003122
## 15 92.421053 0.279761 0.259351 0.003135
## 16 97.736842 0.309376 0.274628 0.003320
## 17 103.052632 0.337187 0.265805 0.003213
## 18 108.368421 0.315351 0.235186 0.002843
## 19 113.684211 0.326619 0.253333 0.003062
## 20 119.000000 0.334454 0.252253 0.003049## | | | 0% | |======================================================================| 100%
## | | | 0% | |======================================================================| 100%Confusion Matrix and feature explanations of gradient boosting model:
All models have similar output predictions when compared side by side. Observations in positive customer churning include month-to-month contracts, tenure <=18, and fiber optic internet service. Observations in customer retention include 1 year contracts, online security, and DSL internet service.
R version 4.4.1 (2024-06-14) – “Race for Your Life” Copyright (C) 2024 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu
RStudio 2024.04.2+764 “Chocolate Cosmos” Release (e4392fc9ddc21961fd1d0efd47484b43f07a4177, 2024-06-05) for Ubuntu Jammy Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) rstudio/2024.04.2+764 Chrome/120.0.6099.291 Electron/28.3.1 Safari/537.36, Quarto 1.4.555