Bookbinder Case Study Isabella Clemandot, Riley, Dunvin, Destiny Campbell
Executive Summary The study evaluated the performance of three different predictive models to identify the most effective approach for targeting a book marketing campaign. The support vector machine (SVM) model demonstrated the highest accuracy, which outperformed logistic regression. Though linear regression is inappropriate for our study due to model assumptions not being met, logistic regression and SVM identified several significant variables such as number of youth books purchased, recency of purchase, and number of art books purchased, which all contributed to whether or not a customer bought a book. Furthermore, variables such as income and education are likely to play a role in book purchasing decisions. In summary, advanced modeling techniques like SVM can help BBBC increase the effectiveness of direct mail campaigns through selective targeting. Moreover, logistic regression offers simplicity, while SVM excels in accuracy. It is important to note that future modeling should opt for logistic or linear regression based on desired complexity. In addition, further optimization is possible by simplifying and automating the modeling process.
The Problem To enhance the efficacy of its direct mail program, BBBC has experimented with predictive modeling approaches, utilizing a subset of its database. However, the company now confronts the challenge of optimizing its strategy based on the analysis of a dataset consisting of 400 customers who purchased a specific book and 1200 customers who did not, potentially over-representing the response group. The primary problem lies in identifying the key factors influencing the purchase decision, as well as refining the response model for more accurate predictions. The task that we were provided for this case study was to determine the variables that significantly impact customers’ choices, particularly their decision to purchase the book under consideration. Additionally, there is a need to evaluate the effectiveness of the current direct mail program and assess whether predictive modeling can be leveraged to enhance targeting and increase response rates. The overarching goal is to devise a data-driven approach that allows BBBC to adapt its marketing strategies, remain competitive, and better cater to the diverse preferences of its customer base in the rapidly changing book publishing landscape.
Review of Related Literature Linear regression, logistic regression (logit models), and support vector machines (SVMs) share some similarities, particularly in the context of binary classification problems. For instance, some commonalities include finding a linear decision boundary and the use of loss functions. Linear regression and logit models attempt to find a linear decision boundary that separates different classes or predicts a continuous target variable. SVMs also find a linear decision boundary but can be extended to find non-linear boundaries using kernel tricks. In addition, linear regression and logit models use different loss functions. Linear regression typically uses mean squared error, while logistic regression uses the log-likelihood. SVMs use a hinge loss function to maximize the margin between classes. While these models have similarities between them, there are also significant differences in the underlying principles and objectives of each model. Linear regression predicts a continuous output, logistic regression models the probability of belonging to a particular class, and SVMs aim to find a decision boundary that maximizes the margin between classes. Additionally, SVMs can be extended to handle non-linear relationships using kernel functions. Regarding the models’ history, linear regression has deep roots in statistics and has been widely used for modeling relationships between variables since the early development of statistical methods, as a result of Sir Francis Galton expanding on the concept in the late 19th century. Logit models emerged in the 20th century, with statisticians, such as Pierre-François Verhulst, working on developing methods for binary classification and probability estimation. Lastly, the formalization of SVMs as a machine learning algorithm came in the 1990s. Vladimir Vapnik, a computer scientist, and Alexey Chervonenkis, a mathematician, developed the theory and algorithms for SVMs, emphasizing the concept of maximizing the margin between classes.
Methodology For this case study, we looked at a linear regression model, a logistic regression model, and lastly a SVM model. For the linear model, we attempted to predict choice with all other variables. We did the same thing for logistic regression. For SVM, we created a classification model with a radial kernel. The data given to us was already split into a training and a testing set. The training data had 2,300 observations and 12 different variables. The test data had the same number of variables, but it had 1,600 observations. The variables told us when the first and last time a purchase was made along with how many this person purchased per genre. For a total of 3,900 observations, we have about a 40/60 split for testing/training. The linear regression model has several assumptions. They are as follows: the predictor and response variables must have a linear relationship, there must be little to no multicollinearity, there must be a multivariate normality, and there must be homoscedasticity among the residuals. The logistic model also has its own set of assumptions. They are as follows: observations must be independent of each other, the response variable is binary or dichotomous, there is little to no multicollinearity between variables, and the relationship between the independent variables and log odds must be linear. The only assumption for SVM models is that all observations must be independent of each other. All of these assumptions talked about in the last few paragraphs must be checked for, otherwise the model would not be a good fit.
Data The Bookbinders data was provided in two Excel files - one for training and one for testing. The training dataset contains 1600 observations and 12 variables. The test set contains 400 observations. Initial exploration showed the data was relatively clean with no missing values. Some adjustments were made to prepare the data for modeling. The variable Choice was converted to a factor since it represents the binary target variable. Outliers were identified in the variable First_purchase and removed to prevent the outlier from having a big influence on our modeling. When looking at the distributions of the variables, they seem to be pretty skewed. Most observations fall on the lower end of the spectrum rather than creating that normal shaped bell curve. There seems to be no highly correlated variables. In summary, data preparation included converting formats, removing outliers, eliminating correlated predictors, and identifying variables to retain for modeling. The resulting dataset allows initial modeling but has limited sample size. Overall, the Bookbinders data provides a sound foundation for analysis but expanded data collection is recommended to improve model accuracy and usefulness.
Results Our findings show that the SVM model showed the highest accuracy at 97.3%. The model created 329 support vectors with 2 classes. The logistic regression model was the next highest at 89% and linear regression was not a good fit for our data. We checked our data for the model’s assumptions. We found that the assumptions were fit for all exception linear regression. Our variables did not fit the normality criteria, and we saw this when our accuracy was 65%. For our models, we found that P_Youth, Last_purchase, and P_Art were some of the more significant variables in that order. These significant variables were consistent with the logistic and SVM models. The least significant variable was First_purchase. Most of the variables tested came back with some significance, however, the ones listed had the most/least impact on the model. These variables could have turned out significant for many reasons. A lot of parents who are buying books for their kids, may also buy this book for their children either due to school purposes, or to encourage their child to read. Additionally, people with a recent purchase could have purchased again due to being a frequent customer, or because they have enjoyed their recent experience are willing to purchase again while the memory is still fresh. Lastly, people who have purchased a lot of art books were more likely to purchase The Art History of Florence. This makes sense, since these people are more likely to be art enthusiasts. Of course, there are some variables that were not tested that could be a factor in whether or not someone purchased the book. Factors such as income or education are likely to have an influence. People with lower income are less likely to make a wanted purchase, and people with a higher education are more likely to purchase a nonfiction book. Because of these results, we believe that the mail campaign should target people who purchase youth books, and who have purchased something relatively recently. Targeting your campaign will save the company money by spending on a lead that is more likely to purchase the book rather than having a broad audience. There are advantages and disadvantages of each model to consider. We will disregard the linear regression model as it is not a good fit for the data. Logistic regression is a more simple model than SVM. Because of this, you will find that logistic regression is easier to understand and interpret, but because of that, it will not be as accurate. On the other hand, SVM is a much more accurate predicting model, but it is harder to understand logistic regression. For future modeling purposes, I would stick to either logistic regression or linear regression (if the data allows) if you want to keep things simple.
Conclusions Our analysis demonstrates that the SVM model achieved the highest accuracy at 97.3%. In contrast, logistic regression trailed behind with 89%, while linear regression was not suitable for our dataset. Upon examining model assumptions, we found that all models, except linear regression, met the expected criteria. Deviations in variable normality affected the accuracy of linear regression, which was only 65%. Analyzing variable significance, we identified P_Youth, Last_purchase, and P_Art as the most influential, consistently observed in both logit and SVM models. Conversely, First_purchase had the least impact. These findings likely result from factors like parental buying patterns, recent customer behavior, and preferences of art enthusiasts. Moreover, while our analysis provided valuable insights, it is essential to acknowledge untested variables such as income or education, which could significantly impact book purchases. Based on these results, we recommend a targeted mail campaign for individuals purchasing youth books and making recent purchases, optimizing resources. Acknowledging model trade-offs, logistic regression offers simplicity, while SVM excels in accuracy. Linear regression, dismissed due to poor fit, is not considered. Future modeling decisions should carefully weigh these considerations, leaning towards logistic or linear regression for simplicity, or embracing the complexity of SVM for superior predictive performance.
library(MASS)
library(ggplot2)
library(tidyverse)
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.3 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.0
## ✔ lubridate 1.9.3 ✔ tibble 3.2.1
## ✔ purrr 1.0.2 ✔ tidyr 1.3.0
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ✖ dplyr::select() masks MASS::select()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(corrplot)
## corrplot 0.92 loaded
library(car)
## Loading required package: carData
##
## Attaching package: 'car'
##
## The following object is masked from 'package:dplyr':
##
## recode
##
## The following object is masked from 'package:purrr':
##
## some
library(readr)
library(e1071)
library(caret)
## Loading required package: lattice
##
## Attaching package: 'caret'
##
## The following object is masked from 'package:purrr':
##
## lift
library(lattice)
library(rpart)
library(rpart.plot)
library(ROCR)
library(readxl)
library(kernlab)
##
## Attaching package: 'kernlab'
##
## The following object is masked from 'package:purrr':
##
## cross
##
## The following object is masked from 'package:ggplot2':
##
## alpha
#Import Datasets
df_train <- read_xlsx("/Users/isabellaclemandot/Downloads/BBBC-Train.xlsx")
df_test <- read_xlsx("/Users/isabellaclemandot/Downloads/BBBC-Test.xlsx")
#Clean data
sum(is.na(df_train))
## [1] 0
sum(is.na(df_test))
## [1] 0
hist(df_train$Frequency)
hist(df_train$First_purchase)
boxplot(df_train$First_purchase)
boxplot(df_train$Last_purchase)
#Clearing Outliers
df_train$First_purchase[df_train$First_purchase >= 82] <- 82
df_train$First_purchase[df_train$First_purchase >= 82] <- 82
#Linear Regression
df_train_linear <- lm(Choice ~ ., data=df_train)
summary(df_train_linear)
##
## Call:
## lm(formula = Choice ~ ., data = df_train)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.67918 -0.21433 0.01905 0.21520 0.60230
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 8.020e-01 2.445e-02 32.801 < 2e-16 ***
## Observation -6.185e-04 1.563e-05 -39.577 < 2e-16 ***
## Gender -6.804e-02 1.430e-02 -4.757 2.14e-06 ***
## Amount_purchased 1.522e-04 7.887e-05 1.930 0.053761 .
## Frequency -4.754e-03 1.555e-03 -3.058 0.002263 **
## Last_purchase 5.310e-02 9.701e-03 5.474 5.10e-08 ***
## First_purchase -7.192e-04 1.299e-03 -0.554 0.579935
## P_Child -7.179e-02 1.172e-02 -6.126 1.13e-09 ***
## P_Youth -4.296e-02 1.433e-02 -2.997 0.002769 **
## P_Cook -8.005e-02 1.189e-02 -6.733 2.32e-11 ***
## P_DIY -7.944e-02 1.411e-02 -5.628 2.15e-08 ***
## P_Art 5.125e-02 1.391e-02 3.684 0.000237 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.2688 on 1588 degrees of freedom
## Multiple R-squared: 0.6175, Adjusted R-squared: 0.6148
## F-statistic: 233 on 11 and 1588 DF, p-value: < 2.2e-16
#normality
#Logistic Regression
# Make Choice a factor
df_train$Choice <- as.factor(df_train$Choice)
levels(df_train$Choice)
## [1] "0" "1"
df_test$Choice <- as.factor(df_test$Choice)
levels(df_test$Choice)
## [1] "0" "1"
# Modeling
book_log = glm(Choice ~ Gender + Amount_purchased + Frequency + Last_purchase + First_purchase + P_Child + P_Youth + P_Cook + P_DIY + P_Art, data = df_train, family = binomial)
summary(book_log)
##
## Call:
## glm(formula = Choice ~ Gender + Amount_purchased + Frequency +
## Last_purchase + First_purchase + P_Child + P_Youth + P_Cook +
## P_DIY + P_Art, family = binomial, data = df_train)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.3520393 0.2143791 -1.642 0.1006
## Gender -0.8631616 0.1374521 -6.280 3.39e-10 ***
## Amount_purchased 0.0018644 0.0007918 2.355 0.0185 *
## Frequency -0.0753812 0.0166062 -4.539 5.64e-06 ***
## Last_purchase 0.6123661 0.0938892 6.522 6.93e-11 ***
## First_purchase -0.0149212 0.0128269 -1.163 0.2447
## P_Child -0.8112627 0.1167077 -6.951 3.62e-12 ***
## P_Youth -0.6369859 0.1433776 -4.443 8.88e-06 ***
## P_Cook -0.9230071 0.1194824 -7.725 1.12e-14 ***
## P_DIY -0.9060559 0.1437110 -6.305 2.89e-10 ***
## P_Art 0.6862316 0.1270098 5.403 6.55e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1799.5 on 1599 degrees of freedom
## Residual deviance: 1392.1 on 1589 degrees of freedom
## AIC: 1414.1
##
## Number of Fisher Scoring iterations: 5
pred.book.test <- predict(book_log, newdata = df_test, type = "response")
pred.book.test.surv = ifelse(pred.book.test >= 0.5, "1", "0")
caret::confusionMatrix(as.factor(pred.book.test.surv), df_test$Choice)
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 1968 125
## 1 128 79
##
## Accuracy : 0.89
## 95% CI : (0.8765, 0.9025)
## No Information Rate : 0.9113
## P-Value [Acc > NIR] : 0.9998
##
## Kappa : 0.324
##
## Mcnemar's Test P-Value : 0.8999
##
## Sensitivity : 0.9389
## Specificity : 0.3873
## Pos Pred Value : 0.9403
## Neg Pred Value : 0.3816
## Prevalence : 0.9113
## Detection Rate : 0.8557
## Detection Prevalence : 0.9100
## Balanced Accuracy : 0.6631
##
## 'Positive' Class : 0
##
pred <- prediction(pred.book.test, df_test$Choice)
perf <- performance (pred, "tpr", "fpr")
unlist(slot(performance(pred, "auc"), "y.values")) #AUC
## [1] 0.8002346
#Support vector Machine
# Generate a formula to create a model that considers all remaining variables as independent
form1 = Choice ~ Gender + Amount_purchased + Frequency + Last_purchase + First_purchase + P_Child + P_Youth + P_Cook + P_DIY + P_Art
# Assign the output from tune.svm to an object called tuned
tuned = tune.svm(form1, data = df_train, gamma = seq(.01, .1, by = .01), cost = seq(.1, 1, by = .1))
# Setting parameter values to know which values for gamma and cost are best
tuned$best.parameters
# To see the errors for a different combinations:
tuned$performances
# SVM model
mysvm = svm(formula = form1, data = df_train, gamma = tuned$best.parameters$gamma, cost = tuned$best.parameters$cost)
summary(mysvm)
##
## Call:
## svm(formula = form1, data = df_train, gamma = tuned$best.parameters$gamma,
## cost = tuned$best.parameters$cost)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 0.7
##
## Number of Support Vectors: 789
##
## ( 390 399 )
##
##
## Number of Classes: 2
##
## Levels:
## 0 1
# Classify test data and get the confusion matrix
svmpredict = predict(mysvm, df_test, type = "response")
table(pred = svmpredict, true = df_test$Choice)
## true
## pred 0 1
## 0 2058 173
## 1 38 31
# Tune SVM
tuned = tune.svm(Last_purchase ~ ., data = df_train, gamma = seq(.01, 0.1, by = .01), cost = seq(0.1, 1, by = 0.1), scale = TRUE)
# Build my SVM
resultsSVM <- svm(formula=Last_purchase ~ ., data = df_train, gamma = tuned$best.parameters$gamma, cost = tuned$best.parameters)
# Make predictions
predSVM <- predict(resultsSVM, df_test, type = "response")
# Compute error (Mean Square Error)
mean((predSVM - df_test$Last_purchase)^2)
## [1] 0.9736808