Modelling

LTM$lag <- LTM$expectedtransit-LTM$transit

LTM$Timetoreview <- LTM$review_answer_timestamp-LTM$order_delivered_customer_date

Modelling

We will now endeavour to predict reviews of customers. Reviews are a crucial kpi to track because they massively influence future customer purchases. A good review will lead to repeat business and greater traction while a negative review may kill your business and drive potential customers to the competition. While it seems that only sellers would suffer from negative reviews, Olist evidently has a big interest in nurturing an ecosystem of quality sellers that make their customers satisfied. Thus whether we adopt the point of view of Olist or its third-party sellers, predicting reviews will be of great use.

To be usefull, this model should predict reviews as soon as possible before the customer puts its review online. Thus we limit our independant variables and eliminate all information that is gathered after the carrier takes charge of the package.

To build this model, we chose a tree based gradient boosting model using the library xgboost. We chose this model because XGBoost almost always outperforms other models.

To improve performance we chose to transform our problem into a classification problem telling whether the review will be bad (1 to 3 stars) or good (4 to 5 stars).

LTM <- LTM[complete.cases(LTM),]
LTM$Support <- ifelse(LTM$review_score<4,1,0)
#Variables available before reception of the package.
LTM_Model2 <- LTM[,c(7,8,9,11,13,14,15,16,17,18,25,26,27,28,29,30,32,33,
                     34,35,36,37,38,39,40,43,45,49)]
LTM_Model2$Month <- as.factor(LTM_Model2$Month)

LTM_Model2 <- one_hot(as.data.table(LTM_Model2))

set.seed(234)

training.samples <- LTM_Model2$Support %>% 
  createDataPartition(p = 0.8, list = FALSE)


train.label <- LTM_Model2[training.samples,c(161)]
test.label <- LTM_Model2[-training.samples,c(161)]
train.data  <- LTM_Model2[training.samples,c(-161)]
test.data <- LTM_Model2[-training.samples,c(-161)]

dtrain <- xgb.DMatrix(label = as.matrix(train.label), data = as.matrix(train.data))
dtest <- xgb.DMatrix(label = as.matrix(test.label), data = as.matrix(test.data))

watchlist = list(train=dtrain, test=dtest)
pred <- predict(model2,dtest)
pred <- ifelse(pred>0.5,1,0)

sum(ifelse(pred>0.5,1,0) == test.label)/nrow(test.label)
## [1] 0.7974076
confusionMatrix(as.factor(test.label$Support),as.factor(pred))
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction     0     1
##          0 13026   335
##          1  3213   939
##                                           
##                Accuracy : 0.7974          
##                  95% CI : (0.7914, 0.8033)
##     No Information Rate : 0.9273          
##     P-Value [Acc > NIR] : 1               
##                                           
##                   Kappa : 0.2642          
##                                           
##  Mcnemar's Test P-Value : <2e-16          
##                                           
##             Sensitivity : 0.8021          
##             Specificity : 0.7370          
##          Pos Pred Value : 0.9749          
##          Neg Pred Value : 0.2262          
##              Prevalence : 0.9273          
##          Detection Rate : 0.7438          
##    Detection Prevalence : 0.7629          
##       Balanced Accuracy : 0.7696          
##                                           
##        'Positive' Class : 0               
## 
importance_matrix <- xgb.importance(model = model2)
xgb.plot.importance(importance_matrix = importance_matrix,top_n = 10)

As a result, the alogrithm has a 73% specificity, which means that the algorithm may predict in 73% of the cases when a customer is going to put a grade below 3 stars. Why this algorithm is so usefull however is because the prediction is made on avearage, almost 12 days before the customer actually receives the parcel! As noted in the exploratory phase, the main feature employed in the model is the number of days that the seller takes to dispatch the product. Some other features such a customer longitude or latitute as well as prefix are hard to explain in words or visualize. While it might be overfitting, there might also be the plausible possibility that the algorithm has detected some real underlying phenomena.

Because there is so much time available, Olist is fully to act proactively and take the necessary measures to placate a future angry customer. We thought of two manner Olist could act on this model to improve its business.

  1. When Olist flags an order that will probably result in an angry review, Olist can contact the seller to inquire whether he has any problem or difficulty, and let know the seller that he will probably have to provide extra customer support.

  2. On its end, Olist can decide to ship “customer care packages” that for example contain a nice message, some goodies or discount coupons. The customer care package could be personalized and sent automatically, so the customer could receive it even before he receives the package that is likely to make him angry. At least, if the customer is upset with the seller he will keep good relationship with Olist.

There are surely many other ways Olist could leverage on this insight. Certainly, many customer will be flagged as “upset” customer while they wont, thus the responsibility lies with Olist managers to calculate how to act upon those insight in terms of cost effectiveness, comparing the cost of supporting an upset customer with the cost of unusefully taking care of a happy customer.

Many things can be done to improve this model further, such as tuning the model more efficiently but also gathering other variables such as age, gender, carrier id, shipment method, etc.