Predicting Closure of Businesses Based on Rating Behaviour

Jesus Ramos
November 22 2015

Motivation & Business Question

  1. Knowing how your business is rated in Yelp as an aggregate is interesting, but unactionable.
  2. Going deeper into how ratings and their stars behave through time can grant insights as to the moments when customer care and attention is more important.
  3. Failing to preemptively pamper your customers at the right moment in time may create an effect that will ultimately reduce the profitability of the business.
  4. Since service is discrete, but perception of service is continuous, bad service will have a lasting effect.
  5. If accumulated, this effect will be enough to force the shutdown of the business.

Question

Can user ratings be used to predict business closure?

Methodology

  1. Load data from Yelp and test if the mean stars for closed business is significantly different from the mean stars for those still alive.
  2. Convert ratings from repeated observations in time, to a yearly accumulated per business.
  3. Calculate year-to-year changes in rating counts and number of stars.
  4. Predict the closure of a business with a statistical learning algorithm.

Sample of final data

  review_count_change_year review_stars_change_year
1                       NA                       NA
2                        0                       -3
3                        2                        2

Note we'll only consider changes in review counts and stars.

Random Forest Performance

The model does a decent job predicting the businesses that will remain open, but a poor job predicting those that have closed, as shown by the confusion matrix for the test set:

          Reference
Prediction FALSE  TRUE
     FALSE     1     0
     TRUE   2226 16008

Also, the ROC curve shows how poorly specific and sensitive the model is, with an accuracy of 87% and a specificity of 0%. plot of chunk unnamed-chunk-3


Call:
plot.roc.default(x = rfModel$pred$obs[selectedIndices], predictor = rfModel$pred$mtry[selectedIndices])

Data: rfModel$pred$mtry[selectedIndices] in 5198 controls (rfModel$pred$obs[selectedIndices] FALSE) < 37352 cases (rfModel$pred$obs[selectedIndices] TRUE).
Area under the curve: 0.5

Results and Conclusion

  1. The developed model is subpar. This may be due to our heavy treatment to transform review data into discrete yearly observations.
  2. The former implies that for businesses that opened in 2008 but received their first review in 2013, the model will have 0s for the first 5 years of operations. These 0s affect the model's performance.
  3. It could also mean that ratings are not a good predictor for a going concern, and that more direct activity indicators, like check-ins, are better suited.
  4. Lessons concerning the overuse (and overkill) of machine learning algorithms are clear with these examples.
  5. Lessons concerning the long, heavy process of cleaning and processing data are also confirmed, since the bulk of the code for this study is to achieve a data format processable by the chosen algorithm.