Unigram-based Sentiment Analysis of Textual Restaurant Reviews for Prediction of Above Average Rating

Raeger Tay
20th November 2015

Question of Interest

In the paper, we looked at the usage of unigram-based sentiment analysis on textual restaurant reviews found in the Yelp Academic Dataset. The analysis was performed using an opinion lexicon and sentiment scores were recorded for each review. Using the scores as predictors, we predicted whether the review has given a restaurant an above average star rating or otherwise. Above average rating is defined as four star or more. We then compared the overall accuracy from three models and the best classifer was selected via 10-fold validation to predict the test set. We aim to answer the following questions.

  • How effective is the use of sentiment analysis on textual reviews to generate predictors?
  • How effective are these predictors in predicting reviews?
  • What are the problems relating to the analysis and prediction?

Methodolgy

Sentiment Analysis: Firstly, the textual review is cleaned of punctuations, numbers and stopwords. Then individual words are extracted which are then compared with an opinion lexicon consisting of about 6,800 unigrams. The lexicon consists of two lists, one positive words and one negative words. Scores are then generated based on the number of positive and negative words matched.

10-fold Cross Validation: The predictors used are the positive score, negative score and the positive-to-negative score ratio. We used 3 models, linear discriminant analysis, quadratic discriminant analysis and logistic regression. The overall accuracy obtained are 0.748, 0.737 and 0.754 respectively.

Results

Logistic Regression is then used to train our model and the overall test accuracy obtained was 75%. The result provides some proof of the effectiveness of our simple sentiment analysis model, however, there is much room for improvement. The plot to the right shows us the problem: there is too much overlapping sentiment across the 2 groups and the range of sentiment scores is too wide in both groups.

plot

Discussion and Conclusion

A few types of sentiment ambiguity were found in the textual reviews that cause difficulty in accurately scoring the sentiment:

  • Softening of negative sentiment with many positive expressions.
  • Positive or negative terms identified were meant for competitors.
  • Connecting words have different meaning compare to the single word.
  • Use of negation.

In conclusion, our sentiment analysis of textual reviews has shown some promise. However, to improve the analysis as well as prediction accuracy, we need to solve the issue of sentiment ambuiguities, as well as coming up with better opinion lexicon and prediction model.

Full Paper: http://rpubs.com/raeger/capstone_report_fullcode