This tool offers a streamlined approach for analyzing sentiment of either social media content, or free-form excel/csv documents. By leveraging cutting-edge Natural Language Processing (NLP) technology, it enables users to effortlessly gauge audience sentiment and highlight notable reactions. All information is automatically compiled and displayed in an accessible format, with the option to easily export data for external use.
To evaluate the validity of our tool, we examined 50,000 Amazon reviews. Specifically, for each review we calculated the customer’s overall sentiment using their text-based feedback, and examined if the calculations from our toolkit correlate with their overall product rating (out of five-stars). Stated differently, we examined if automated sentiment analyzer, when applied to the way customers talk about a product, can be used to predict their overall product ratings. Our analysis revealed that the sentiment scores calculated by applying our automated tool on the textual comments significantly predicted buyers’ numerical rating out of 5-stars, r = .58, t(49918) = 158.67, p < .001. These findings compellingly indicate that our sentiment analysis toolkit is capable of quantifying and reflecting consumers’ sentiments, thus offering a reliable and valid measure of their underlying attitudes.
We next examined the correspondence between the automated sentiment scores derived from our toolkit, and those established by human elevators. To do this, we examined the sentiment of reviews across three established datasets commonly used to validate Natural Language Processing (NLP) tools (Kotzias et al., 2015). Each dataset includes reviews from either Amazon (1067 reviews), IMDb (1041 reviews), and Yelp (1040 reviews), paired with sentiment assessed by human judges. For the purposes of this analysis we categorized our sentiment scores into “positive” and “negative”, based on if sentiment calculations were greater than or less than neutral. Agreement rates with human judgments for each dataset were as follows:
As a follow-up analysis, we employed logistic regression models and calculated odds ratios, to quantify the likelihood of comments being classified as “positive” by human raters if they were classified as “positive” by our tool. The findings indicated that comments deemed unequivocally positive by our tool were significantly more likely to align with human “positive” ratings. Specifically, comments identified as unequivocally positive were:
These findings affirm that our automated sentiment analysis toolkit yields sentiment scores that align closely with the assessments made by human elevators.