AI Writing Application Sentiment Analysis

Author

Connor Soethout

Introduction

There are multiple different AI tools that are designed to help in the writing process. Two that are rather large in this sector of AI are “Aiseo” and “Text Cortext AI”. Lets compare what the sentiment is across these two platforms by using reviews on https://www.trustpilot.com/. This will help derive some evidence to which writing AI tool is preferred or recommended.

To control for differing quantities of reviews, the 450 most recent reviews have been scraped to leverage our analysis. A breakdown of the two sets of reviews is listed below for reference.

Ranking	Application	Total Reviews	Average Rating	Standard Deviation
1	Text Cortex AI	450	4.757778	0.6164013
2	AISEO	450	4.657778	0.9019871

Analysis

This analysis will dive into the words used when providing a review comment. This will help us better understand the perception of each application further than just a review rating. While each of us are entitled to shape our own opinion. The hope is to decide which of these writing applications is more recommended.

Positive vs. Negative

It is important to understand what scorable words, meaning words that are either negative or positive, are within these reviews. To do so we will extract the words used at least 20 times in the totality of reviews for each application. The values in the graph that appear negative are just demonstrating that it is a negative. For example, if “bad” appeared 20 times, its value would be -20.

These results are interesting. The first realization that comes to mind is that there are multiple words that show up 22 or 23 times Aiseo. This could be due to the presence of fake reviews as they are describing the application in the same ways. This is ironic because we are using AI writing tools to do our analysis. The word “helpful” way seen 134 times over these 450 reviews, which is a great deal larger than the second highest use of a word and a very solid word a AI writing application would want to be described as.

There are very few words that appear more than 20 times for Text Cortext AI. This is probably due to a larger variance in review comments making it more likely to be real reviews.

The word “easy” is used extensively more both application. That is a common word I use when describing Artificial Intelligence tools, so this is a good sign for both applications.

There do not seem to be any words out of the ordinary for a AI writing tool application. These are the words that will have the most impact on sentiment as they are seen most oftne. Therefore, I feel comfortable moving forward without the removale any words.

Positivity by Month

AI Writing Applications are likely treated with more detail during specific times of the year. In the perspective of a student, an application like these can become very valuable during the end of each semester or quarter when there is a higher influx of essays. For business, on the other hand, there is likely a higher need during times with a large volume of press releases. This could happen at the end of each quarter or if there is something large going on in a political sphere.

Due to the reviews available. Only reviews between June and November will be on the following graph. None of the 450 last reviews occurred prior to June or in the month of December for Aesio.

It does seem that there is an increase in review positivity in the months of September and. November for Aiseo and Text Cortext AI joins in on the positivity increase in the month of November. This seems to fit our earlier predictions as midterms would occur around September and the semester begins to wrap up in November. Furthermore, many companies have their quarterly reports in the months of September and late November.

It is important to keep in mind that that the increasing positivity score above could also be the result of the increase of application use stated previously. After all, artificial intelligence is a rather new. This could also be due to the applications having flaws early on in their existence.

Sentiment Scores

Lastly, lets take a look at the sentiments used in these reviews. This will categorize scorable words into emotions. This will be done on the basis of the NRC Word-Emotion Association Lexicon and more information can be found here.

We will compare the number of words that fit each emotion between the two applications. This will help us understand, in further depth, how reviewers feel about these applications.

Right away, the biggest take away from the graphic above is that there are more words associated with nearly all emotions coming from the Aiseo reviews compared to the Text Cortext AI reviews. This gives further evidence to Aiseo reviews being more word or scorable as the number of reviews was controlled for.

There is a great deal more positive words for Aiseo than Text Cortext AI, however, there is nearly the same difference in negative words. This allows us to consider that they could be comparably similar at equal word levels.

It is a good sign that the highest emotion, outside of positivity is with trust. One of the uncertainties that many have with artificial intelligence is that they cannot trust the work produced or the modeling behind it. This is probably the most meaningful emotion when evaluating AI writing applications. Aiseo wins this category by a sizable mark, but Text Cortext AI has a high score as well.

One of the more interesting findings comes when examining the word counts that align with fear. This is the only category that Text Cortext AI surpasses Aiseo in word count. This is not a emotion I would want to be associated with an AI tool. This emotion total for Text Cortext AI would most likely be higher when the number of reviews is weighted.

Conclusion

Both of these application are rated rather highly and the review content seems to be overwhelming positive. The reviews for Aiseo happen more often and relatively more wordy than reviews for Text Cortext AI. There was also no control for fake reviews. This is important limitations and biases that had the ability to eftected our analysis.

With all of that said, it seems as if Aiseo is the more recommend AI writing tool. The reviews for Aiseo were relatively more positive and trusting. It overwhelmed Text Cortext AI in nearly every aspect and, without ever using either application personally, I would recommend Aiseo between these two writing tools.

--- title: "AI Writing Application Sentiment Analysis" author: "Connor Soethout" toc: true # Generates an automatic table of contents. format: html: code-tools: TRUE # Allow the code tools option showing in the output. self-contained: TRUE # Embeds all components into a single HTML file. execute: warning: FALSE # Prevent all warnings from displaying in the output. message: FALSE # Prevent all messages from displaying in the output. echo: FALSE # Show all code in the output. --- ```{r} #| label: setup #| include: FALSE library(tidyverse) # For all things tidy. library(knitr) # Pretty HTML tables library(readr) # Too help import data ( not the data dictionary) library(stringr) # Help deal with any string data type library(ggplot2) library(tidytext) ai_reviews <- read_csv("https://myxavier-my.sharepoint.com/:x:/g/personal/soethoutc_xavier_edu/EXDSRd68Q1JOuVpbVQ3YRV8BqICH_jbqtj1iIc1gUO4hKQ?download=1") ai_reviews <- ai_reviews %>% mutate(review_date = mdy_hm(review_date)) ai_writing_reviews <- ai_reviews %>% filter(application %in% c("AISEO", "Text Cortex AI")) ai_writing_reviews <- ai_writing_reviews %>% group_by(application) %>% arrange(desc(review_date)) %>% slice_head(n = 450) %>% mutate(review_id = row_number()) %>% ungroup() ``` ## Introduction There are multiple different AI tools that are designed to help in the writing process. Two that are rather large in this sector of AI are "Aiseo" and "Text Cortext AI". Lets compare what the sentiment is across these two platforms by using reviews on https://www.trustpilot.com/. This will help derive some evidence to which writing AI tool is preferred or recommended. To control for differing quantities of reviews, the 450 most recent reviews have been scraped to leverage our analysis. A breakdown of the two sets of reviews is listed below for reference. ```{r} rating_result_table <- ai_writing_reviews %>% group_by(application) %>% summarize(`Total Reviews` = n(), `Average Rating` = mean(rating), `Standard Deviation` = sd(rating)) %>% arrange(desc(`Average Rating`)) %>% mutate(Ranking = rank(desc(`Average Rating`))) %>% mutate(Application = application) %>% mutate(Application = ifelse(Application == "PFPMaker - AI Profile picture Maker", "PFP Maker", ifelse(grepl("Kickresume", Application), "Kickresume", Application))) %>% select(Ranking, Application, `Total Reviews`, `Average Rating`, `Standard Deviation`) kable(rating_result_table) ``` ## Analysis This analysis will dive into the words used when providing a review comment. This will help us better understand the perception of each application further than just a review rating. While each of us are entitled to shape our own opinion. The hope is to decide which of these writing applications is more recommended. ```{r} #| include: FALSE tidy_ai_writing <- ai_writing_reviews %>% unnest_tokens(word, full_comment) %>% anti_join(stop_words) tidy_ai_writing %>% group_by(application, word) %>% summarize(n = n()) %>% arrange(-n) bing <- get_sentiments("bing") app_counts <- tidy_ai_writing %>% group_by(application, word) %>% summarize(n = n()) %>% inner_join(bing) # checking for positive and negative words app_counts %>% group_by(application,sentiment) %>% summarize(n = n()) %>% arrange(-n) ``` ### Positive vs. Negative It is important to understand what scorable words, meaning words that are either negative or positive, are within these reviews. To do so we will extract the words used at least 20 times in the totality of reviews for each application. The values in the graph that appear negative are just demonstrating that it is a negative. For example, if "bad" appeared 20 times, its value would be -20. ```{r fig.height=10} app_counts %>% group_by(application) %>% filter(n > 20) %>% mutate(n = ifelse(sentiment == "negative", -n, n)) %>% mutate(word = reorder(word, n)) %>% ggplot(aes(word, n)) + geom_col() + coord_flip() + facet_wrap(~application, ncol = 2) + geom_text(aes(label = signif(n, digits = 3)), nudge_y = 8) + labs(title = "Positive and Negative Words for Aiseo and Text Cortext AI", subtitle = "Only words appearing at least 20 times are shown") ``` These results are interesting. The first realization that comes to mind is that there are multiple words that show up 22 or 23 times Aiseo. This could be due to the presence of fake reviews as they are describing the application in the same ways. This is ironic because we are using AI writing tools to do our analysis. The word "helpful" way seen 134 times over these 450 reviews, which is a great deal larger than the second highest use of a word and a very solid word a AI writing application would want to be described as. There are very few words that appear more than 20 times for Text Cortext AI. This is probably due to a larger variance in review comments making it more likely to be real reviews. The word "easy" is used extensively more both application. That is a common word I use when describing Artificial Intelligence tools, so this is a good sign for both applications. There do not seem to be any words out of the ordinary for a AI writing tool application. These are the words that will have the most impact on sentiment as they are seen most oftne. Therefore, I feel comfortable moving forward without the removale any words. ### Positivity by Month AI Writing Applications are likely treated with more detail during specific times of the year. In the perspective of a student, an application like these can become very valuable during the end of each semester or quarter when there is a higher influx of essays. For business, on the other hand, there is likely a higher need during times with a large volume of press releases. This could happen at the end of each quarter or if there is something large going on in a political sphere. Due to the reviews available. Only reviews between June and November will be on the following graph. None of the 450 last reviews occurred prior to June or in the month of December for Aesio. ```{r} tidy_ai_writing %>% inner_join(bing) %>% mutate(month = month(review_date)) %>% filter(month < 12 & month > 5) %>% group_by(application, month, sentiment) %>% summarize(n = n()) %>% spread(sentiment, n, fill = 0) %>% mutate(sentiment = positive - negative) %>% ggplot(aes(x = as.factor(month), y = sentiment, fill = application)) + geom_col(position = "dodge") + scale_fill_manual(values = c("#E9D5FF","#0E1227")) + labs(title = "Application Positivity Scores by Month", subtitle = "Positivity score is the total number of positive words minus total negative words", y = "Total Positivity Score", x = "Month of the Year", fill = "Application") ``` It does seem that there is an increase in review positivity in the months of September and. November for Aiseo and Text Cortext AI joins in on the positivity increase in the month of November. This seems to fit our earlier predictions as midterms would occur around September and the semester begins to wrap up in November. Furthermore, many companies have their quarterly reports in the months of September and late November. It is important to keep in mind that that the increasing positivity score above could also be the result of the increase of application use stated previously. After all, artificial intelligence is a rather new. This could also be due to the applications having flaws early on in their existence. ### Sentiment Scores Lastly, lets take a look at the sentiments used in these reviews. This will categorize scorable words into emotions. This will be done on the basis of the NRC Word-Emotion Association Lexicon and more information can be found [here](https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm). We will compare the number of words that fit each emotion between the two applications. This will help us understand, in further depth, how reviewers feel about these applications. ```{r fig.width=8} nrc <- get_sentiments("nrc") # Visualize emotional sentiment counts for each brewery tidy_ai_writing %>% inner_join(nrc, by = "word", relationship = "many-to-many") %>% group_by(sentiment, application) %>% summarize(n = n()) %>% ggplot(aes(x = sentiment, y = n, fill = application)) + geom_bar(stat = "identity", position = "dodge") + scale_fill_manual(values = c("#E9D5FF","#0E1227")) + labs(title = "AI Writing Tool Sentiment Scores", subtitle = "Total number of emotive words scored ", y = "Total Number of Words", x = "Emotional Sentiment", fill = "Application") ``` Right away, the biggest take away from the graphic above is that there are more words associated with nearly all emotions coming from the Aiseo reviews compared to the Text Cortext AI reviews. This gives further evidence to Aiseo reviews being more word or scorable as the number of reviews was controlled for. There is a great deal more positive words for Aiseo than Text Cortext AI, however, there is nearly the same difference in negative words. This allows us to consider that they could be comparably similar at equal word levels. It is a good sign that the highest emotion, outside of positivity is with trust. One of the uncertainties that many have with artificial intelligence is that they cannot trust the work produced or the modeling behind it. This is probably the most meaningful emotion when evaluating AI writing applications. Aiseo wins this category by a sizable mark, but Text Cortext AI has a high score as well. One of the more interesting findings comes when examining the word counts that align with fear. This is the only category that Text Cortext AI surpasses Aiseo in word count. This is not a emotion I would want to be associated with an AI tool. This emotion total for Text Cortext AI would most likely be higher when the number of reviews is weighted. ## Conclusion Both of these application are rated rather highly and the review content seems to be overwhelming positive. The reviews for Aiseo happen more often and relatively more wordy than reviews for Text Cortext AI. There was also no control for fake reviews. This is important limitations and biases that had the ability to eftected our analysis. With all of that said, it seems as if Aiseo is the more recommend AI writing tool. The reviews for Aiseo were relatively more positive and trusting. It overwhelmed Text Cortext AI in nearly every aspect and, without ever using either application personally, I would recommend Aiseo between these two writing tools.