Lexicons

The key to sentiment analysis are what are called lexicons. There are multiple different types of lexicons, but all of them have the goal of helping to identify the sentiment of words. Remember that in text analysis, words are our data!

The first lexicon we are going to explore focuses on just two types of sentiment: positive or negative. The lexicon is a long list of English words, and each word in the lexicon is assigned a numeric value identifying how positive or negative that word is.

Let’s see if we could build such a lexicon. Here are five different words: fine, amazing, great, good, spectacular. All of these words express a positive sentiment, but do they all express positivity equally? In other words, is “fine” stronger or less strong than “amazing”?

Hmm…this might seem a little subjective. Luckily, experts have been working on building lexicons for a long time. These individual are experts on words and sentiment. We will be using their work to help us identify the sentiment of our airplane data.

The lexicon we will start with is the AFINN lexicon. To load this lexicon into R, use the following:

# Load the libraries 
library(tidytext)
library(textdata)

# Load the lexicon
get_sentiments("afinn")

The AFINN lexicon assigns words a score between -5 and 5. Negative scores indicate a negative sentiment and positive scores indicate a positive sentiment. The closer to 0 that a word is scored, the more neutral the word.

Okay, so thus far we know that the AFINN lexicon is a list of words in English that have been assigned a score between -5 and 5. Will we find every word in the English language in the lexicon?? Nope. A lot of words in English are neutral (grass, tree, sidewalk, etc.). The words in the lexicon are those that express a sentiment of some kind.

What can we do with this lexicon now that we have it? One option is to look at the words in each Tweet. For all the words in the lexicon we can assign them their scores. We can then add up all the scores to decide if a Tweet is positive or negative overall!

Let’s try that. First of all, there are a lot of columns in the data set. We only need to keep two of them for now, the ID of the tweet and the text

TweetsSmall <- Tweets[,c(1,11)]

We are going to do one more thing what will make our lives easier.

TweetsSmall$tweet_id <- 1:nrow(TweetsSmall)

At this point, we are ready to tokenize the data, which we did last time.

# Tokenize each tweet
tidy_Tweets <- TweetsSmall %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

We currently have broken the tweets down into words. Now, we want to look at these words and figure out which of them are also in the lexicon. For those words that are in the lexicon, we want to assign the scores from the lexicon to the words. We can do that using one short code.

textdata_sentiment <- tidy_Tweets %>%
  inner_join(get_sentiments("afinn"))

We don’t really want to do add up the scores by ourselves for each Tweet. Luckily, we don’t have to. To create a column of scores for each tweet, use the following:

textdata_sentiment <- tidy_Tweets %>%
  # Add in the lexicon scores
  inner_join(get_sentiments("afinn")) %>%
  # Group together the words in each Tweet
  group_by(tweet_id) %>%
  # Create a new variable that is a sum of 
  # the sentiment scores for each word
  mutate(sentimentScore= sum(value)) %>%
  # The last step
  filter(row_number(tweet_id) == 1)

As a last step, let’s get rid of the columns we don’t need.

textdata_sentiment <- textdata_sentiment[,c(1,4)]

Now we have a score for each tweet! Let’s plot those.

library(ggplot2)

textdata_sentiment %>%
  ggplot(aes(sentimentScore)) +
  geom_bar(show.legend = FALSE, fill = "purple") +
   labs(x = "Sentiment Score",
       y = "Number of Tweets", title = "Figure 1")

At this point, we can identify whether a tweet is positive or negative, so let’s add that in.

textdata_sentiment <- textdata_sentiment %>%
  mutate(category = ifelse(sentimentScore > 0, "positive",    ifelse(sentimentScore ==0, "neutral", "negative")))

Now that we have the Tweets assigned to categories, let’s see what words are showing up in these Tweets.

textdata_sentiment <- tidy_Tweets %>%
  # Add in the lexicon scores
  inner_join(get_sentiments("afinn")) %>%
  # Group together the words in each Tweet
  group_by(tweet_id) %>%
  # Create a new variable that is a sum of 
  # the sentiment scores for each word
  mutate(sentimentScore= sum(value)) %>%
  # Adding on the sentiments
  mutate(category = ifelse(sentimentScore > 0, "positive", ifelse(sentimentScore ==0, "neutral", "negative")))

afinn_word_counts <- textdata_sentiment %>%
  count(word, category, sort = TRUE)

afinn_word_counts %>%
  group_by(category) %>%
  slice_max(n, n = 10) %>% 
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word, fill = category)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~category, scales = "free_y") +
  labs(x = "Contribution to sentiment",
       y = NULL)

A Second Lexicon

The first lexicon we looked at, the AFINN lexicon, assigned scores to positive and negative sentiments. However, what if we want to have more than just “negative” or “positive” as our sentiments?

There is a different kind of lexicon that assigns words to sentiments. For instance, “happy” might be assigned the sentiment “joy”. One such lexicon is called the NRC lexicon.

# Load the lexicon
get_sentiments("nrc")

Here, we see that the same word can have multiple sentiments!

When people write airline reviews, one sentiment they often feel is anger or frustration. Let’s see what words in the NRC lexicon are classified as “anger” terms. To create a list of “anger” terms, use the code below.

nrc_anger <- get_sentiments("nrc") %>% 
  filter(sentiment == "anger")

The next step is to see how many of these “anger” words show up in our flight reviews.

angertext <- tidy_Tweets %>%
  inner_join(nrc_anger) %>%
  count(word, sort = TRUE)

Now this is cool, but what we probably want to do is determine the sentiment of all the words contained in each Tweet. So let’s do that!

nrc_word_counts <- tidy_Tweets %>%
  inner_join(get_sentiments("nrc")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

Let’s try creating a plot of the top 10 words in each category.

nrc_word_counts %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment",
       y = NULL)