The Goal

For our activity today, we are going to work with a brand new kind of data - text data! Text data means data that comes in the form of words or sentences. This could be a review posted online, a book, a journal article, or anything else that involves words. Our goal for today will be to start exploring how we work with this sort of data. We will continue this work next week!

The Data

One of the forms of text data that is analyzed frequently is online reviews. It is common for customers to be asked to provided open-ended feedback on a product or service, meaning that this data is provided to companies in the form of text data. For today, we are looking at reviews provided about airlines.

The data we will be working with today has 14640 rows. Each row in the data set represents a tweet that someone posted about an airline. To download the data, follow this link or copy and paste the following into a browser window: https://drive.google.com/file/d/1O1yBrvxwuGIYSdplRhYeDi-dB4jQugI-/view?usp=share_link

This data set has 15 columns. For the moment, we are interested in the column called text, which contains the actual text of the Tweet. Warning: These are real Tweets, which means they may contain objectionable content and language.

Exploring the data: Tokenizing

Now that we have the data set loaded, let’s start actually looking at some text data.

Okay, so this is a short sentence. Now what? Well, the power of text data lies in looking at what are called tokens. Generally, a token is a word, though there are some cases where we use short phrases as tokens. We know that Tweets are composed of written text, and that means that to understand the content of the text we need to look at the individual words that make up the Tweet.

There are two packages we need in order to tokenize our text data:

# Load dplyr
library(dplyr)

# Load tidytext 
library(tidytext)

The first package is familiar to us, but the second (tidytext) is specifically designed to help us work with text data. As the name of the package suggests, this is a tidy package, which means it works with the tidyverse in R. Because of this, before we can work with the text, we need to put it in the form of a tibble (a tidy table). This is the format that the package is expecting. To make the conversion, use the following:

# Pull the text of the 15th Tweet 
textdata <- Tweets$text[15]

# Convert to the form we need for tidyText
textdata  <- tibble(textdata )

Once the data is in the correct form, we are ready to tokenize the text of the 15th Tweet!

# Break the text into tokens 
textdata %>%
  unnest_tokens(word, textdata )

What appears on your screen is 2 words. These are the two unique words in the text. What happens if the text has more than two words?

Aside from converting the text into tokens, two other changes are made to the text data when we use the unnest_tokens() command.

Okay, so we can break our text into words. Great!! However, are all of these words really useful?

Stop Words

Words that convey no useful information are called stop words. These are different from language to language, but in English they include things like “a”, “the”, “it”, and “is”. There is a whole list of these contained in R. To load them (you need to do this!), use the function below:

# Load the list of English stop words
data(stop_words)

These words are necessary for writing, but they don’t tell us much about how the writer felt about the airline or the flight. So, let’s remove them!

Luckily, removing stop words can be done using one function:

# Pull the text of the 13th Tweet 
textdata  <- Tweets$text[13]

# Convert to tidy form 
textdata <-tibble(textdata)

textdata  %>%
  unnest_tokens(word, textdata ) %>%
  anti_join(stop_words)

Let’s practice!

Okay, so we can tokenize and remove stop words in individual comments. Now what?

Looking at all the words

Instead of looking Tweet by Tweet, let’s take a look at what types of words we are seeing across the whole data set. To do this, we need to tokenize the entire column text in the data set.

# Pull the whole column 
textdata <- Tweets$text

# Convert to tibble
textdata <-tibble(textdata)

# Store all the words
words_only <- textdata %>%
  unnest_tokens(word, textdata) %>%
  anti_join(stop_words)

Running this code creates the object words_only. Let’s take a look at this object.

That’s a lot of words!!! Are they all unique? Or do certain words come up more than once. Let’s see.

unique_words <- words_only %>%
  count(word, sort = TRUE) 

Running the code above creates a data frame with two columns.

library(ggplot2)

# Start with the unique words data frame
unique_words %>%
  # Choose only words that occur at least 600 times 
  filter(n > 600) %>%
  # Order the words from most frequent to least frequent
  mutate(word = reorder(word, n)) %>%
  # Create a plot
  ggplot(aes(n, word)) +
  geom_col() +
  labs(y = NULL)

# Store all the words
words_only <- textdata  %>%
  unnest_tokens(word, textdata ) %>%
  anti_join(stop_words)

# Store all the words
words_only1 <- textdata  %>%
  unnest_tokens(word, textdata ) %>%
  anti_join(stop_words) %>%
  filter( word != "united") 

# Create the list of words to exclude
to_exclude <- c("united", "flight")

# Store all the words
words_only1 <- textdata  %>%
  unnest_tokens(word, textdata ) %>%
  anti_join(stop_words) %>%
  filter( !(word %in% to_exclude))