Text often comes in sentences or paragraphs, such as customer reviews or social media posts. Tidy text organizes text into a table with one token per row. A token is a unit of text, such as a word. This structure makes it easier to filter, count, and analyze words.
Tokenization splits text into smaller units. In this example, I use unnest_tokens() to separate a short poem into individual words.
text <- c(
"Because I could not stop for Death -",
"He kindly stopped for me -",
"The Carriage held but just Ourselves -",
"and Immortality"
)
text_df <- tibble(line = 1:4, text = text)
text_df
## # A tibble: 4 × 2
## line text
## <int> <chr>
## 1 1 Because I could not stop for Death -
## 2 2 He kindly stopped for me -
## 3 3 The Carriage held but just Ourselves -
## 4 4 and Immortality
tidy_text <- text_df %>%
unnest_tokens(word, text)
tidy_text
## # A tibble: 20 × 2
## line word
## <int> <chr>
## 1 1 because
## 2 1 i
## 3 1 could
## 4 1 not
## 5 1 stop
## 6 1 for
## 7 1 death
## 8 2 he
## 9 2 kindly
## 10 2 stopped
## 11 2 for
## 12 2 me
## 13 3 the
## 14 3 carriage
## 15 3 held
## 16 3 but
## 17 3 just
## 18 3 ourselves
## 19 4 and
## 20 4 immortality
The original table contains four rows, with one line of the poem in each row. After tokenization, the table contains 20 rows, with one word per row. By default, unnest_tokens() converts words to lowercase and removes punctuation while keeping the line numbers.
For this analysis, I use the six novels included in the janeaustenr package. First, I keep track of each line and chapter within each book.
original_books <- austen_books() %>%
group_by(book) %>%
mutate(
linenumber = row_number(),
chapter = cumsum(
str_detect(text, regex("^chapter [\\divxlc]", ignore_case = TRUE))
)
) %>%
ungroup()
head(original_books)
## # A tibble: 6 × 4
## text book linenumber chapter
## <chr> <fct> <int> <int>
## 1 "SENSE AND SENSIBILITY" Sense & Sensibility 1 0
## 2 "" Sense & Sensibility 2 0
## 3 "by Jane Austen" Sense & Sensibility 3 0
## 4 "" Sense & Sensibility 4 0
## 5 "(1811)" Sense & Sensibility 5 0
## 6 "" Sense & Sensibility 6 0
I use unnest_tokens() to separate the books into individual words while retaining the book, line number, and chapter.
tidy_books <- original_books %>%
unnest_tokens(word, text)
head(tidy_books)
## # A tibble: 6 × 4
## book linenumber chapter word
## <fct> <int> <int> <chr>
## 1 Sense & Sensibility 1 0 sense
## 2 Sense & Sensibility 1 0 and
## 3 Sense & Sensibility 1 0 sensibility
## 4 Sense & Sensibility 3 0 by
## 5 Sense & Sensibility 3 0 jane
## 6 Sense & Sensibility 3 0 austen
Stop words are common words, such as “the,” “and,” and “of,” that often add little information to a word frequency analysis. I use anti_join() to remove words found in the stop_words dataset.
tidy_books <- tidy_books %>%
anti_join(stop_words, by = "word")
head(tidy_books)
## # A tibble: 6 × 4
## book linenumber chapter word
## <fct> <int> <int> <chr>
## 1 Sense & Sensibility 1 0 sense
## 2 Sense & Sensibility 1 0 sensibility
## 3 Sense & Sensibility 3 0 jane
## 4 Sense & Sensibility 3 0 austen
## 5 Sense & Sensibility 5 0 1811
## 6 Sense & Sensibility 10 1 chapter
After removing stop words, I count how often each remaining word appears across the six novels.
word_counts <- tidy_books %>%
count(word, sort = TRUE)
head(word_counts, 10)
## # A tibble: 10 × 2
## word n
## <chr> <int>
## 1 miss 1855
## 2 time 1337
## 3 fanny 862
## 4 dear 822
## 5 lady 817
## 6 sir 806
## 7 day 797
## 8 emma 787
## 9 sister 727
## 10 house 699
The graph displays words that appear more than 600 times across the novels after stop words are removed.
word_counts %>%
filter(n > 600) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(x = n, y = word)) +
geom_col(fill = "steelblue") +
labs(
title = "Most Common Words in Jane Austen's Novels",
x = "Number of Occurrences",
y = NULL
) +
theme_minimal()
After removing stop words, “miss” is the most frequent word, appearing 1,855 times, followed by “time” with 1,337 occurrences. Names such as “fanny” and “emma” also appear frequently. Words such as “lady,” “sir,” and “sister” suggest that social titles and family relationships are common in these novels. However, word counts alone do not explain how each word is used, so reading the surrounding text would help confirm these patterns.
Marketers can use this same process to analyze customer reviews, survey responses, and social media posts. Frequently mentioned words such as “price,” “delivery,” or “quality” can identify topics that deserve further investigation. These findings can help businesses improve products, address service problems, and develop relevant advertising. However, frequency alone does not show whether customers feel positively or negatively about a topic.
This exercise follows Sections 1.1–1.3 of Julia Silge and David Robinson’s Text Mining with R: A Tidy Approach. The poem example is by Emily Dickinson, and the novel texts come from the janeaustenr package.