Lab 12: Text Analysis
Tidy Text, Tokenization, and Word Frequency
David G
November 12, 2026
Introduction
This lab follows Sections 1.1–1.3 of Text Mining with R: A Tidy Approach.
I explore how to organize unstructured text, separate sentences into
individual words with unnest_tokens(), remove stop words,
and visualize word frequencies.
I also explain how these techniques can support marketing decisions.
1.1 — The tidy text format
Unstructured text consists of original sentences or paragraphs, such as a customer review.
Tidy text organizes the data so that each word (or token) appears in its own row.
This format makes text easier to count, filter, and analyze in R.
Install packages
Run this installation code once in the RStudio Console if needed.
install.packages(c(
"dplyr",
"tibble",
"tidytext",
"janeaustenr",
"stringr",
"ggplot2"
))Load libraries
library(dplyr)
library(tibble)
library(tidytext)
library(janeaustenr)
library(stringr)
library(ggplot2)1.2 — Tokenizing text with unnest_tokens()
For a small example, I use four lines from Emily Dickinson’s poem Because I could not stop for Death.
Each element contains a complete line of text.
Original text
text <- c(
"Because I could not stop for Death -",
"He kindly stopped for me -",
"The Carriage held but just Ourselves -",
"and Immortality"
)
text## [1] "Because I could not stop for Death -"
## [2] "He kindly stopped for me -"
## [3] "The Carriage held but just Ourselves -"
## [4] "and Immortality"Convert text into a tibble
text_df <- tibble(
line = seq_along(text),
text = text
)
text_df## # A tibble: 4 × 2
## line text
## <int> <chr>
## 1 1 Because I could not stop for Death -
## 2 2 He kindly stopped for me -
## 3 3 The Carriage held but just Ourselves -
## 4 4 and ImmortalitySeparate text into individual words
The unnest_tokens() function creates one row per
word.
By default, it converts words to lowercase and removes punctuation.
tidy_poem <- text_df %>%
unnest_tokens(word, text)
tidy_poem## # A tibble: 20 × 2
## line word
## <int> <chr>
## 1 1 because
## 2 1 i
## 3 1 could
## 4 1 not
## 5 1 stop
## 6 1 for
## 7 1 death
## 8 2 he
## 9 2 kindly
## 10 2 stopped
## 11 2 for
## 12 2 me
## 13 3 the
## 14 3 carriage
## 15 3 held
## 16 3 but
## 17 3 just
## 18 3 ourselves
## 19 4 and
## 20 4 immortalityObservation
The output preserves each word’s original line number while converting the poem into one-word-per-row data.
This structure makes it easier to examine individual words instead of full sentences.
1.3 — Tidying Jane Austen’s novels
The janeaustenr package provides six novels written by
Jane Austen.
This larger dataset makes it possible to explore word patterns across many pages of text.
Load Jane Austen’s novels
original_books <- austen_books() %>%
group_by(book) %>%
mutate(
linenumber = row_number(),
chapter = cumsum(str_detect(
text,
regex(
"^chapter\\s+([0-9]+|[ivxlcdm]+)\\b",
ignore_case = TRUE
)
))
) %>%
ungroup()
original_books## # A tibble: 73,422 × 4
## text book linenumber chapter
## <chr> <fct> <int> <int>
## 1 "SENSE AND SENSIBILITY" Sense & Sensibility 1 0
## 2 "" Sense & Sensibility 2 0
## 3 "by Jane Austen" Sense & Sensibility 3 0
## 4 "" Sense & Sensibility 4 0
## 5 "(1811)" Sense & Sensibility 5 0
## 6 "" Sense & Sensibility 6 0
## 7 "" Sense & Sensibility 7 0
## 8 "" Sense & Sensibility 8 0
## 9 "" Sense & Sensibility 9 0
## 10 "CHAPTER 1" Sense & Sensibility 10 1
## # ℹ 73,412 more rowsTokenize the novels
tidy_books <- original_books %>%
unnest_tokens(word, text)
tidy_books## # A tibble: 725,064 × 4
## book linenumber chapter word
## <fct> <int> <int> <chr>
## 1 Sense & Sensibility 1 0 sense
## 2 Sense & Sensibility 1 0 and
## 3 Sense & Sensibility 1 0 sensibility
## 4 Sense & Sensibility 3 0 by
## 5 Sense & Sensibility 3 0 jane
## 6 Sense & Sensibility 3 0 austen
## 7 Sense & Sensibility 5 0 1811
## 8 Sense & Sensibility 10 1 chapter
## 9 Sense & Sensibility 10 1 1
## 10 Sense & Sensibility 13 1 the
## # ℹ 725,054 more rowsEach row now contains the book name, line number, chapter, and one word.
Remove stop words
Common words such as “the,” “and,” and “is” can overwhelm the results.
The stop_words dataset identifies these words, and
anti_join() removes them.
clean_books <- tidy_books %>%
anti_join(stop_words, by = "word")
clean_books## # A tibble: 217,602 × 4
## book linenumber chapter word
## <fct> <int> <int> <chr>
## 1 Sense & Sensibility 1 0 sense
## 2 Sense & Sensibility 1 0 sensibility
## 3 Sense & Sensibility 3 0 jane
## 4 Sense & Sensibility 3 0 austen
## 5 Sense & Sensibility 5 0 1811
## 6 Sense & Sensibility 10 1 chapter
## 7 Sense & Sensibility 10 1 1
## 8 Sense & Sensibility 13 1 family
## 9 Sense & Sensibility 13 1 dashwood
## 10 Sense & Sensibility 13 1 settled
## # ℹ 217,592 more rowsCount word frequencies
Word frequency measures how often individual words appear in the dataset.
word_counts <- clean_books %>%
count(word, sort = TRUE)
word_counts## # A tibble: 13,910 × 2
## word n
## <chr> <int>
## 1 miss 1855
## 2 time 1337
## 3 fanny 862
## 4 dear 822
## 5 lady 817
## 6 sir 806
## 7 day 797
## 8 emma 787
## 9 sister 727
## 10 house 699
## # ℹ 13,900 more rowsObservation
Character names and frequently used terms appear near the top of the results.
This reveals patterns in the vocabulary used throughout Jane Austen’s novels.
However, word frequency alone cannot determine the meaning or emotional tone of a sentence.
Visualize word frequencies
The following chart displays words that appear more than 600 times after removing stop words.
word_counts %>%
filter(n > 600) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(x = n, y = word)) +
geom_col(
fill = "#42657B",
width = 0.75
) +
labs(
title = "Most Frequent Words in Jane Austen's Novels",
subtitle = "Word counts after removing stop words",
x = "Number of occurrences",
y = NULL
) +
theme_minimal(base_size = 12) +
theme(
plot.title = element_text(face = "bold"),
panel.grid.major.y = element_blank()
)
Marketing Application
Text mining provides marketers with an effective way to understand customer feedback and behavior.
Businesses can apply the same process to customer reviews, social media comments, and survey responses.
For example, frequently mentioned words such as “price,” “delivery,” or “quality” may identify topics that customers consider important.
By examining these patterns, companies can recognize recurring customer concerns and improve marketing strategies.
However, marketers should examine the original text before drawing conclusions, since word frequency does not necessarily indicate positive or negative sentiment.
Conclusion
This exercise demonstrated how unstructured text can be transformed
into organized data using the tidytext package in R.
Through tokenization, stop-word removal, and word frequency analysis, I learned how to identify and visualize commonly used words.
These techniques provide a foundation for further text mining and can help businesses better understand customer communication and preferences.
Reference
Silge, J., & Robinson, D. (2017). Text Mining with R: A Tidy Approach.