Text Analysis
Logan Sparks
November 12, 2026
1 Introduction
Customer reviews, social media posts, and survey comments contain information that does not fit neatly into numeric columns. Text mining provides a way to organize that information and look for patterns. This exercise follows sections 1.1–1.3 of Chapter 1 in Text Mining with R by Julia Silge and David Robinson. It uses the book’s literary examples to practice preparing text, counting words, and interpreting a chart.
2 Section 1.1: Unstructured text and tidy text
Unstructured text is written language, such as a review or paragraph. In its original form, a single entry may contain many words. Tidy text organizes tokens into rows, with one token per row and identifying information in other columns. A token is a unit selected for analysis; in this report, it is one word (Silge & Robinson, 2017).
For example, a skincare comment such as “This moisturizer feels light” can be split into separate word rows while retaining the comment’s ID. This lets an analyst count words across many comments and trace each word back to its source.
Text may also be stored as character strings, a corpus with document
information, or a document-term matrix. Those formats serve different
purposes; tidy text is convenient for using tools such as
dplyr and ggplot2 (Silge & Robinson,
2017).
3 Section 1.2: Tokenization with unnest_tokens()
Tokenization splits text into smaller units. The following short excerpt from Emily Dickinson provides a manageable example.
poem_lines <- c(
"Because I could not stop for Death -",
"He kindly stopped for me -",
"The Carriage held but just Ourselves -",
"and Immortality"
)
poem <- tibble(line = seq_along(poem_lines), text = poem_lines)
knitr::kable(poem, caption = "Original text: one line per row")| line | text |
|---|---|
| 1 | Because I could not stop for Death - |
| 2 | He kindly stopped for me - |
| 3 | The Carriage held but just Ourselves - |
| 4 | and Immortality |
poem_words <- poem %>%
unnest_tokens(word, text)
knitr::kable(poem_words, caption = "Tidy text: one word per row")| line | word |
|---|---|
| 1 | because |
| 1 | i |
| 1 | could |
| 1 | not |
| 1 | stop |
| 1 | for |
| 1 | death |
| 2 | he |
| 2 | kindly |
| 2 | stopped |
| 2 | for |
| 2 | me |
| 3 | the |
| 3 | carriage |
| 3 | held |
| 3 | but |
| 3 | just |
| 3 | ourselves |
| 4 | and |
| 4 | immortality |
Here, word names the output column and text
identifies the input column. The default word tokenizer converts words
to lowercase and removes punctuation while retaining the line identifier
(Silge & Robinson, 2017).
The original table has 4 rows, and the tokenized table has 20 rows. That change makes individual words available for counting and filtering.
4 Section 1.3: Tidying Jane Austen’s novels
4.1 Prepare and tokenize the books
The janeaustenr package supplies six novels. The
following code preserves each book’s title, adds a line number and
chapter identifier, and creates individual word rows. Chapter 0 contains
material before the first chapter heading.
books_by_line <- austen_books() %>%
group_by(book) %>%
mutate(
line_number = row_number(),
chapter = cumsum(str_detect(
text, regex("^chapter [\\divxlc]", ignore_case = TRUE)
))
) %>%
ungroup()
book_words <- books_by_line %>%
unnest_tokens(word, text)
knitr::kable(head(book_words, 12),
caption = "A sample of the tokenized novels")| book | line_number | chapter | word |
|---|---|---|---|
| Sense & Sensibility | 1 | 0 | sense |
| Sense & Sensibility | 1 | 0 | and |
| Sense & Sensibility | 1 | 0 | sensibility |
| Sense & Sensibility | 3 | 0 | by |
| Sense & Sensibility | 3 | 0 | jane |
| Sense & Sensibility | 3 | 0 | austen |
| Sense & Sensibility | 5 | 0 | 1811 |
| Sense & Sensibility | 10 | 1 | chapter |
| Sense & Sensibility | 10 | 1 | 1 |
| Sense & Sensibility | 13 | 1 | the |
| Sense & Sensibility | 13 | 1 | family |
| Sense & Sensibility | 13 | 1 | of |
4.2 Remove stop words
Stop words are common words that often add little information to a
frequency analysis. The anti_join() below removes words
matched in tidytext::stop_words. The remaining words stay
connected to their book and line.
clean_words <- book_words %>%
anti_join(tidytext::stop_words, by = "word")
cleaning_summary <- tibble(
stage = c("Before stop-word removal", "After stop-word removal"),
word_occurrences = c(nrow(book_words), nrow(clean_words))
)
knitr::kable(cleaning_summary, format.args = list(big.mark = ","),
caption = "How stop-word removal changes the dataset")| stage | word_occurrences |
|---|---|
| Before stop-word removal | 725,064 |
| After stop-word removal | 217,602 |
Removing these words helps other vocabulary stand out. However, cleaning choices depend on the research question. In marketing, a word such as “not” can change the meaning of a review, so an analyst should inspect the original comments when interpreting a result.
4.3 Count and visualize word frequencies
word_counts <- clean_words %>%
count(word, sort = TRUE)
top_words <- word_counts %>%
slice_head(n = 15)
knitr::kable(top_words, col.names = c("Word", "Occurrences"),
caption = "The 15 most frequent words after stop-word removal")| Word | Occurrences |
|---|---|
| miss | 1855 |
| time | 1337 |
| fanny | 862 |
| dear | 822 |
| lady | 817 |
| sir | 806 |
| day | 797 |
| emma | 787 |
| sister | 727 |
| house | 699 |
| elizabeth | 687 |
| elinor | 623 |
| hope | 601 |
| friend | 593 |
| family | 578 |
top_words %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(x = n, y = word)) +
geom_col(fill = "#71618C", width = 0.75) +
geom_text(aes(label = scales::comma(n)), hjust = -0.15, size = 3.5) +
scale_x_continuous(labels = scales::comma,
expand = expansion(mult = c(0, 0.18))) +
labs(
title = "Which words appear most often?",
subtitle = "Jane Austen's six novels, after stop-word removal",
x = "Number of occurrences", y = NULL,
caption = "Source: janeaustenr; analysis adapted from Text Mining with R, Chapter 1"
) +
theme(panel.grid.major.y = element_blank(),
plot.title = element_text(face = "bold"))
The 15 most frequent remaining words across all six Jane Austen novels.
4.4 Interpret the results
The most frequent remaining word is “miss”, with 1,855 occurrences. The next two words are “time” and “fanny”. These values are calculated from the dataset when the report is knitted.
Names and forms of address can appear frequently because novels repeatedly refer to characters and their interactions. A high count therefore does not automatically make a word the central theme of a book. The chart summarizes vocabulary across the collection; it does not measure a reader’s feelings or explain why an author chose a word. Longer novels also contribute more words to the combined totals.
5 How text mining can support marketing decisions
The same workflow could be applied to skincare reviews or social media comments. Instead of keeping book titles and line numbers, an analyst could retain brand names, review IDs, and dates. Frequent references to “texture,” “price,” or “packaging” could help identify subjects for closer review and guide follow-up research.
For example, repeated mentions of “dry” could refer to dry skin, a product that feels drying, or a product that helps dryness. A marketer would need to examine the surrounding comments before recommending a change. Frequency counts can identify a subject to investigate, but they do not establish whether customers like it.
Comparing brands also requires care. A brand with more collected posts will often have larger raw word counts. Rates per review or per thousand words can make comparisons more useful. Duplicate promotional posts should be checked so that repeated advertising is not mistaken for widespread customer interest. These examples illustrate potential marketing uses; the novel dataset does not provide evidence about skincare consumers.
6 What I learned
This exercise shows how text can be transformed into a table suitable for analysis. Tokenization makes individual words available, stop-word removal reduces common vocabulary, and frequency counts reveal repeated terms. A chart makes those patterns easier to compare. For marketing decisions, the next step is to connect repeated words to their original context and check whether the collected text represents the customers being studied.
7 Reference
Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O’Reilly Media. Chapter 1, sections 1.1–1.3.
The literary examples and core workflow are adapted from this chapter. The chart styling and marketing discussion were developed for this exercise.