This exercise follows Sections 1.1–1.3 of Chapter 1 of Julia Silge and David Robinson’s Text Mining with R. It converts text into individual words, removes stop words, and visualizes frequent terms. The literary dataset demonstrates the method; the final discussion explains how the same workflow could inform marketing research.
Unstructured text stores language as passages rather than separate analytical variables. A review such as “The delivery was fast” contains several words in one field. A table of reviews can be structured at the review level while its text still needs tokenization for word-level analysis.
Tidy text uses one token per row, with source identifiers retained. A token is the unit being analyzed: here, one word. Tokenization splits a passage into those units.
example_reviews <- tibble(
review_id = c(1L, 2L),
text = c("The delivery was fast!", "The packaging was damaged.")
)
knitr::kable(example_reviews, caption = "Before: one review per row")
| review_id | text |
|---|---|
| 1 | The delivery was fast! |
| 2 | The packaging was damaged. |
example_tokens <- example_reviews %>%
unnest_tokens(output = word, input = text)
knitr::kable(example_tokens, caption = "After: one word occurrence per row")
| review_id | word |
|---|---|
| 1 | the |
| 1 | delivery |
| 1 | was |
| 1 | fast |
| 2 | the |
| 2 | packaging |
| 2 | was |
| 2 | damaged |
These two sentences are invented teaching examples, not collected
customer feedback. In unnest_tokens(word, text),
word names the new output column and text
names the input column. By default, word tokenization lowercases words
and removes punctuation. The review identifier remains available. See
the function
documentation.
The janeaustenr package supplies six novels. We retain
each book and its line number so a frequent word can be traced back to
its context.
original_books <- austen_books() %>%
group_by(book) %>%
mutate(line_number = row_number()) %>%
ungroup()
tidy_books <- original_books %>%
unnest_tokens(word, text)
knitr::kable(head(tidy_books, 12), caption = "A sample of the tokenized novels")
| book | line_number | word |
|---|---|---|
| Sense & Sensibility | 1 | sense |
| Sense & Sensibility | 1 | and |
| Sense & Sensibility | 1 | sensibility |
| Sense & Sensibility | 3 | by |
| Sense & Sensibility | 3 | jane |
| Sense & Sensibility | 3 | austen |
| Sense & Sensibility | 5 | 1811 |
| Sense & Sensibility | 10 | chapter |
| Sense & Sensibility | 10 | 1 |
| Sense & Sensibility | 13 | the |
| Sense & Sensibility | 13 | family |
| Sense & Sensibility | 13 | of |
Each row now represents a word occurrence, so repeated words appear repeatedly. Tokenization does not automatically combine forms such as “walk” and “walked.”
Stop words are common terms that an analyst may exclude to focus on
content. Here we use tidytext’s built-in English list, which combines
three lexicons. anti_join() removes tokens matching the
list by word. See the stop-word
documentation.
stop_list <- tidytext::stop_words %>% distinct(word)
removed_words <- tidy_books %>%
semi_join(stop_list, by = "word") %>%
count(word, sort = TRUE)
knitr::kable(head(removed_words, 10),
caption = "Frequent words removed by the stop list")
| word | n |
|---|---|
| the | 26351 |
| to | 24044 |
| and | 22515 |
| of | 21178 |
| a | 13408 |
| her | 13055 |
| i | 12006 |
| in | 11217 |
| was | 11204 |
| it | 10234 |
clean_books <- tidy_books %>%
anti_join(stop_list, by = "word") %>%
filter(grepl("[[:alpha:]]", word))
audit <- tibble(
stage = c("All word tokens", "After stop-word removal", "After excluding tokens without letters"),
tokens = c(nrow(tidy_books),
nrow(anti_join(tidy_books, stop_list, by = "word")),
nrow(clean_books))
)
knitr::kable(audit, caption = "How preprocessing changes the dataset")
| stage | tokens |
|---|---|
| All word tokens | 725064 |
| After stop-word removal | 217602 |
| After excluding tokens without letters | 217396 |
The final filter excludes tokens such as numeric chapter labels. It is separate from stop-word removal. Stop lists require judgment: removing “not” from a review can erase a critical part of its meaning. For sentiment analysis, preserve the original text and reconsider negation before applying a generic list.
word_counts <- clean_books %>%
count(word, sort = TRUE) %>%
arrange(desc(n), word) %>%
mutate(percent_of_retained_tokens = 100 * n / sum(n))
knitr::kable(head(word_counts, 15), digits = 2,
col.names = c("Word", "Occurrences", "% of retained tokens"),
caption = "Most frequent retained words across all six novels")
| Word | Occurrences | % of retained tokens |
|---|---|---|
| miss | 1855 | 0.85 |
| time | 1337 | 0.62 |
| fanny | 862 | 0.40 |
| dear | 822 | 0.38 |
| lady | 817 | 0.38 |
| sir | 806 | 0.37 |
| day | 797 | 0.37 |
| emma | 787 | 0.36 |
| sister | 727 | 0.33 |
| house | 699 | 0.32 |
| elizabeth | 687 | 0.32 |
| elinor | 623 | 0.29 |
| hope | 601 | 0.28 |
| friend | 593 | 0.27 |
| family | 578 | 0.27 |
The percentage denominator is all retained word occurrences across the six novels. It is not the percentage of novels or readers mentioning a term.
word_counts %>%
slice_head(n = 15) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(x = n, y = word)) +
geom_col(fill = "#217A89", width = 0.75) +
scale_x_continuous(expand = expansion(mult = c(0, 0.06))) +
labs(title = "The 15 most frequent retained words in Jane Austen's novels",
subtitle = "English stop words and tokens without letters excluded",
x = "Word occurrences", y = NULL,
caption = "Source: janeaustenr; all six novels pooled") +
theme_minimal(base_size = 12) +
theme(panel.grid.major.y = element_blank(),
panel.grid.minor = element_blank())
After preprocessing, the dataset contains 217,396 word occurrences and 13,829 distinct words. The most frequent retained word is “miss”, with 1,855 occurrences, accounting for 0.85% of retained tokens. The next two are “time” and “fanny.” These values are calculated when the report is knitted.
Frequency shows which terms recur under these preprocessing choices. Names and forms of address may rank highly because characters recur throughout novels. A high count alone does not establish a theme’s importance or a word’s emotional meaning. Pooling also gives longer novels more influence.
To examine usage rather than guessing from the chart, the following table retrieves original lines containing the leading word:
leading_word <- word_counts$word[1]
context_keys <- clean_books %>%
filter(word == leading_word) %>%
distinct(book, line_number) %>%
slice_head(n = 5)
original_books %>%
semi_join(context_keys, by = c("book", "line_number")) %>%
knitr::kable(caption = "Original lines containing the leading word")
| text | book | line_number |
|---|---|---|
| of so large a sum? And what possible claim could the Miss Dashwoods, | Sense & Sensibility | 189 |
| probability be settled for life. We shall miss her; but SHE will be | Sense & Sensibility | 478 |
| he had now procured for his cottage at Barton. The Miss Dashwoods were | Sense & Sensibility | 992 |
| honour of calling tomorrow to enquire after Miss Dashwood. The honour | Sense & Sensibility | 1306 |
| tell you, Miss Dashwood; he has a pretty little estate of his own in | Sense & Sensibility | 1369 |
The same workflow can summarize product reviews, survey responses, or support comments. Retaining a review ID, product, channel, and date would allow comparisons and checks against the original wording.
| Hypothetical review finding | Possible decision | Validation before acting |
|---|---|---|
| “Packaging” appears frequently | Investigate packaging quality or instructions | Read the comments to distinguish praise from damage complaints |
| Customers repeatedly mention “delivery” | Investigate fulfillment or test delivery-focused messaging | Separate speed, reliability, and cost; compare products and periods |
| “Battery” recurs in one product’s reviews | Review product information and battery performance | Inspect context and compare the share of reviews mentioning it |
These are hypothetical applications, not findings from Austen’s novels. Word frequency can identify topics for investigation and customer language for message development. It cannot establish satisfaction, purchase intent, or which campaign will increase sales. A follow-up survey or controlled campaign test would be needed to evaluate those questions.
For differently sized customer segments, compare proportions rather than raw counts. Review-level mention rates are often useful because one verbose reviewer can repeat a word many times. Inspect phrases such as “not fast,” protect customer identifiers, and check whether the feedback represents the wider customer base.
This exercise demonstrates the sequence text → tokens → stop-word filtering → frequency table → chart → contextual interpretation. The main analytical lesson is that transforming text makes counting possible, while context and the business question determine what those counts mean.
unnest_tokens()
reference and stop_words
reference.sessionInfo()
## R version 4.6.2 (2026-10-29)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
## [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
## [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
## [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
##
## time zone: UTC
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggplot2_4.0.3 janeaustenr_1.0.0 tidytext_0.4.3 tibble_3.3.1
## [5] dplyr_1.2.1
##
## loaded via a namespace (and not attached):
## [1] Matrix_1.7-6 gtable_0.3.6 jsonlite_2.0.0 compiler_4.6.2
## [5] tidyselect_1.2.1 Rcpp_1.1.2 stringr_1.6.0 jquerylib_0.1.4
## [9] scales_1.4.0 yaml_2.3.12 fastmap_1.2.0 lattice_0.23-1
## [13] R6_2.6.1 labeling_0.4.3 SnowballC_0.7.1 generics_0.1.4
## [17] knitr_1.52 RColorBrewer_1.1-3 bslib_0.12.0 pillar_1.11.1
## [21] tokenizers_0.3.0 rlang_1.3.0 cachem_1.1.0 stringi_1.8.9
## [25] xfun_0.62 S7_0.2.2 sass_0.4.10 otel_0.3.0
## [29] cli_3.6.6 withr_3.0.3 magrittr_2.0.5 digest_0.6.39
## [33] grid_4.6.2 rstudioapi_0.19.0 lifecycle_1.0.5 vctrs_0.7.3
## [37] evaluate_1.0.5 glue_1.8.1 farver_2.1.2 purrr_1.2.2
## [41] rmarkdown_2.32 tools_4.6.2 pkgconfig_2.0.3 htmltools_0.5.9