1 Purpose and scope

This exercise follows Sections 1.1–1.3 of Chapter 1 of Julia Silge and David Robinson’s Text Mining with R. It converts text into individual words, removes stop words, and visualizes frequent terms. The literary dataset demonstrates the method; the final discussion explains how the same workflow could inform marketing research.

2 Unstructured text and tidy text

Unstructured text stores language as passages rather than separate analytical variables. A review such as “The delivery was fast” contains several words in one field. A table of reviews can be structured at the review level while its text still needs tokenization for word-level analysis.

Tidy text uses one token per row, with source identifiers retained. A token is the unit being analyzed: here, one word. Tokenization splits a passage into those units.

example_reviews <- tibble(
  review_id = c(1L, 2L),
  text = c("The delivery was fast!", "The packaging was damaged.")
)

knitr::kable(example_reviews, caption = "Before: one review per row")
Before: one review per row
review_id text
1 The delivery was fast!
2 The packaging was damaged.
example_tokens <- example_reviews %>%
  unnest_tokens(output = word, input = text)

knitr::kable(example_tokens, caption = "After: one word occurrence per row")
After: one word occurrence per row
review_id word
1 the
1 delivery
1 was
1 fast
2 the
2 packaging
2 was
2 damaged

These two sentences are invented teaching examples, not collected customer feedback. In unnest_tokens(word, text), word names the new output column and text names the input column. By default, word tokenization lowercases words and removes punctuation. The review identifier remains available. See the function documentation.

3 Tokenizing Jane Austen’s novels

The janeaustenr package supplies six novels. We retain each book and its line number so a frequent word can be traced back to its context.

original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(line_number = row_number()) %>%
  ungroup()

tidy_books <- original_books %>%
  unnest_tokens(word, text)

knitr::kable(head(tidy_books, 12), caption = "A sample of the tokenized novels")
A sample of the tokenized novels
book line_number word
Sense & Sensibility 1 sense
Sense & Sensibility 1 and
Sense & Sensibility 1 sensibility
Sense & Sensibility 3 by
Sense & Sensibility 3 jane
Sense & Sensibility 3 austen
Sense & Sensibility 5 1811
Sense & Sensibility 10 chapter
Sense & Sensibility 10 1
Sense & Sensibility 13 the
Sense & Sensibility 13 family
Sense & Sensibility 13 of

Each row now represents a word occurrence, so repeated words appear repeatedly. Tokenization does not automatically combine forms such as “walk” and “walked.”

4 Identifying and removing stop words

Stop words are common terms that an analyst may exclude to focus on content. Here we use tidytext’s built-in English list, which combines three lexicons. anti_join() removes tokens matching the list by word. See the stop-word documentation.

stop_list <- tidytext::stop_words %>% distinct(word)

removed_words <- tidy_books %>%
  semi_join(stop_list, by = "word") %>%
  count(word, sort = TRUE)

knitr::kable(head(removed_words, 10),
             caption = "Frequent words removed by the stop list")
Frequent words removed by the stop list
word n
the 26351
to 24044
and 22515
of 21178
a 13408
her 13055
i 12006
in 11217
was 11204
it 10234
clean_books <- tidy_books %>%
  anti_join(stop_list, by = "word") %>%
  filter(grepl("[[:alpha:]]", word))

audit <- tibble(
  stage = c("All word tokens", "After stop-word removal", "After excluding tokens without letters"),
  tokens = c(nrow(tidy_books),
             nrow(anti_join(tidy_books, stop_list, by = "word")),
             nrow(clean_books))
)
knitr::kable(audit, caption = "How preprocessing changes the dataset")
How preprocessing changes the dataset
stage tokens
All word tokens 725064
After stop-word removal 217602
After excluding tokens without letters 217396

The final filter excludes tokens such as numeric chapter labels. It is separate from stop-word removal. Stop lists require judgment: removing “not” from a review can erase a critical part of its meaning. For sentiment analysis, preserve the original text and reconsider negation before applying a generic list.

5 Calculating and visualizing frequencies

word_counts <- clean_books %>%
  count(word, sort = TRUE) %>%
  arrange(desc(n), word) %>%
  mutate(percent_of_retained_tokens = 100 * n / sum(n))

knitr::kable(head(word_counts, 15), digits = 2,
             col.names = c("Word", "Occurrences", "% of retained tokens"),
             caption = "Most frequent retained words across all six novels")
Most frequent retained words across all six novels
Word Occurrences % of retained tokens
miss 1855 0.85
time 1337 0.62
fanny 862 0.40
dear 822 0.38
lady 817 0.38
sir 806 0.37
day 797 0.37
emma 787 0.36
sister 727 0.33
house 699 0.32
elizabeth 687 0.32
elinor 623 0.29
hope 601 0.28
friend 593 0.27
family 578 0.27

The percentage denominator is all retained word occurrences across the six novels. It is not the percentage of novels or readers mentioning a term.

word_counts %>%
  slice_head(n = 15) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(x = n, y = word)) +
  geom_col(fill = "#217A89", width = 0.75) +
  scale_x_continuous(expand = expansion(mult = c(0, 0.06))) +
  labs(title = "The 15 most frequent retained words in Jane Austen's novels",
       subtitle = "English stop words and tokens without letters excluded",
       x = "Word occurrences", y = NULL,
       caption = "Source: janeaustenr; all six novels pooled") +
  theme_minimal(base_size = 12) +
  theme(panel.grid.major.y = element_blank(),
        panel.grid.minor = element_blank())

6 Interpreting the results

After preprocessing, the dataset contains 217,396 word occurrences and 13,829 distinct words. The most frequent retained word is “miss”, with 1,855 occurrences, accounting for 0.85% of retained tokens. The next two are “time” and “fanny.” These values are calculated when the report is knitted.

Frequency shows which terms recur under these preprocessing choices. Names and forms of address may rank highly because characters recur throughout novels. A high count alone does not establish a theme’s importance or a word’s emotional meaning. Pooling also gives longer novels more influence.

To examine usage rather than guessing from the chart, the following table retrieves original lines containing the leading word:

leading_word <- word_counts$word[1]
context_keys <- clean_books %>%
  filter(word == leading_word) %>%
  distinct(book, line_number) %>%
  slice_head(n = 5)

original_books %>%
  semi_join(context_keys, by = c("book", "line_number")) %>%
  knitr::kable(caption = "Original lines containing the leading word")
Original lines containing the leading word
text book line_number
of so large a sum? And what possible claim could the Miss Dashwoods, Sense & Sensibility 189
probability be settled for life. We shall miss her; but SHE will be Sense & Sensibility 478
he had now procured for his cottage at Barton. The Miss Dashwoods were Sense & Sensibility 992
honour of calling tomorrow to enquire after Miss Dashwood. The honour Sense & Sensibility 1306
tell you, Miss Dashwood; he has a pretty little estate of his own in Sense & Sensibility 1369

7 How this supports marketing decisions

The same workflow can summarize product reviews, survey responses, or support comments. Retaining a review ID, product, channel, and date would allow comparisons and checks against the original wording.

Hypothetical review finding Possible decision Validation before acting
“Packaging” appears frequently Investigate packaging quality or instructions Read the comments to distinguish praise from damage complaints
Customers repeatedly mention “delivery” Investigate fulfillment or test delivery-focused messaging Separate speed, reliability, and cost; compare products and periods
“Battery” recurs in one product’s reviews Review product information and battery performance Inspect context and compare the share of reviews mentioning it

These are hypothetical applications, not findings from Austen’s novels. Word frequency can identify topics for investigation and customer language for message development. It cannot establish satisfaction, purchase intent, or which campaign will increase sales. A follow-up survey or controlled campaign test would be needed to evaluate those questions.

For differently sized customer segments, compare proportions rather than raw counts. Review-level mention rates are often useful because one verbose reviewer can repeat a word many times. Inspect phrases such as “not fast,” protect customer identifiers, and check whether the feedback represents the wider customer base.

8 Reflection

This exercise demonstrates the sequence text → tokens → stop-word filtering → frequency table → chart → contextual interpretation. The main analytical lesson is that transforming text makes counting possible, while context and the business question determine what those counts mean.

9 References

10 Reproducibility

sessionInfo()
## R version 4.6.2 (2026-10-29)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.5 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## time zone: UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] ggplot2_4.0.3     janeaustenr_1.0.0 tidytext_0.4.3    tibble_3.3.1     
## [5] dplyr_1.2.1      
## 
## loaded via a namespace (and not attached):
##  [1] Matrix_1.7-6       gtable_0.3.6       jsonlite_2.0.0     compiler_4.6.2    
##  [5] tidyselect_1.2.1   Rcpp_1.1.2         stringr_1.6.0      jquerylib_0.1.4   
##  [9] scales_1.4.0       yaml_2.3.12        fastmap_1.2.0      lattice_0.23-1    
## [13] R6_2.6.1           labeling_0.4.3     SnowballC_0.7.1    generics_0.1.4    
## [17] knitr_1.52         RColorBrewer_1.1-3 bslib_0.12.0       pillar_1.11.1     
## [21] tokenizers_0.3.0   rlang_1.3.0        cachem_1.1.0       stringi_1.8.9     
## [25] xfun_0.62          S7_0.2.2           sass_0.4.10        otel_0.3.0        
## [29] cli_3.6.6          withr_3.0.3        magrittr_2.0.5     digest_0.6.39     
## [33] grid_4.6.2         rstudioapi_0.19.0  lifecycle_1.0.5    vctrs_0.7.3       
## [37] evaluate_1.0.5     glue_1.8.1         farver_2.1.2       purrr_1.2.2       
## [41] rmarkdown_2.32     tools_4.6.2        pkgconfig_2.0.3    htmltools_0.5.9