Lab 12: Text Analysis — Session 2
Genesis Delgadillo
November 12, 2026
Purpose
This lab follows sections 1.1–1.3 of Julia Silge and David Robinson’s Text Mining with R. It demonstrates how to organize text, separate words, remove common words, and create a word-frequency chart. It also explains how the same process could help a business study customer comments.
If the packages are not installed, run this command in the RStudio Console once before clicking Knit. This installation block is not run during knitting.
1.1 Contrasting tidy text with other data structures
Unstructured text is ordinary writing, such as a review or paragraph. Its words are not separated into observations that can easily be counted.
| Structure | How it stores text |
|---|---|
| Character vector | A sequence of text strings, such as sentences or lines. |
| Corpus | A collection of documents with information about them. |
| Document-term matrix | Documents are rows, terms are columns, and cells contain values such as counts. |
| Tidy text | Each row contains one token and identifying information about its source. |
For this lab, a token is one word. Tokenization separates text into these smaller units. Tidy text makes it easier to filter and count words while keeping track of where they appeared.
1.2 Using unnest_tokens()
The example below uses the same four lines of Emily Dickinson’s public-domain poem that appear in the textbook. First, I store one line per row.
poem <- tibble::tibble(
line = 1:4,
text = c(
"Because I could not stop for Death -",
"He kindly stopped for me -",
"The Carriage held but just Ourselves -",
"and Immortality"
)
)
knitr::kable(poem, caption = "Original text: one line per row")| line | text |
|---|---|
| 1 | Because I could not stop for Death - |
| 2 | He kindly stopped for me - |
| 3 | The Carriage held but just Ourselves - |
| 4 | and Immortality |
Next, I split each line into separate words.
poem_words <- unnest_tokens(poem, output = word, input = text)
knitr::kable(poem_words, caption = "Tidy text: one word per row")| line | word |
|---|---|
| 1 | because |
| 1 | i |
| 1 | could |
| 1 | not |
| 1 | stop |
| 1 | for |
| 1 | death |
| 2 | he |
| 2 | kindly |
| 2 | stopped |
| 2 | for |
| 2 | me |
| 3 | the |
| 3 | carriage |
| 3 | held |
| 3 | but |
| 3 | just |
| 3 | ourselves |
| 4 | and |
| 4 | immortality |
The original text column becomes a new word
column. The default settings make words lowercase and remove
punctuation. The line column stays with each word, so its
source can still be identified. The four input rows become 20
word rows.
1.3 Tidying the works of Jane Austen
Load and identify the text
The janeaustenr package supplies six completed novels. I
keep the book title, number each line within its book, and count chapter
headings.
book_lines <- austen_books() %>%
group_by(book) %>%
mutate(
line = row_number(),
chapter = cumsum(str_detect(
text,
regex("^chapter\\s+[0-9ivxlc]+", ignore_case = TRUE)
))
) %>%
ungroup()
knitr::kable(head(book_lines, 10), caption = "First ten lines of the source data")| text | book | line | chapter |
|---|---|---|---|
| SENSE AND SENSIBILITY | Sense & Sensibility | 1 | 0 |
| Sense & Sensibility | 2 | 0 | |
| by Jane Austen | Sense & Sensibility | 3 | 0 |
| Sense & Sensibility | 4 | 0 | |
| (1811) | Sense & Sensibility | 5 | 0 |
| Sense & Sensibility | 6 | 0 | |
| Sense & Sensibility | 7 | 0 | |
| Sense & Sensibility | 8 | 0 | |
| Sense & Sensibility | 9 | 0 | |
| CHAPTER 1 | Sense & Sensibility | 10 | 1 |
The chapter value starts at zero before the first chapter heading. It counts detected headings within each book; it does not attempt to preserve printed chapter numbers when numbering restarts in a new volume.
Convert the novels to tidy text
book_words <- unnest_tokens(book_lines, output = word, input = text)
knitr::kable(head(book_words, 10), caption = "First ten word rows")| book | line | chapter | word |
|---|---|---|---|
| Sense & Sensibility | 1 | 0 | sense |
| Sense & Sensibility | 1 | 0 | and |
| Sense & Sensibility | 1 | 0 | sensibility |
| Sense & Sensibility | 3 | 0 | by |
| Sense & Sensibility | 3 | 0 | jane |
| Sense & Sensibility | 3 | 0 | austen |
| Sense & Sensibility | 5 | 0 | 1811 |
| Sense & Sensibility | 10 | 1 | chapter |
| Sense & Sensibility | 10 | 1 | 1 |
| Sense & Sensibility | 13 | 1 | the |
Identify and remove stop words
Stop words include common words such as “the,” “and,” and “of.”
Removing them helps other terms stand out in a frequency analysis. The
stop_words dataset combines three English word lists.
stop_list <- tidytext::stop_words %>%
distinct(word)
clean_words <- book_words %>%
anti_join(stop_list, by = "word")
processing_summary <- tibble::tibble(
measure = c(
"Novels", "Original text lines", "Words before stop-word removal",
"Words after stop-word removal", "Word rows removed"
),
value = c(
n_distinct(book_lines$book), nrow(book_lines), nrow(book_words),
nrow(clean_words), nrow(book_words) - nrow(clean_words)
)
)
knitr::kable(
processing_summary,
col.names = c("Measure", "Count"),
format.args = list(big.mark = ","),
caption = "Text-processing summary"
)| Measure | Count |
|---|---|
| Novels | 6 |
| Original text lines | 73,422 |
| Words before stop-word removal | 725,064 |
| Words after stop-word removal | 217,602 |
| Word rows removed | 507,462 |
anti_join() retains words that do not appear in the
stop-word list. This removes every matching word occurrence while
retaining the book and line information. I do not remove names or
numbers, or combine word endings, so this analysis follows the basic
cleaning approach in the assigned sections.
Calculate word frequencies
word_counts <- clean_words %>%
count(word, name = "frequency", sort = TRUE)
top_words <- head(word_counts, 10)
knitr::kable(
top_words,
col.names = c("Word", "Frequency"),
format.args = list(big.mark = ","),
caption = "Ten most frequent remaining words across all six novels"
)| Word | Frequency |
|---|---|
| miss | 1,855 |
| time | 1,337 |
| fanny | 862 |
| dear | 822 |
| lady | 817 |
| sir | 806 |
| day | 797 |
| emma | 787 |
| sister | 727 |
| house | 699 |
Each frequency is the total number of occurrences across the six novels. It is not the number of books containing that word.
Visualize the results
ggplot(top_words, aes(x = frequency, y = reorder(word, frequency))) +
geom_col(fill = "#286778", width = 0.72) +
geom_text(aes(label = format(frequency, big.mark = ",")),
hjust = -0.15, size = 3.8) +
scale_x_continuous(expand = expansion(mult = c(0, 0.18))) +
labs(
title = "Most frequent words in Jane Austen's novels",
subtitle = "Top ten words after removing stop words",
x = "Number of occurrences", y = NULL
) +
theme_minimal(base_size = 12) +
theme(panel.grid.major.y = element_blank(),
panel.grid.minor = element_blank())
Ten most frequent words after stop-word removal. Counts are pooled across the six novels.
Interpret the results
The most frequent remaining word is miss, with 1,855 occurrences. The second is time, with 1,337 occurrences. The horizontal bars make the differences in these counts easy to compare.
Names such as “Fanny” and “Emma,” and terms such as “lady,” “sir,” and “sister,” stand out in the textbook’s results. These words suggest that characters, relationships, and social forms of address are useful topics for closer reading. A count alone does not prove a theme or show how a word was used. For example, “miss” can be a title or a verb.
These are pooled raw counts. Longer novels can contribute more occurrences. Comparing individual books fairly would require adjusting for their lengths.
How text mining can support marketing decisions
A business could apply this workflow to customer reviews, survey responses, or comments on social media. Separating the words and counting them can reveal topics customers mention repeatedly.
For example, if “delivery” and “late” frequently appear in reviews, a retailer could inspect those comments and investigate delivery problems. If customers repeatedly praise “quality,” the company could consider emphasizing product quality in its advertising. These are hypothetical examples, not results from the Jane Austen dataset.
Word counts are a starting point. “Good quality” and “not good quality” communicate different opinions even though both contain “quality.” Before making decisions, the business should read the surrounding comments, check whether the feedback is positive or negative, and consider whether the reviewers represent its wider customer base. Removing stop words can also remove words such as “not,” so cleaning choices should match the question.
Conclusion
This workflow turns writing into a table that can be analyzed. Tokenization separates words, stop-word removal reduces common-word clutter, and a frequency table and chart show which remaining terms appear most often. For marketing, the same process can help identify customer concerns and strengths to examine more closely.
Sources
- Silge, J., & Robinson, D. Text Mining with R: A Tidy Approach, sections 1.1–1.3. The poem example and Jane Austen workflow follow these assigned sections.
- tidytext documentation:
unnest_tokens()andstop_words. - Jane Austen novels supplied by the
janeaustenrpackage.
Reproducibility
The following records the R and package versions used when this file is knitted.
## R version 4.6.2 (2026-10-29)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
## [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
## [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
## [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
##
## time zone: UTC
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggplot2_4.0.3 stringr_1.6.0 janeaustenr_1.0.0 tidytext_0.4.3
## [5] dplyr_1.2.1
##
## loaded via a namespace (and not attached):
## [1] Matrix_1.7-6 gtable_0.3.6 jsonlite_2.0.0 compiler_4.6.2
## [5] tidyselect_1.2.1 Rcpp_1.1.2 jquerylib_0.1.4 scales_1.4.0
## [9] yaml_2.3.12 fastmap_1.2.0 lattice_0.23-1 R6_2.6.1
## [13] labeling_0.4.3 SnowballC_0.7.1 generics_0.1.4 knitr_1.52
## [17] tibble_3.3.1 RColorBrewer_1.1-3 bslib_0.12.0 pillar_1.11.1
## [21] tokenizers_0.3.0 rlang_1.3.0 cachem_1.1.0 stringi_1.8.9
## [25] xfun_0.62 S7_0.2.2 sass_0.4.10 cli_3.6.6
## [29] withr_3.0.3 magrittr_2.0.5 digest_0.6.39 grid_4.6.2
## [33] lifecycle_1.0.5 vctrs_0.7.3 evaluate_1.0.5 glue_1.8.1
## [37] farver_2.1.2 purrr_1.2.2 rmarkdown_2.32 tools_4.6.2
## [41] pkgconfig_2.0.3 htmltools_0.5.9