1. Unstructured Text and Tidy Text

Unstructured text contains sentences or paragraphs that have not been organized into individual words for analysis. Tidy text organizes text into a table with one token per row. A token is a unit of text, such as a word. This structure makes it easier to filter, count, and compare words.

2. Tokenization with unnest_tokens()

Tokenization means splitting text into smaller units. In this example, I use unnest_tokens() to split sentences into individual words. By default, this function converts words to lowercase and removes punctuation.

Original text

sample_text <- tibble(
  line = 1:3,
  text = c(
    "Customers enjoy helpful service.",
    "Helpful service builds customer trust.",
    "Customers value quality products."
  )
)

sample_text
## # A tibble: 3 × 2
##    line text                                  
##   <int> <chr>                                 
## 1     1 Customers enjoy helpful service.      
## 2     2 Helpful service builds customer trust.
## 3     3 Customers value quality products.

One word per row

tidy_sample <- sample_text %>%
  unnest_tokens(word, text)

tidy_sample
## # A tibble: 13 × 2
##     line word     
##    <int> <chr>    
##  1     1 customers
##  2     1 enjoy    
##  3     1 helpful  
##  4     1 service  
##  5     2 helpful  
##  6     2 service  
##  7     2 builds   
##  8     2 customer 
##  9     2 trust    
## 10     3 customers
## 11     3 value    
## 12     3 quality  
## 13     3 products

The original table has one sentence per row. After tokenization, each row contains one word, and the line column identifies the sentence it came from. ## 3. Tidying Jane Austen’s Novels

For this analysis, I use Jane Austen’s six novels from the janeaustenr package. The original data contains one line of text per row. I keep the book name and add line and chapter information before splitting the text into words.

original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(
      str_detect(text, regex("^chapter [\\divxlc]",
                             ignore_case = TRUE))
    )
  ) %>%
  ungroup()

head(original_books)
## # A tibble: 6 × 4
##   text                    book                linenumber chapter
##   <chr>                   <fct>                    <int>   <int>
## 1 "SENSE AND SENSIBILITY" Sense & Sensibility          1       0
## 2 ""                      Sense & Sensibility          2       0
## 3 "by Jane Austen"        Sense & Sensibility          3       0
## 4 ""                      Sense & Sensibility          4       0
## 5 "(1811)"                Sense & Sensibility          5       0
## 6 ""                      Sense & Sensibility          6       0
tidy_books <- original_books %>%
  unnest_tokens(word, text)

head(tidy_books)
## # A tibble: 6 × 4
##   book                linenumber chapter word       
##   <fct>                    <int>   <int> <chr>      
## 1 Sense & Sensibility          1       0 sense      
## 2 Sense & Sensibility          1       0 and        
## 3 Sense & Sensibility          1       0 sensibility
## 4 Sense & Sensibility          3       0 by         
## 5 Sense & Sensibility          3       0 jane       
## 6 Sense & Sensibility          3       0 austen

4. Removing Stop Words

Stop words are common words, such as “the,” “and,” and “of,” that often contribute little to a basic word frequency analysis. I use anti_join() to remove words found in the tidytext stop-word list.

clean_books <- tidy_books %>%
  anti_join(stop_words, by = "word")

tibble(
  stage = c("Before removing stop words",
            "After removing stop words"),
  word_count = c(nrow(tidy_books), nrow(clean_books))
)
## # A tibble: 2 × 2
##   stage                      word_count
##   <chr>                           <int>
## 1 Before removing stop words     725055
## 2 After removing stop words      217609

Removing stop words helps highlight names and other words that may reveal topics. However, the choice of stop words depends on the purpose of the analysis.

5. Word Frequencies

I count how often each remaining word appears across the six novels and display the 10 most frequent words.

word_counts <- clean_books %>%
  count(word, sort = TRUE)

knitr::kable(
  head(word_counts, 10),
  col.names = c("Word", "Frequency"),
  caption = "Ten Most Frequent Words After Stop-Word Removal"
)
Ten Most Frequent Words After Stop-Word Removal
Word Frequency
miss 1855
time 1337
fanny 862
dear 822
lady 817
sir 806
day 797
emma 787
sister 727
house 699

6. Word Frequency Chart

word_counts %>%
  slice_max(n, n = 10, with_ties = FALSE) %>%
  ggplot(aes(x = n, y = reorder(word, n))) +
  geom_col(fill = "steelblue") +
  labs(
    title = "Most Frequent Words in Jane Austen’s Novels",
    subtitle = "After removing stop words",
    x = "Number of Occurrences",
    y = NULL
  ) +
  theme_minimal()

7. Interpretation and Marketing Applications

After removing stop words, the most frequent word is miss, appearing 1,855 times. The chart highlights recurring names, forms of address, and everyday terms. Names can appear frequently because particular characters receive substantial attention. Word counts show repetition, but they do not explain the meaning or emotional tone of each occurrence.

A business could apply the same process to customer reviews, survey responses, or social media posts. Frequently mentioned words could suggest product features or service issues that deserve closer review. For example, repeated mentions of delivery could prompt a company to investigate shipping experiences. Marketers could use these patterns to develop advertising ideas or identify questions for additional research. They should read the original comments before making decisions because frequent mentions alone do not establish whether customers are satisfied.

Source

Silge, J., & Robinson, D. Text Mining with R: A Tidy Approach. Sections 1.1–1.3. https://www.tidytextmining.com/tidytext