1.1 Text Data and Tidy Text

Text often comes in sentences or paragraphs, such as customer reviews or social media posts. Tidy text organizes text into a table with one token per row. A token is a unit of text, such as a word. This structure makes it easier to filter, count, and analyze words.

1.2 Tokenization with unnest_tokens()

Tokenization splits text into smaller units. In this example, I use unnest_tokens() to separate a short poem into individual words.

text <- c(
  "Because I could not stop for Death -",
  "He kindly stopped for me -",
  "The Carriage held but just Ourselves -",
  "and Immortality"
)

text_df <- tibble(line = 1:4, text = text)

text_df
## # A tibble: 4 × 2
##    line text                                  
##   <int> <chr>                                 
## 1     1 Because I could not stop for Death -  
## 2     2 He kindly stopped for me -            
## 3     3 The Carriage held but just Ourselves -
## 4     4 and Immortality
tidy_text <- text_df %>%
  unnest_tokens(word, text)

tidy_text
## # A tibble: 20 × 2
##     line word       
##    <int> <chr>      
##  1     1 because    
##  2     1 i          
##  3     1 could      
##  4     1 not        
##  5     1 stop       
##  6     1 for        
##  7     1 death      
##  8     2 he         
##  9     2 kindly     
## 10     2 stopped    
## 11     2 for        
## 12     2 me         
## 13     3 the        
## 14     3 carriage   
## 15     3 held       
## 16     3 but        
## 17     3 just       
## 18     3 ourselves  
## 19     4 and        
## 20     4 immortality

The original table contains four rows, with one line of the poem in each row. After tokenization, the table contains 20 rows, with one word per row. By default, unnest_tokens() converts words to lowercase and removes punctuation while keeping the line numbers.

1.3 Tidying the Works of Jane Austen

For this analysis, I use the six novels included in the janeaustenr package. First, I keep track of each line and chapter within each book.

original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(
      str_detect(text, regex("^chapter [\\divxlc]", ignore_case = TRUE))
    )
  ) %>%
  ungroup()

head(original_books)
## # A tibble: 6 × 4
##   text                    book                linenumber chapter
##   <chr>                   <fct>                    <int>   <int>
## 1 "SENSE AND SENSIBILITY" Sense & Sensibility          1       0
## 2 ""                      Sense & Sensibility          2       0
## 3 "by Jane Austen"        Sense & Sensibility          3       0
## 4 ""                      Sense & Sensibility          4       0
## 5 "(1811)"                Sense & Sensibility          5       0
## 6 ""                      Sense & Sensibility          6       0

Converting the Books into Tidy Text

I use unnest_tokens() to separate the books into individual words while retaining the book, line number, and chapter.

tidy_books <- original_books %>%
  unnest_tokens(word, text)

head(tidy_books)
## # A tibble: 6 × 4
##   book                linenumber chapter word       
##   <fct>                    <int>   <int> <chr>      
## 1 Sense & Sensibility          1       0 sense      
## 2 Sense & Sensibility          1       0 and        
## 3 Sense & Sensibility          1       0 sensibility
## 4 Sense & Sensibility          3       0 by         
## 5 Sense & Sensibility          3       0 jane       
## 6 Sense & Sensibility          3       0 austen

Removing Stop Words

Stop words are common words, such as “the,” “and,” and “of,” that often add little information to a word frequency analysis. I use anti_join() to remove words found in the stop_words dataset.

tidy_books <- tidy_books %>%
  anti_join(stop_words, by = "word")

head(tidy_books)
## # A tibble: 6 × 4
##   book                linenumber chapter word       
##   <fct>                    <int>   <int> <chr>      
## 1 Sense & Sensibility          1       0 sense      
## 2 Sense & Sensibility          1       0 sensibility
## 3 Sense & Sensibility          3       0 jane       
## 4 Sense & Sensibility          3       0 austen     
## 5 Sense & Sensibility          5       0 1811       
## 6 Sense & Sensibility         10       1 chapter

Counting Word Frequencies

After removing stop words, I count how often each remaining word appears across the six novels.

word_counts <- tidy_books %>%
  count(word, sort = TRUE)

head(word_counts, 10)
## # A tibble: 10 × 2
##    word       n
##    <chr>  <int>
##  1 miss    1855
##  2 time    1337
##  3 fanny    862
##  4 dear     822
##  5 lady     817
##  6 sir      806
##  7 day      797
##  8 emma     787
##  9 sister   727
## 10 house    699

Visualizing Word Frequencies

The graph displays words that appear more than 600 times across the novels after stop words are removed.

word_counts %>%
  filter(n > 600) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(x = n, y = word)) +
  geom_col(fill = "steelblue") +
  labs(
    title = "Most Common Words in Jane Austen's Novels",
    x = "Number of Occurrences",
    y = NULL
  ) +
  theme_minimal()

Interpreting the Results

After removing stop words, “miss” is the most frequent word, appearing 1,855 times, followed by “time” with 1,337 occurrences. Names such as “fanny” and “emma” also appear frequently. Words such as “lady,” “sir,” and “sister” suggest that social titles and family relationships are common in these novels. However, word counts alone do not explain how each word is used, so reading the surrounding text would help confirm these patterns.

Marketing Application

Marketers can use this same process to analyze customer reviews, survey responses, and social media posts. Frequently mentioned words such as “price,” “delivery,” or “quality” can identify topics that deserve further investigation. These findings can help businesses improve products, address service problems, and develop relevant advertising. However, frequency alone does not show whether customers feel positively or negatively about a topic.

Source

This exercise follows Sections 1.1–1.3 of Julia Silge and David Robinson’s Text Mining with R: A Tidy Approach. The poem example is by Emily Dickinson, and the novel texts come from the janeaustenr package.

https://www.tidytextmining.com/tidytext