Introduction

In this lab, I explore how the tidytext package makes it possible to analyze written text with R. I focus on Sections 1.1 through 1.3 of Text Mining with R. The main steps are organizing text, separating it into words, removing stop words, and displaying word frequencies. ## Load the packages Install these packages in the Console once if needed.

install.packages(c(
  "tidytext",
  "dplyr",
  "janeaustenr",
  "stringr",
  "ggplot2"
))
library(dplyr)
library(tidytext)
library(janeaustenr)
library(stringr)
library(ggplot2)

1.1 - Text and tidy data

Text is often stored as a character vector, where each element can contain an entire sentence. This is not yet tidy text because each row can contain multiple words. In tidy text, each word is stored in its own row, making it easier to count and analyze. ## 1.2 - The unnest_tokens() function I will start with four lines from a poem by Emily Dickinson.

text <- c(
  "Because I could not stop for Death -",
  "He kindly stopped for me -",
  "The Carriage held but just Ourselves -",
  "and Immortality"
)
text
## [1] "Because I could not stop for Death -"  
## [2] "He kindly stopped for me -"            
## [3] "The Carriage held but just Ourselves -"
## [4] "and Immortality"

First, I put the poem into a tibble that keeps track of the line number.

text_df <- tibble(
  line = 1:4,
  text = text
)
text_df
## # A tibble: 4 × 2
##    line text                                  
##   <int> <chr>                                 
## 1     1 Because I could not stop for Death -  
## 2     2 He kindly stopped for me -            
## 3     3 The Carriage held but just Ourselves -
## 4     4 and Immortality

Next, I use unnest_tokens() to place each word in a separate row. R also converts the words to lowercase and removes punctuation by default.

tidy_poem <- text_df %>%
  unnest_tokens(word, text)
tidy_poem
## # A tibble: 20 × 2
##     line word       
##    <int> <chr>      
##  1     1 because    
##  2     1 i          
##  3     1 could      
##  4     1 not        
##  5     1 stop       
##  6     1 for        
##  7     1 death      
##  8     2 he         
##  9     2 kindly     
## 10     2 stopped    
## 11     2 for        
## 12     2 me         
## 13     3 the        
## 14     3 carriage   
## 15     3 held       
## 16     3 but        
## 17     3 just       
## 18     3 ourselves  
## 19     4 and        
## 20     4 immortality

Observation: Each line is now separated into individual words, while the original line number is preserved. This structure makes it possible to compare and count words instead of analyzing entire sentences manually. ## 1.3 - Tidying the works of Jane Austen The janeaustenr package includes six of Jane Austen’s published novels. I will use these books to practice the same text-mining steps with a much larger dataset. ### Load the novels

original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(str_detect(
      text,
      regex(
        "^chapter [\\divxlc]",
        ignore_case = TRUE
      )
    ))
  ) %>%
  ungroup()
original_books
## # A tibble: 73,422 × 4
##    text                    book                linenumber chapter
##    <chr>                   <fct>                    <int>   <int>
##  1 "SENSE AND SENSIBILITY" Sense & Sensibility          1       0
##  2 ""                      Sense & Sensibility          2       0
##  3 "by Jane Austen"        Sense & Sensibility          3       0
##  4 ""                      Sense & Sensibility          4       0
##  5 "(1811)"                Sense & Sensibility          5       0
##  6 ""                      Sense & Sensibility          6       0
##  7 ""                      Sense & Sensibility          7       0
##  8 ""                      Sense & Sensibility          8       0
##  9 ""                      Sense & Sensibility          9       0
## 10 "CHAPTER 1"             Sense & Sensibility         10       1
## # ℹ 73,412 more rows

The data contains each book’s text along with the book name, line number, and chapter information. ### Tokenize the novels

tidy_books <- original_books %>%
  unnest_tokens(word, text)
tidy_books
## # A tibble: 725,064 × 4
##    book                linenumber chapter word       
##    <fct>                    <int>   <int> <chr>      
##  1 Sense & Sensibility          1       0 sense      
##  2 Sense & Sensibility          1       0 and        
##  3 Sense & Sensibility          1       0 sensibility
##  4 Sense & Sensibility          3       0 by         
##  5 Sense & Sensibility          3       0 jane       
##  6 Sense & Sensibility          3       0 austen     
##  7 Sense & Sensibility          5       0 1811       
##  8 Sense & Sensibility         10       1 chapter    
##  9 Sense & Sensibility         10       1 1          
## 10 Sense & Sensibility         13       1 the        
## # ℹ 725,054 more rows

The text is now in tidy format, with one word per row. ### Remove stop words Stop words are very common terms, such as “the,” “and,” and “of.” They are often removed because they can make it harder to see more informative vocabulary.

data(stop_words)
clean_books <- tidy_books %>%
  anti_join(stop_words, by = "word")
clean_books
## # A tibble: 217,602 × 4
##    book                linenumber chapter word       
##    <fct>                    <int>   <int> <chr>      
##  1 Sense & Sensibility          1       0 sense      
##  2 Sense & Sensibility          1       0 sensibility
##  3 Sense & Sensibility          3       0 jane       
##  4 Sense & Sensibility          3       0 austen     
##  5 Sense & Sensibility          5       0 1811       
##  6 Sense & Sensibility         10       1 chapter    
##  7 Sense & Sensibility         10       1 1          
##  8 Sense & Sensibility         13       1 family     
##  9 Sense & Sensibility         13       1 dashwood   
## 10 Sense & Sensibility         13       1 settled    
## # ℹ 217,592 more rows

Find the most common words

word_counts <- clean_books %>%
  count(word, sort = TRUE)
word_counts
## # A tibble: 13,910 × 2
##    word       n
##    <chr>  <int>
##  1 miss    1855
##  2 time    1337
##  3 fanny    862
##  4 dear     822
##  5 lady     817
##  6 sir      806
##  7 day      797
##  8 emma     787
##  9 sister   727
## 10 house    699
## # ℹ 13,900 more rows

Observation: Some of the most frequent remaining words are character names and words commonly used in Jane Austen’s stories. These frequencies reveal recurring language across the novels, although they do not explain the full context of each word. ### Visualize word frequencies The chart below shows words that occur more than 600 times after removing stop words.

word_counts %>%
  filter(n > 600) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(x = n, y = word)) +
  geom_col(fill = "#42657B") +
  labs(
    title = "Most Common Words in Jane Austen's Novels",
    x = "Number of occurrences",
    y = NULL
  ) +
  theme_minimal(base_size = 12)

## Marketing application Text mining is useful beyond books. In marketing, the same process can be applied to customer reviews, social media comments, and open-ended survey responses. For example, a company could tokenize product reviews, remove stop words, and measure how often customers mention delivery, quality, price, or service. These patterns can help marketers decide which concerns deserve closer attention. However, counting words alone does not reveal whether a comment is positive or negative, so the original text should also be reviewed. ## Conclusion This exercise showed me how to transform unstructured text into tidy data with unnest_tokens(), remove common stop words using anti_join(), and calculate and visualize frequent words. Organizing text in this way creates a useful starting point for marketing analytics and further text analysis. ## Reference Silge, J., & Robinson, D. (2017). Text Mining with R: A Tidy Approach. https://www.tidytextmining.com/tidytext