Introduction

In this assignment, I use tidy text analysis to organize text into individual words, remove common stop words, and visualize word frequencies. These steps make unstructured text easier to analyze.

Create a Text Dataset

First, I create a text vector using four lines from Emily Dickinson’s poem “Because I could not stop for Death.”

text <- c("Because I could not stop for Death -",
          "He kindly stopped for me -",
          "The Carriage held but just Ourselves -",
          "and Immortality")

text
## [1] "Because I could not stop for Death -"  
## [2] "He kindly stopped for me -"            
## [3] "The Carriage held but just Ourselves -"
## [4] "and Immortality"

Next, I organize the text into a tibble with one row per line.

text_df <- tibble(line = 1:4, text = text)

text_df
## # A tibble: 4 × 2
##    line text                                  
##   <int> <chr>                                 
## 1     1 Because I could not stop for Death -  
## 2     2 He kindly stopped for me -            
## 3     3 The Carriage held but just Ourselves -
## 4     4 and Immortality

Tokenization

Tokenization splits text into smaller units, such as words. The unnest_tokens() function creates one word per row and, by default, converts words to lowercase and removes punctuation.

text_df %>%
  unnest_tokens(word, text)
## # A tibble: 20 × 2
##     line word       
##    <int> <chr>      
##  1     1 because    
##  2     1 i          
##  3     1 could      
##  4     1 not        
##  5     1 stop       
##  6     1 for        
##  7     1 death      
##  8     2 he         
##  9     2 kindly     
## 10     2 stopped    
## 11     2 for        
## 12     2 me         
## 13     3 the        
## 14     3 carriage   
## 15     3 held       
## 16     3 but        
## 17     3 just       
## 18     3 ourselves  
## 19     4 and        
## 20     4 immortality

Analyze Jane Austen’s Books

I use the janeaustenr package to load Jane Austen’s books. I also add line numbers and chapter numbers within each book.

original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(
      str_detect(
        text,
        regex("^chapter [\\divxlc]", ignore_case = TRUE)
      )
    )
  ) %>%
  ungroup()

original_books
## # A tibble: 73,422 × 4
##    text                    book                linenumber chapter
##    <chr>                   <fct>                    <int>   <int>
##  1 "SENSE AND SENSIBILITY" Sense & Sensibility          1       0
##  2 ""                      Sense & Sensibility          2       0
##  3 "by Jane Austen"        Sense & Sensibility          3       0
##  4 ""                      Sense & Sensibility          4       0
##  5 "(1811)"                Sense & Sensibility          5       0
##  6 ""                      Sense & Sensibility          6       0
##  7 ""                      Sense & Sensibility          7       0
##  8 ""                      Sense & Sensibility          8       0
##  9 ""                      Sense & Sensibility          9       0
## 10 "CHAPTER 1"             Sense & Sensibility         10       1
## # ℹ 73,412 more rows

Next, I convert the books into a tidy dataset with one word per row.

tidy_books <- original_books %>%
  unnest_tokens(word, text)

tidy_books
## # A tibble: 725,064 × 4
##    book                linenumber chapter word       
##    <fct>                    <int>   <int> <chr>      
##  1 Sense & Sensibility          1       0 sense      
##  2 Sense & Sensibility          1       0 and        
##  3 Sense & Sensibility          1       0 sensibility
##  4 Sense & Sensibility          3       0 by         
##  5 Sense & Sensibility          3       0 jane       
##  6 Sense & Sensibility          3       0 austen     
##  7 Sense & Sensibility          5       0 1811       
##  8 Sense & Sensibility         10       1 chapter    
##  9 Sense & Sensibility         10       1 1          
## 10 Sense & Sensibility         13       1 the        
## # ℹ 725,054 more rows

Remove Stop Words

Stop words are common words, such as “the” and “and,” that often provide limited information in a word frequency analysis. I remove them using anti_join().

data(stop_words)

tidy_books <- tidy_books %>%
  anti_join(stop_words, by = "word")

Count Word Frequencies

I count how often each remaining word appears across all six books and sort the results from most to least frequent.

word_counts <- tidy_books %>%
  count(word, sort = TRUE)

word_counts
## # A tibble: 13,910 × 2
##    word       n
##    <chr>  <int>
##  1 miss    1855
##  2 time    1337
##  3 fanny    862
##  4 dear     822
##  5 lady     817
##  6 sir      806
##  7 day      797
##  8 emma     787
##  9 sister   727
## 10 house    699
## # ℹ 13,900 more rows

Visualize Word Frequencies

This bar chart shows words that appear more than 600 times after stop words are removed.

word_counts %>%
  filter(n > 600) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(x = n, y = word)) +
  geom_col(fill = "steelblue") +
  labs(
    title = "Frequent Words in Jane Austen’s Books",
    subtitle = "Words appearing more than 600 times, excluding stop words",
    x = "Word Frequency",
    y = NULL
  ) +
  theme_minimal()

Reflection

This exercise helped me understand how unstructured text becomes tidy data with one word per row. I used tokenization, removed stop words, and counted and visualized frequently used words. Words such as “miss,” “time,” and character names appear often, although frequency alone does not explain their context. In marketing, these methods could help identify common topics in customer reviews and guide further analysis of customer feedback.

Source

Code and examples adapted from Julia Silge and David Robinson’s Text Mining with R, Chapter 1: https://www.tidytextmining.com/tidytext.