R Markdown

This lab follows sections 1.1–1.3 of Text Mining with R. Unstructured text contains sentences or paragraphs. Tidy text places each word on its own row so the words can be filtered and counted. A character vector stores text strings, a corpus groups documents, and a document-term matrix organizes counts by document and word.

text <- c("Because I could not stop for Death -",
          "He kindly stopped for me -",
          "The Carriage held but just Ourselves -",
          "and Immortality")

text
## [1] "Because I could not stop for Death -"   "He kindly stopped for me -"            
## [3] "The Carriage held but just Ourselves -" "and Immortality"

Get the tidy text dataset

library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
text_df <- tibble(line = 1:4, text = text)

text_df
## # A tibble: 4 × 2
##    line text                                  
##   <int> <chr>                                 
## 1     1 Because I could not stop for Death -  
## 2     2 He kindly stopped for me -            
## 3     3 The Carriage held but just Ourselves -
## 4     4 and Immortality

Tokenization separates text into smaller units. Here, each token is one word. The line number identifies where each word came from.

library(tidytext)

tidy_text <- text_df %>%
  unnest_tokens(word, text)

print(tidy_text, n = 20)
## # A tibble: 20 × 2
##     line word       
##    <int> <chr>      
##  1     1 because    
##  2     1 i          
##  3     1 could      
##  4     1 not        
##  5     1 stop       
##  6     1 for        
##  7     1 death      
##  8     2 he         
##  9     2 kindly     
## 10     2 stopped    
## 11     2 for        
## 12     2 me         
## 13     3 the        
## 14     3 carriage   
## 15     3 held       
## 16     3 but        
## 17     3 just       
## 18     3 ourselves  
## 19     4 and        
## 20     4 immortality

The four lines become 20 word rows. The default settings convert words to lowercase and remove punctuation.

1.3 Tidying the works of Jane Austen

library(janeaustenr)
library(dplyr)
library(stringr)

original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(str_detect(
      text,
      regex("^chapter [\\divxlc]", ignore_case = TRUE)
    ))
  ) %>%
  ungroup()

original_books
## # A tibble: 73,422 × 4
##    text                    book                linenumber chapter
##    <chr>                   <fct>                    <int>   <int>
##  1 "SENSE AND SENSIBILITY" Sense & Sensibility          1       0
##  2 ""                      Sense & Sensibility          2       0
##  3 "by Jane Austen"        Sense & Sensibility          3       0
##  4 ""                      Sense & Sensibility          4       0
##  5 "(1811)"                Sense & Sensibility          5       0
##  6 ""                      Sense & Sensibility          6       0
##  7 ""                      Sense & Sensibility          7       0
##  8 ""                      Sense & Sensibility          8       0
##  9 ""                      Sense & Sensibility          9       0
## 10 "CHAPTER 1"             Sense & Sensibility         10       1
## # ℹ 73,412 more rows
library(tidytext)

tidy_books <- original_books %>%
  unnest_tokens(word, text)

tidy_books
## # A tibble: 725,064 × 4
##    book                linenumber chapter word       
##    <fct>                    <int>   <int> <chr>      
##  1 Sense & Sensibility          1       0 sense      
##  2 Sense & Sensibility          1       0 and        
##  3 Sense & Sensibility          1       0 sensibility
##  4 Sense & Sensibility          3       0 by         
##  5 Sense & Sensibility          3       0 jane       
##  6 Sense & Sensibility          3       0 austen     
##  7 Sense & Sensibility          5       0 1811       
##  8 Sense & Sensibility         10       1 chapter    
##  9 Sense & Sensibility         10       1 1          
## 10 Sense & Sensibility         13       1 the        
## # ℹ 725,054 more rows

Stop words are common words such as “the,” “and,” and “of.” Removing them helps other words stand out in the frequency results.

data("stop_words", package = "tidytext")

tidy_books <- tidy_books %>%
  anti_join(stop_words, by = "word")
word_counts <- tidy_books %>%
  count(word, sort = TRUE)

word_counts
## # A tibble: 13,910 × 2
##    word       n
##    <chr>  <int>
##  1 miss    1855
##  2 time    1337
##  3 fanny    862
##  4 dear     822
##  5 lady     817
##  6 sir      806
##  7 day      797
##  8 emma     787
##  9 sister   727
## 10 house    699
## # ℹ 13,900 more rows
library(ggplot2)

word_counts %>%
  filter(n > 600) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word)) +
  geom_col() +
  labs(y = NULL)

Interpretation and marketing use

The most frequent remaining word is miss, with 1,855 occurrences. The chart shows words appearing more than 600 times across the six novels. Names and social terms stand out, but word counts alone do not explain the context or meaning of each use.

A business could use the same process to study customer reviews. For example, frequent mentions of “delivery” could identify a topic to investigate. Reading the surrounding comments would show whether customers are pleased or complaining, helping the business decide what to improve or emphasize.

Source: Julia Silge and David Robinson, Text Mining with R, sections 1.1–1.3. The poem example and Jane Austen analysis follow the assigned textbook.