Lab 12: Text Analysis - Session 2

1. Setup and Loading Libraries

First, we load the required packages for data manipulation, tidy text mining, visualization, and accessing Jane Austen’s novels.

library(dplyr)
library(stringr)
library(tidytext)
library(janeaustenr)
library(ggplot2)

---

2. Unstructured Text vs. Tidy Text & Tokenization (Section 1.1 - 1.2)

• Unstructured text consists of raw strings, paragraphs, or documents without a standardized tabular format.
• Tidy text follows the tidy data principle: a table with one-token-per-row.
• Tokenization is the process of splitting text into meaningful units (tokens), such as individual words, using unnest_tokens().

Below, we take the raw text from Jane Austen’s books, track line and chapter numbers, and tokenize the text into individual words.

# Organize raw book text with line and chapter tracking
original_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(str_detect(text, 
                                regex("^chapter [\\divxlc]",
                                      ignore_case = TRUE)))
  ) %>%
  ungroup()

# Convert unstructured text into one-word-per-row tidy format
tidy_books <- original_books %>%
  unnest_tokens(word, text)

# Preview the tidy dataset
head(tidy_books)

## # A tibble: 6 × 4
##   book                linenumber chapter word       
##   <fct>                    <int>   <int> <chr>      
## 1 Sense & Sensibility          1       0 sense      
## 2 Sense & Sensibility          1       0 and        
## 3 Sense & Sensibility          1       0 sensibility
## 4 Sense & Sensibility          3       0 by         
## 5 Sense & Sensibility          3       0 jane       
## 6 Sense & Sensibility          3       0 austen

---

3. Removing Stop Words (Section 1.3)

Stop words are extremely common words (such as “the”, “of”, “to”) that add little analytical value. We remove them using anti_join() with the built-in stop_words dataset.

data(stop_words)

tidy_books_clean <- tidy_books %>%
  anti_join(stop_words)

# Preview the cleaned dataset
head(tidy_books_clean)

## # A tibble: 6 × 4
##   book                linenumber chapter word       
##   <fct>                    <int>   <int> <chr>      
## 1 Sense & Sensibility          1       0 sense      
## 2 Sense & Sensibility          1       0 sensibility
## 3 Sense & Sensibility          3       0 jane       
## 4 Sense & Sensibility          3       0 austen     
## 5 Sense & Sensibility          5       0 1811       
## 6 Sense & Sensibility         10       1 chapter

---

4. Calculating and Visualizing Word Frequencies (Section 1.3)

Now that stop words are removed, we can count the most frequent words across the dataset and visualize words that appear more than 600 times.

# Calculate top word counts
word_counts <- tidy_books_clean %>%
  count(word, sort = TRUE)

head(word_counts, 10)

## # A tibble: 10 × 2
##    word       n
##    <chr>  <int>
##  1 miss    1855
##  2 time    1337
##  3 fanny    862
##  4 dear     822
##  5 lady     817
##  6 sir      806
##  7 day      797
##  8 emma     787
##  9 sister   727
## 10 house    699

# Visualize words appearing more than 600 times
tidy_books_clean %>%
  count(word, sort = TRUE) %>%
  filter(n > 600) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word)) +
  geom_col(fill = "steelblue") +
  labs(
    title = "Most Frequent Words in Jane Austen's Novels",
    subtitle = "Words occurring more than 600 times (excluding stop words)",
    x = "Frequency (Count)",
    y = NULL
  ) +
  theme_minimal()

---

5. Marketing Interpretation

• Key Findings: After removing stop words, the most frequent words are “miss” (1,855 occurrences) and “time” (1,337 occurrences), followed by terms like “dear”, “lady”, “sister”, and “house”.
• Application to Marketing Decisions: In a business context, applying this exact workflow to customer reviews, survey responses, or social media mentions allows marketers to quantify unstructured feedback. High-frequency keywords highlight core customer priorities, recurring product features, or common pain points—enabling teams to refine brand messaging, improve customer experience, and tailor promotional campaigns to the exact language their audience uses.