Introduction

This exercise explores how to organize text for analysis. I begin with a short example, then analyze word frequencies in Jane Austen’s novels. The goal is to understand how written information can become data that supports business decisions.

1.1 Unstructured text and tidy text

Unstructured text consists of written content, such as reviews or comments, that has not been organized into analytical variables. Tidy text places one token in each row, making individual words easier to filter and count.

Here is an original example using fictional customer reviews.

reviews <- tibble(
  review_id = 1:3,
  text = c(
    "Great shoes and great customer service.",
    "The shoes are comfortable for soccer practice.",
    "Delivery was slow but the shoes feel comfortable."
  )
)

knitr::kable(reviews)
review_id text
1 Great shoes and great customer service.
2 The shoes are comfortable for soccer practice.
3 Delivery was slow but the shoes feel comfortable.

Although this table has one review per row, each review still contains multiple words.

1.2 Tokenization with unnest_tokens()

Tokenization means dividing text into smaller units. In this example, each token is a word.

review_words <- reviews %>%
  unnest_tokens(word, text)

knitr::kable(review_words)
review_id word
1 great
1 shoes
1 and
1 great
1 customer
1 service
2 the
2 shoes
2 are
2 comfortable
2 for
2 soccer
2 practice
3 delivery
3 was
3 slow
3 but
3 the
3 shoes
3 feel
3 comfortable

In unnest_tokens(word, text), word names the new output column and text identifies the input column. The review ID stays attached to each word.

1.3 Tidying Jane Austen’s novels

The janeaustenr package provides the text of six novels. First, I inspect the original data.

books <- austen_books()

knitr::kable(head(books, 10))
text book
SENSE AND SENSIBILITY Sense & Sensibility
Sense & Sensibility
by Jane Austen Sense & Sensibility
Sense & Sensibility
(1811) Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
CHAPTER 1 Sense & Sensibility

Next, I convert each line into individual words while retaining the book title.

book_words <- books %>%
  unnest_tokens(word, text)

knitr::kable(head(book_words, 15))
book word
Sense & Sensibility sense
Sense & Sensibility and
Sense & Sensibility sensibility
Sense & Sensibility by
Sense & Sensibility jane
Sense & Sensibility austen
Sense & Sensibility 1811
Sense & Sensibility chapter
Sense & Sensibility 1
Sense & Sensibility the
Sense & Sensibility family
Sense & Sensibility of
Sense & Sensibility dashwood
Sense & Sensibility had
Sense & Sensibility long

Remove stop words

Stop words are common words, such as “the” and “and,” that are often removed when exploring topics through word frequency.

clean_words <- book_words %>%
  anti_join(stop_words, by = "word")

word_summary <- tibble(
  stage = c("Before removing stop words", "After removing stop words"),
  word_occurrences = c(nrow(book_words), nrow(clean_words))
)

knitr::kable(
  word_summary,
  col.names = c("Stage", "Word occurrences"),
  format.args = list(big.mark = ",")
)
Stage Word occurrences
Before removing stop words 725,064
After removing stop words 217,602

Calculate word frequencies

word_counts <- clean_words %>%
  count(word, sort = TRUE)

top_words <- word_counts %>%
  slice_head(n = 15)

knitr::kable(
  top_words,
  col.names = c("Word", "Frequency")
)
Word Frequency
miss 1855
time 1337
fanny 862
dear 822
lady 817
sir 806
day 797
emma 787
sister 727
house 699
elizabeth 687
elinor 623
hope 601
friend 593
family 578

Visualize the most frequent words

ggplot(
  top_words,
  aes(x = n, y = reorder(word, n))
) +
  geom_col(fill = "#246B8E") +
  labs(
    title = "15 Most Frequent Words in Jane Austen's Novels",
    subtitle = "After removing stop words",
    x = "Number of occurrences",
    y = NULL
  ) +
  theme_minimal(base_size = 12)

Interpretation

The most frequent remaining word is “miss,” appearing 1,855 times. The chart also includes character names and words describing social relationships. These results suggest topics to investigate further, but reading the surrounding passages would help explain their meaning. A frequent word alone does not establish a theme or an opinion.

How text mining supports marketing decisions

A business could use this process to study customer reviews, survey responses, or social media comments. For example, a sportswear company could count mentions of comfort, sizing, price, and delivery to identify topics customers discuss frequently.

As a business administration and management student, I could use this information to help prioritize questions for customer research or identify possible service improvements. However, frequency alone does not show satisfaction. A customer saying “comfortable” and another saying “not comfortable” express different opinions. Businesses should check context before making decisions.

Conclusion

This exercise demonstrates how to tokenize text, remove stop words, count words, and present the results in a chart. Organizing text into rows makes written information easier to explore with data analysis tools.

Reference

Silge, J., & Robinson, D. Text Mining with R: A Tidy Approach. Sections 1.1–1.3. https://www.tidytextmining.com/tidytext