The Goal

In our last class, we talked about TF-IDF, which is a tool we can use if we want to figure out what words characterize certain types of text. Today, we are going to see an application of how we can use this in practice.

The Data Set

The Federalist papers are a famous series of 97 articles written in the 1700s in the early days of the United States. They debated and proposed ideas for the new government. Three famous authors of these papers were Alexander Hamilton, James Madison, and John Jay.

Most of the Federalist papers (85 of them) were known to have been written by either Hamilton, Madison, or Jay, but for 12 papers, the papers were submitted anonymously. After Madison’s death, Hamilton declared that he had written the articles. Since then, many scholars have worked to determine which of the three authors had written the anonymous papers using text analysis.

Our goal for today is to see if we can use the text analysis skills we have learned so far to determine who wrote the mystery papers.

To read in the data on the \(n = 97\) Federalist papers, use the following code:

# Load the data
Federalist <- read.csv("https://www.dropbox.com/scl/fi/lb7n6eoc094epytqqkuc2/FederalistAll.csv?rlkey=nhrr5m90iyy7bueggffslxm0e&st=5jvps3wz&dl=1")

# Convert to a data frame
Federalist <- data.frame(Federalist)

# Make sure author is treated as categorical
Federalist[, "author"] <- as.factor(Federalist[, "author"])

The columns are:

• paper: the number given to the paper; think of this like an identifier for the article.
• text: the text of the entire paper.
• author: either Hamilton or Madison or Jay or mystery (for the 12 papers with unknown author)

Once you have loaded the data, load the packages you will need for this lab:

library(tidytext)
library(tidyr)
library(dplyr)
library(ggplot2)
library(tm)
library(naivebayes)

# NEW 
library(stringr)
library(forcats)

Feature Engineering

Our goal for today is to determine which of the three authors (Hamilton, Madison, or Jay) likely wrote each of the mystery papers. This means we need to figure out which words characterize or distinguish each author’s writing, and then see if those words are present in the mystery papers!

It may seem natural to use frequency as a way to determine which words we should use as features. In other words, we could the number of times each word occurs in a text (in this case a paper) and we choose the words that are most frequent for each author as our features. However, it turns out that this is not the best approach. Let’s see why.

Suppose we want to find the top 5 words in Madison’s articles after removing stop words. I’ve given you a skeleton of the code you need below:

top5_Madison <- Federalist |>
  filter(author == ...) |>
  unnest_tokens(word,... ) |>
  anti_join(stop_words, by = join_by(word)) |>
  count( ...   ) |>
  slice_max(n , n = ... )

ggplot( ... , aes(n, fct_reorder( ... , ... ) )) +
  geom_...() +
  labs( x = ... , y = ..., title = ...)

This shows us potential words to use as features for Madison, but we also need words for Hamilton and Jay and the mystery author. We could repeat the process 3 times, but it turns out we actually don’t have to. As we saw in class, we can use grouping in R to find the top 5 words for each author in one code, without having to repeat! A skeleton code for this is included below.

top5_all <- Federalist |>
  unnest_tokens(word,... ) |>
  anti_join(stop_words, by = join_by(word)) |>
  group_by( ... ) |>
  count( ... ) |>
  slice_max(n , n = ... )

We can also create a plot to compare the authors using the following code.

ggplot( top5_all , aes(n, reorder_within( word , n, author),fill =author)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~author,ncol = 2, scales = "free_y") +
  scale_y_reordered() + 
  labs( x = ... , y = ..., title = ...)

You will note two new things in this plotting code.

• reorder_within: We use fct_reorder when we want to make sure that the words appear in order on our bar graph. However, because we now have three different authors, we actually need to make sure that the words appear in order for all authors. reorder_within allows us to include author in our ordering so that happens!

• facet_wrap(): This is what allows us to make one plot for each group. In this case, facet_wrap(~author) allows to make one plot for each author.

The goal is for us to find words we can use as features in a model to predict author. For this purpose, it is helpful if we can find words that help us separate the writing of the three authors, meaning finding words that tend to be used commonly by one author but not the others. A word that is used often by multiple authors is not useful in determining among these authors for prediction.

All of this means that frequency alone is not enough. Instead, we need something that will help us find words that are commonly used by an author, but not commonly used by other authors. This is exactly the set up that motivates TF-IDF.

TF-IDF

The TF-IDF is a number that is high if a word is commonly used by an author but not commonly used by other authors. In other words, words with high TF-IDF scores fulfill the properties that we want in features for our Naive Bayes model to predict author.

As we learned in class, the TF-IDF score has two components: the TF score and IDF score.

Recall that the TF of word \(i\) in text group \(j\) is defined as:

\[TF_{i,j} = \frac{\text{Number of times word i appears in group j}}{\text{Total Number of words in group j}}\]

Once we have the TF score, the next step is to compute the IDF score.

The TF-IDF is computed by multiplying the TF and the IDF together.

tfidf_all <- Federalist |>

  unnest_tokens(word, text)|>
  
  group_by(author) |>
  
  count(word) |>
  
  # NEW!! Create the TF IDF Score
  bind_tf_idf(word, author, n)

tfidf_all <- Federalist |>

  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  group_by(author) |>
  
  count(word) |>
  
  bind_tf_idf(word, author, n)

tfidf_all <- Federalist |>

  # Tokenize 
  unnest_tokens(word, text)|>
  
  # Remove the numbers 
  filter(!grepl('[0-9]', word)) |>
  
  # Remove the punctuation 
  filter(!grepl('[[:punct:]]', word)) |>
  
  # Group by author
  group_by(author) |>
  
  # Count the number of times each word appears
  # for each author
  count(word) |>
  
  # Compute the tf-idf 
  bind_tf_idf(word, author, n)

tfidf_top5 <- Federalist |>

  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  filter(!grepl('[[:punct:]]', word)) |>
  
  group_by(author) |>
  
  count(word) |>

  bind_tf_idf(word, author, n) |>
  
  slice_max( tf_idf, n = 5)

check_tfidf_match <- function( howmany , whichauthor){
  # Find the tf=idf
  tfidf_top <- Federalist |>

  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  filter(!grepl('[[:punct:]]', word)) |>
  
  filter(!grepl('madison', word)) |>
  
  filter(!grepl('hamilton', word)) |>
  
  filter(!grepl('jay', word)) |>
  
  group_by(author) |>
  
  count(word) |>

  bind_tf_idf(word, author, n) |>
  
  slice_max( tf_idf, n = howmany)
  
  whichauthor  = subset(tfidf_top, author == whichauthor)
  mystery  = subset(tfidf_top, author == "mystery")
  sum(whichauthor$word %in% mystery$word)/nrow(mystery)
}

check_tfidf_match( 10, "Madison")

References

Data

Code

The code was adapted from Chapter 3 of “Text Mining with R: A Tidy Approach”, written by Julia Silge and David Robinson. The book was last built on 2024-06-20.

Activity