The Goal

In our last class, we talked about TF-IDF, which is a tool we can use if we want to figure out what words characterize certain types of text. Today, we are going to see an application of how we can use this in practice.

The Data Set

The Federalist papers are a famous series of articles written in the 1700s in the early days of the United States. They debated and proposed ideas for the new government. Three famous authors of these papers were Alexander Hamilton, James Madison, and John Jay.

Most of the Federalist papers were known to have been written by either Hamilton, Madison, or Jay, but for 12 papers, the papers were submitted anonymously. After Madison’s death, Hamilton declared that he had written the articles. Since then, many scholars have worked to determine which of the three authors had written the anonymous papers using text analysis.

Our goal for today is to see if we can use the text analysis skills we have learned so far to determine who wrote the mystery papers.

To read in the data on the \(n = 85\) Federalist papers with known authors, use the following code:

# Load the data
Federalist <- read.csv("https://www.dropbox.com/scl/fi/5hzqwsvlnym5u1mhmn0jy/Federalist.csv?rlkey=55x0p9fl02zls9ixxek64vlqy&st=isjnztof&dl=1")

# Convert to a data frame
Federalist <- data.frame(Federalist)

# Make sure author is treated as categorical
Federalist[, "author"] <- as.factor(Federalist[, "author"])

The columns are:

• paper: the number given to the paper; think of this like an identifier for the article.
• text: the text of the entire paper.
• author: either Hamilton or Madison or Jay

Once you have loaded the data, load the packages you will need for this lab:

library(tidytext)
library(tidyr)
library(dplyr)
library(ggplot2)
library(tm)
library(naivebayes)

# NEW 
library(stringr)
library(forcats)

Modeling: Naive Bayes

So far, the only model we have learned for text data is Naive Bayes, though that will change very soon! This means we will be using Naive Bayes to predict the author for the 12 mystery papers after using the 85 papers with known authors to train our model.

To train a Naive Bayes model, we need features. We have been exploring techniques like bag of words models that use words for features, so let’s use an approach like that today. We learned that a key is to choose words that will help us determine which author wrote each paper, and we can use the presence of these words in each paper as features to predict author.

It may seem natural to use frequency as a way to determine which words we should use as features. In other words, we could the number of times each word occurs in a text (in this case a paper) and we choose the words that are most frequent for each author as our features. However, it turns out that this is not the best approach. Let’s see why.

Suppose we want to find the top 10 words in Madison’s articles after removing stop words. I’ve given you a skeleton of the code you need below:

top10_Madison <- Federalist |>
  filter(author == ...) |>
  unnest_tokens(word,... ) |>
  anti_join(stop_words, by = join_by(word)) |>
  count( ...   ) |>
  slice_max(n , n = ... )

ggplot( ... , aes(n, fct_reorder( ... , ... ) )) +
  geom_...() +
  labs( x = ... , y = ..., title = ...)

This shows us potential words to use as features for Madison, but we also need words for Hamilton and Jay. We could repeat the process 3 times, but it turns out we actually don’t have to. As we saw in class, we can use grouping in R to find the top 10 words for each author in one code, without having to repeat! A skeleton code for this is included below.

top10_all <- Federalist |>
  unnest_tokens(word,... ) |>
  anti_join(stop_words, by = join_by(word)) |>
  count( ...   ) |>
  group_by( ... ) |>
  slice_max(n , n = ... )

We can also create a plot to compare the authors using the following code.

ggplot( top10_all , aes(n, reorder_within( word , n, author),fill =author)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~author,ncol = 2, scales = "free_y") +
  scale_y_reordered() + 
  labs( x = ... , y = ..., title = ...)

You will note two new things in this plotting code.

• reorder_within: We use fct_reorder when we want to make sure that the words appear in order on our bar graph. However, because we now have three different authors, we actually need to make sure that the words appear in order for all authors. reorder_within allows us to include author in our ordering so that happens!

• facet_wrap(): This is what allows us to make one plot for each group. In this case, facet_wrap(~author) allows to make one plot for each author.

The goal is for us to find words we can use as features in a model to predict author. For this purpose, it is helpful if we can find words that help us separate the writing of the three authors, meaning finding words that tend to be used commonly by one author but not the others. A word that is used often by multiple authors is not useful in determining among these authors for prediction.

All of this means that frequency alone is not enough. Instead, we need something that will help us find words that are commonly used by an author, but not commonly used by other authors. This is exactly the set up that motivates TF-IDF.

TF-IDF

The TF-IDF is a number that is high if a word is commonly used by an author but not commonly used by other authors. In other words, words with high TF-IDF scores fulfill the properties that we want in features for our Naive Bayes model to predict author.

As we learned in class, the TF-IDF score has two components: the TF score and IDF score.

Recall that the TF of word \(i\) in text group \(j\) is defined as:

\[TF_{i,j} = \frac{\text{Number of times word i appears in group j}}{\text{Total Number of words in group j}}\]

Once we have the TF score, the next step is to compute the IDF score.

The TF-IDF is computed by multiplying the TF and the IDF together.

tfidf_all <- Federalist |>

  unnest_tokens(word, text)|>
  
  count(word, author) |>
  
  # NEW!! Create the TF IDF Score
  bind_tf_idf(word, author, n)

tfidf_all <- Federalist |>

  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  count(word, author) |>

  bind_tf_idf(word, author, n)

tfidf_all <- Federalist |>

  mutate(text = str_remove_all(text, "_")) |>

  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  count(word, author) |>
  
  bind_tf_idf(word, author, n)

tfidf_top4 <- Federalist |>

  mutate(text = str_remove_all(text, "_")) |>

  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  count(word, author) |>
  
  bind_tf_idf(word, document = author, n) |>
  
  group_by(author) |>
  
  slice_max( tf_idf, n = 4)

tfidf_top4 <- Federalist |>

  # Remove the word Hamilton (all in caps) from the text
  mutate(text = removeWords(text,"HAMILTON"))|>
  
  # Remove the word Madison (all in caps) from the text
  mutate(text = removeWords(text,"MADISON"))|>
  
  # Remove the word Jay (all in caps) from the text 
  mutate(text = removeWords(text,"JAY"))|>
  
  mutate(text = str_remove_all(text, "_")) |>
  
  unnest_tokens(word, text)|>
  
  filter(!grepl('[0-9]', word)) |>
  
  count(word, author, sort = TRUE) |>
  
  bind_tf_idf(word, document = author, n) |>
  
  group_by(author) |>
  
  slice_max(tf_idf , n = 4 )

The Anonymous Papers

To load the 12 anonymous papers, you can use the code below:

test <- read.csv("https://www.dropbox.com/scl/fi/yrfzwwn7olyaeq3mhb0bd/test.csv?rlkey=c6bs0ytyoomkq34aw7rx3bc62&st=tzvp12cp&dl=1")

Recall that Hamilton claimed to have written all of these papers. To determine whether or not this seems true, we are going create features for our Naive Bayes model using the words in tfidf_top4.The words that we want to use are all stored in tfidf_top4 in the first column. We need to create indicators for these features for both the training (Federalist) and test (test) data sets.

This is the code you need. You are not responsible for this code on exams!!!

# Create the features in the training data 
for( i in 1:12){
  Federalist[,i+3] <- grepl(paste("\\b",tfidf_top4$word[i] , "\\b", sep =""), tolower(Federalist$text))
  colnames(Federalist)[i+3] <- tfidf_top4$word[i]
}

# Create the features in the test data 

for( i in 1:12){
  test[,i+2] <- grepl(paste("\\b",tfidf_top4$word[i] , "\\b", sep=""), tolower(test$text))
  colnames(test)[i+2] <- tfidf_top4$word[i]
}

When you run the codes above, Federalist should be 85 rows with 15 columns, and test should be 12 rows with 14 columns.

holder <- table( yhat, Federalist$author)
rownames(holder) <- c("Predicted Hamilton", "Predicted Jay", "Predicted Madison")
colnames(holder) <- c("True Hamilton", "True Jay", "True Madison")

knitr::kable(holder)

At this point, we are ready to do what we set out to do in the beginning - make predictions to see who we think wrote the anonymous papers in test.

This is what we can do with the features we have so far, but there are a few things to think about. We could have used more than 4 words as features - could this have changed our predictions? Are there other features we could create with text, like length of papers or writing style features? What about emotional context or tone?? All of these are things we can now consider as we have practiced our modeling foundations, and creating new features and seeing what we can do with them is our next step!

References

Data

Code

The code was adapted from Chapter 3 of “Text Mining with R: A Tidy Approach”, written by Julia Silge and David Robinson. The book was last built on 2024-06-20.

Activity