The Goal

We have been working with models that we can use with text data, specifically when the goal is classification. This means models that work when our response variable is categorical. We work with such data a lot when we are dealing with text.

Today, we are going to apply the models we have learned so far to the Federalist papers data set we looked at in Lab 3. In that lab, we used graphs to determine which author likely wrote the papers. In this lab, we will use models.

The Data Set

As a reminder, the Federalist papers are a famous series of articles written in the 1700s in the early days of the United States. They debated and proposed ideas for the new government. Three famous authors of these papers were Alexander Hamilton, John Jay, and James Madison. Most of the papers were know to have been written by either Hamilton or Madison or Jay, but for several papers, the author who wrote them was not known. Since then, many scholars have worked to determine which of the authors had written the anonymous papers. Today we are going to use statistical modeling to try and do the same thing.

Let’s start off by loading our libraries. Most of these will look familiar, but a few are new. The new ones are the libraries that we need for classification trees.

# Load the libraries
library(dplyr)
library(stringr)
library(ggplot2)
library(tidyr)
library(tidytext)
library(nnet)
library(tm)

# New - needed for trees
library(rpart)
library(rattle)
library(rpart.plot)

Once you have loaded the libraries, you can use the following code to load the data.

# Load the data
Federalist <- read.csv("https://www.dropbox.com/scl/fi/nche5o1oi5lag1kvoles5/LIWC_Federalist.csv?rlkey=0yvpqash13gcur6hkh7m7uwbl&st=3xntfjzh&dl=1")

# Convert to a data frame
Federalist <- data.frame(Federalist)[,-4]

# Make sure author is treated as categorical
Federalist[, "author"] <- as.factor(Federalist[, "author"])
Federalist$text <-  removeWords(Federalist$text, "hamilton")
Federalist$text <-  removeWords(Federalist$text, "madison")

Just as we did in Lab 3, our goal is to use the text of the essays to predict \(Y = author\). In Lab 3, we created features ourselves like word count and we used TF-IDF to determine words that were likely to be in papers written by each author, but were not commonly found in papers written by the other two authors.

Today, we are going to use text features derived from the LIWC. This means we have 118 possible text based features to consider. Our goal for today is to use these features to (1) predict which author wrote each paper and (2) to identify and discuss key traits in the text that identify the writing of each author.

Feature Selection

If you look at the Federalist dataset you have loaded, the dataset should have 85 rows and 121 columns. The first column is an id for the paper, the second column is the actual essay text, and the third is the author. The remaining 118 columns are features about the text created from the LIWC.

As a reminder, the LIWC is a tool that allows us access to multiple lexicons that we can use to create features about text data. For instance, how much of the paper is written in past tense? How much involves discussions of economics? How long are the sentences, and what is the tone like? The list of possible features is extensive!

If you are curious about what each feature represents, you can look at the list on page 12 of this document. This details each of the features created by the LIWC and gives examples of the types of words in each of the lexicons it uses to create these features.

Recall that our goal for today is to (1) predict which author wrote each Federalist paper and (2) to identify and discuss key traits in the text that identify the writing of each author. It is this second goal which is going to motivate us to consider feature selection. Trying to describe how 118 features relate to the writing of each author is very difficult. Ideally, we would like to narrow this down to just the key features that embody the work of each author. This is a motivator for feature selection.

Feature selection just means that we look at all the features we have available to use (right now 118) and we decide which of them we want to use in our models. As we discussed in class, there are a few different ways we can do this, including forward selection, backwards elimination, and BSS. We are going to start with forward selection.

levels(Federalist[ ,"author"])

modelFeature <- multinom(author ~ Feature, data = Federalist, trace = FALSE)

forward_selection_multinomial <- function(features, response){

  # Load the library
  suppressMessages(library(nnet))
  # Initialize a
  a = 1 
  
  check <- apply(features, 2, function(x) length(unique(x)))
  to.remove <- which(check == 1)
  features <- features[,-to.remove]
  
  # Count the number of features
  p <- ncol(features)
  
  # Find the best feature with only one feature in the model

  for( i in 1:p){
    modelholder <- multinom(response ~ ., data = data.frame(features[,i]), trace = FALSE)
    if(a == 1){
      modelcurrent = i
      AICcurrent = AIC(modelholder)
      a = a+1
      }
    if(AIC(modelholder) < AICcurrent){
      AICcurrent <- AIC(modelholder)
      modelcurrent <- i
    }
    rm(modelholder)
  }


  model1 <- multinom(response ~ ., data = data.frame(features[,modelcurrent]), trace = FALSE)
  AIC_tobeat <- AIC(model1)
  tokeep = modelcurrent
  
    print(1)
  print(AIC(model1))

  for( x in 1:(p-1)){
    a = 1 
    for( i in c(1:p)[-tokeep]){
      modelholder <- multinom( response ~ ., data = features[,c(i,tokeep)], trace = FALSE)
      if(a == 1){
        modelcurrent = i
        AICcurrent = AIC(modelholder)
        a = a+2
      }
      if(AIC(modelholder) < AICcurrent){
        AICcurrent <- AIC(modelholder)
        modelcurrent <- i
        }
      rm(modelholder)
    } 
    if(a >1 & AIC_tobeat- AICcurrent < 1){
      break
    }
    tokeep <- c(tokeep,modelcurrent)
    print(x+1)
    print(AICcurrent)
    AIC_tobeat = AICcurrent
    }

  print(colnames(features)[tokeep])

}

# Run forward selection
features_chosen <- forward_selection_multinomial(features,response)

forward_model <- multinom( Federalist$author ~ ., data = Federalist[, features_chosen], trace = FALSE)

knitr::kable(t(coefficients(forward_model)))

test <- read.csv("https://www.dropbox.com/scl/fi/45x2atyykounkdc811qmv/LIWC_Test.csv?rlkey=rcuxfme51l032cj1s16x3to4x&st=lvnp4z8i&dl=1")

predict(forward_model, type = "class", newdata = test)

predict(forward_model, type = "probs", newdata = test)

Classification Trees

Let’s try a different way we can explore the relationships between the features and the response by building a classification tree. Classification trees use the values of the features to create clusters. Inside of these clusters, we want to have rows that have similar values of \(Y\). In other words, we would like to define one or more Madison clusters, Hamilton clusters, and so on.

Classification trees have a cool property that they perform feature selection as a natural part of the tree building process. All features are considered, but only features that improve the Gini Index the most are actually used.

To build a classification tree in R, we can use:

tree_class <- rpart( author ~ . , data = Federalist[,-c(1:2)], method = "class")

To visualize the tree, we use

fancyRpartPlot(tree_class, sub = "" )

As we can see, classification trees have a few nice properties! Even though we started with 118 features, only 3 of them are actually used in our model. This makes is very easy to visualize and interpret the relationships in the data. We can also use trees to make predictions! And speaking of predictions…

Naive Bayes

The third and final approach we are going to use is Naive Bayes. Recall that this is a prediction approach that uses conditional probabilities.

To run Naive Bayes in R, we use the following:

# Load the library
library(naivebayes)

# Run Naive Bayes 
naive_result <- naive_bayes(author~., data= Federalist[,-c(1,2)])

# Make predictions on the test data 
predict( naive_result, newdata = test)

At this point, we have three different sets of predictions: one from multinomial regression, one from classification trees, and one from Naive Bayes. We could consider choosing just one approach to use…or we could do what is called an ensemble learning method. This just means that we combine the results from multiple predictive approaches to obtain our final prediction.

For each row in the test data, we look at the predictions from each of the 3 methods. The final predicted value is the value that is most popular across the three methods!

To see our 3 predictions side by side, we can use the following:

Predictions <- data.frame("Multinomial" = predict( forward_model, type = "class", newdata = test))

Predictions$Tree <- predict( tree_class, type = "class", newdata = test)

Predictions$NaiveBayes <- predict(naive_result,  type = "class", newdata = test)

knitr::kable(Predictions)