STA 363 Lab 8

Goal

In today’s lab, we are going to explore forests. In class, we did a regression forest. Today, we will focus on a classification forest. Most of the ideas are the same, but this will give us practice with both!

The Data

We are going to work with the same data set as the last lab. As a reminder, this data set contains information on n = 333 penguins. To load the data, you need to use the following three lines of code:

library(palmerpenguins)
data(penguins)
penguins <- na.omit(penguins)

We have a client who is interested in building a model for Y = the sex of the penguin.

In addition to this response variable, we have information on 7 features.

Classification Tree

Before we start building a forest, let’s build a single tree. This is generally good practice, because it will help us to see what the trees that make up our forest might look like.

Because we are working with a categorical response variable, this means that we are going to be using classification metrics to evaluate our tree.

Now that we have assessed the predictive performance of a single tree, let’s consider growing a forest.

Building a Forest: One Tree at a time

Before we use the default R codes to help us build our forest, let’s make sure we take some time to understand what is going on under the hood when that code runs.

We know that forests are built by creating bootstrap samples from the original penguins data set. So, let’s start there.

Now we have two trees, grown on different data sets.

Once we can grow one tree, we could grow more than one!

In practice, it is faster to use some R packages to grow a forest. Let’s do that next.

Building a Forest: Using the Package

In practice, we typically use the randomForest package to build our forests.

You will see in the output of the random forest that you also get a confusion matrix on the OOB observations! The rows are the true values and the columns are the predicted values.

Interpreting a Forest

Okay, so we can now build a forest. What about interpretability? Our forest is composed of \(B=1000\) trees. How in the world can we decide which features are important in our 1000 tree forest?

Suppose we are attempting to express how important the feature body mass is to our tree building process. To determine this, we grow our bagged forest, using all predictors we are considering. We compute and store the OOB error rate (the test CER) of this forest. Now, to see how important body mass was to our prediction process, we want to see what happens if we assume that body mass has no relationship with sex. How do we do that? We scramble up the order of the rows in the body mass column in the data before we grow our trees!

What this does is essentially break the relationship between Y and X. If X was important in our model, our predictive metrics (test MSE/RMSE for a regression forest or test CER for a classification forest) should become worse once we do so.

To check the importance of a classification forest, use the following code:

# Load the library to make the graph
suppressMessages(library(lattice))

# Plot the importance
barchart(sort(randomForest::importance(forest1)[,3]),
xlab = "Percent Increase in OOB CER",
main = "Figure: Importance")