Goal

In today’s lab, we are going to explore forests. We are going to work with the same data set as the last lab. As a reminder, this data set contains information on n = 333 penguins. To load the data, you need to use the following three lines of code:

library(palmerpenguins)
data(penguins)
penguins <- na.omit(penguins)

We have information on:

Regression Forest

There are two different classes of forests. We use regression forests when our response variable is numeric and categorical forests when our response variable is categorical.

We will start with a regression forest predicting \(Y\) = body mass.

Before we use the default R codes to help us build our forest, let’s make sure we take some time to understand what is going on under the hood when that code runs.

# Load the library 
library(randomForest)

# Set a seed 
set.seed(363663)

# Grow a forest
forest1 <- randomForest(y ~ ., data = , 
                       ntree =  ,
                       importance = TRUE)

Interpreting a Forest

Okay, so we can now build a forest. What about interpretability? Our forest is composed of \(B=1000\) trees. How in the world can we decide how the features are related to \(Y\) and what features matter in our 1000 tree forest???

With forests, we use something called feature importance as a tool to describe how much different features are contributing to the fit of the forest.

Let’s consider the feature flipper length. Suppose flipper length is not at all important to the forest, meaning this feature were not related to body mass. Under this assumption, we could completely change the ordering of the flipper length variable in the data set, and the forest should not be impacted.

In other words, I should be able to put the flipper length of the 3rd penguin in row 5, and the flipper length of the 5th penguin in row 1, and so on, and the forest should not be impacted.

This suggests that one way to see how important a feature is to a forest is to do just that. We fit the forest with the original version of the feature, and then we fit a second forest where we have permuted (scrambled up) the ordering of the feature. We can then compare the fit of the forests to see if changing the ordering made a difference! If a feature is important in our forest, our predictive metrics (test MSE for a regression forest or test CER for a classification forest) should become worse once we permute the feature.

To permute the ordering of flipper length, we can use:

# Create a copy of the data set 
penguinsPermute <- penguins

# Permute the feature
penguinsPermute$flipper_length_mm<- penguins$flipper_length_mm[sample(1:333,333)]

We then grow a tree on this new data set penguinsPermute and compute the percent increase in the MSE. This is the importance of the feature.

We did this using just one tree to see the idea of permuting a feature, but we don’t typically use importance with a single tree. Instead, we use it for the whole forest.

To check the importance across an entire forest, use the following code:

# Print out the importance
knitr::kable( randomForest::importance(forest1)[,1], col.names = "% Inc in MSE" )

The value that you see in the column represents the percent increase in the estimated test MSE when we permuted each feature.

We can also graph the importance, which is particularly helpful when you have a lot of features.

# Load the library to make the graph
suppressMessages(library(lattice))

# Plot the importance
barchart(sort(randomForest::importance(forest1)[,1]),
xlab = "Percent Increase in OOB MSE",
main = "Figure 1: Importance")

barchart(sort(randomForest::importance(forest1)[,1]),
         xlab = "Percent Increase in OOB MSE",
         main = "Figure 1: Importance",
         col = "coloryouwant")

If we only wanted to show the top 3 features in terms of importance, we can change the graph to:

barchart(sort(randomForest::importance(forest1)[,1])[-c(1:4)],
         xlab = "Percent Increase in OOB MSE",
         main = "Figure 1: Importance")

How did I know to remove 1:4? Well, there are 7 features in total. If I only want the top 3, I need to remove the lowest 4.

Partial Plots

Another option for interpreting how features in a forest are related to a response involves what are called partial plots.

These plots can allow us to see the impact of each variable on the predictions made by our forest. It can show us the shape of the relationship, as well as the direction.

For a regression forest, the partial plot shows us that if we hold all other features fixed, but we change the value of a certain feature, how does this change \(Y\)?

For a classification forest, the partial plot shows us that if we hold all other features fixed, but we change the value of a certain feature, how does this change the log odds that \(Y= 1\)?

penguins <- data.frame(penguins)

partialPlot( forest1, penguins,
             x.var = "flipper_length_mm",
             xlab = "Flipper Length (in mm)",
             ylab = "Body Mass (in grams)",
             main = "Partial Plot: Flipper Length")

partialPlot( forest1, penguins,
             x.var = "species",
             xlab = "Penguin Species",
             ylab = "Body Mass (in grams)",
             main = "Partial Plot: Species")

Turning in your assignment

When your Markdown document is complete, do a final knit and make sure everything compiles. You will then submit your document on Canvas. You must submit a PDF or html document to Canvas. No other formats will be accepted. Make sure you look through the final document to make sure everything has knit correctly.

References