STA 363 Lab 6

Goal

Our goal today is to start exploring decision trees in R. There are two different kinds of decision trees: regression trees and classification trees. Regression trees are used when \(Y\) is numeric and classification trees are used when \(Y\) is categorical. We will do regression trees today and we will start classification trees next class.

The Data

We are going to work with our familiar penguins data set with information on n = 333 penguins. To load the data, you need to use the following three lines of code:

library(palmerpenguins)
data(penguins)
penguins <- na.omit(penguins)
penguins <-data.frame(penguins)

We have a client who is interested in building a model for Y = the body mass of a penguin in grams. In addition to this response variable, we have information on 7 features.

The client is particularly interested in understanding the relationship between different values of the features and higher body mass.

We are going to use a decision tree model to approach this task today.

Regression Trees

Regression trees are supervised models (like all the models we do in this course) with a response variable Y that is numeric. We will use classification trees if Y is categorical, but we won’t get to those until next week.

When we build regression trees, the first step is data cleaning, just like it is for every other model. We still need to be on the look out for outliers and features that are impossible to use, and other traits of the data that we need to be aware of. The EDA step, however, is a little less intensive with regression trees than with a linear model. Regression trees don’t have the assumption that the relationship between Y and each X be represented by a line, or a curve, etc., so we don’t need to check for shape the way we are used to for LSLR models. The tree model is not built that way.

With that in mind, let’s start!

Building a Regression Tree with a Categorical Feature

The first tree we are going to build will use only one feature: species. Why only one feature? Well, because it is much easier to explore how the tree grows if we limit ourselves to only one feature to start with. We will add more features later.

All trees begin with all of the rows of the data in one giant cluster, called the root.

Why do we care about this? Well, knowing the information about the root gives us an idea of the starting point of our tree. We build a tree by minimizing the RSS, so it helps to know what the RSS in the root node is so we can track the improvement as the tree grows.

Now that we have explored the root node, it is time to start growing the tree! Remember, we only have one feature: species.

Now that we know all the possible splitting rules we can choose, how do we decide which splitting rule to use to grow our tree? We choose the splitting rule that gives us the smallest RSS after the split. This means that in order to decide which splitting rule from Question 4 to choose, we need to actually compute the RSS we would get if we used each splitting rule.

Building a Tree in R

Now that we have determined ourselves which splitting rule we should use, let’s verify it using R. The package that we need to do that is called rpart(). To use this function, we need three libraries.

# Grows the tree
library(rpart)
# Allows us to visualize the tree
library(rattle)
library(rpart.plot)

To build a tree with one feature (species) and only one split, we use the following code:

tree1<- rpart(body_mass_g ~ species, data = penguins,
        method = "anova", maxdepth = 1)

You will notice that the structure of this code is similar to other regression codes. We specific the Y variable, the feature(s), and the data. We use method = "anova" to tell R that we are fitting a regression tree rather than a classification tree.

The maxdepth=1 part of the code specifies that right now, we are only allowing one split in the tree. If you don’t put anything in the maxdepth part of a tree code, the tree grows until it hits the built in stopping rules. More on that later!

Once your tree is built (meaning you have run the line of code above), it helps to visualize the results. One of the biggest advantages of trees is that they are highly interpretable, meaning it is relatively direct to use trees to explain relationships in the data.

To visualize our tree, we use one of the following:

# Option 1 
fancyRpartPlot(tree1, sub = "Figure 1: One Split")

# Option 2 
prp(tree1)

The sub= part of the code is where you add a title to your tree. It is much easier in a tree to add a title at the bottom rather than at the top - there is more space.

Look at your tree and verify that your answer to Question 6 matches the tree you have drawn!

Now that we have drawn the tree, let’s use it.

Now, with only this one binary feature, it turns out that the tree can be compared to a least-squares linear regression model:

\[BodyMass_i = \beta_0 + \beta_1 GentooPenguin_i + \epsilon_i, \epsilon_i \sim N(0,\sigma)\]

Let’s see why.

What do we notice? Well, with a single categorical feature, trees yield one predicted values for each level of the feature. LSLR models do exactly the same thing. That prediction, for both models, is the mean of the response variable for all the rows at each level of the feature. Translation: The models may look different, but they are in fact the same at this stage.

Okay, then what is the point of a tree?? When we start adding in multiple features, or when the features are numeric, we will find that trees and LSLR models are not the same.

Regression Trees: Numeric Feature

Now that we have explored using a categorical feature, let’s switch to using a numeric feature to predict \(Y\).

Before we use the built in code, let’s make sure we can verify ourselves what splitting rule we should use.

One we know that this is the splitting rule we need, we can verify using the built in R commands.

Now, we saw in a previous example that using a tree with a single split and a single categorical feature was the same as fitting an LSLR model with a single categorical feature. Let’s see if the same thing holds now that we have a numeric feature.

This time, you will see that our values are not identical. We are still using one split - why the change? This is because LSLR assumes the relationship between body mass (Y) and flipper length (X) is linear, so the prediction is based on the slope of a line. The tree models assumes that body mass is different depending on whether X is greater than a particular value of X. There are exactly two possible predictions at this point, dependent on flipper length. This is very different from an LSLR line, which right now can make multiple different predictions depending on the exact value of the flipper length.

So…are trees always better? Or is LSLR always better? Neither. Remember, there is almost never an “always” situation in statistical learning. As we have been doing all semester, we have to think through what our goal of modeling is, what properties our data have, what properties we want our model to have, and then compare a few plausible options!

Adding More Splits

So far, we have limited ourselves to trees with one split. Now, let’s let the tree grow a bit.

Fitting the Tree with multiple features

Now we have explored two trees, but both included only one feature. One of the great things about trees is that they can very easily handle a lot of features in the data. Let’s now grow a tree using them all!

We seem to have specified no stopping rules…but we did. R has default stopping rules built in that are designed to protect your computer. Let’s take a look at these.

Now, trees sometimes have a habit of getting too big to be clear to see in the visuals. There are some great tips at this website, starting on Page 9 that can help make our plot more readable.

Conclusion