Regression Trees

Regression Trees and Model Trees

Understanding regression trees and model trees

Example: Calculating SDR

# set up the data
tee <- c(1, 1, 1, 2, 2, 3, 4, 5, 5, 6, 6, 7, 7, 7, 7)
at1 <- c(1, 1, 1, 2, 2, 3, 4, 5, 5)
at2 <- c(6, 6, 7, 7, 7, 7)
bt1 <- c(1, 1, 1, 2, 2, 3, 4)
bt2 <- c(5, 5, 6, 6, 7, 7, 7, 7)

summary(tee)
str(tee)

#We test

# compute the SDR
sdr_a <- sd(tee) - (length(at1) / length(tee) * sd(at1) + length(at2) / length(tee) * sd(at2))
sdr_b <- sd(tee) - (length(bt1) / length(tee) * sd(bt1) + length(bt2) / length(tee) * sd(bt2))

#these values provide some indication of how well the main signal (tee) stands out from the #variations present in its sub-sets (at1, at2, bt1, bt2). Higher values of sdr_a and sdr_b #would suggest stronger signals relative to the sub-sets' noise.

# compare the SDR for each split
sdr_a
sdr_b

#sdr_a is = to 1.202815
#sdr_b is =  to 1.392751

#This indicates that sdr_b represents a stronger signal relative to its sub-sets compared to sdr_a.

# If sdr_a or sdr_b has a larger value, it means that even after removing the sub-sets' variability, the remaining standard deviation (representing the variability of the main signal) is still high. This implies that the main signal is strong and distinct from the variations in the sub-sets.

Exercise No 3: Estimating Wine Quality

Step 2: Exploring and preparing the data

wine <- read.csv("whitewines.csv")

# examine the wine data
str(wine)

# the distribution of quality ratings
hist(wine$quality)

# summary statistics of the wine data
summary(wine)

wine_train <- wine[1:3750, ]
wine_test <- wine[3751:4898, ]

#used 3750 observations to train to model and 1148 to test it

Step 3: Training a model on the data

# regression tree using rpart
library(rpart)
m.rpart <- rpart(quality ~ ., data = wine_train)

# get basic information about the tree
m.rpart

# get more detailed information about the tree
summary(m.rpart)

Variable Importance:

The most important variable for predicting wine quality is alcohol, followed by density, volatile acidity, and chlorides.

Tree Structure:

The tree starts by splitting the data based on alcohol content. Wines with alcohol < 10.85 are more likely to have lower quality (mean quality around 5.6). Wines with alcohol >= 10.85 are more likely to have higher quality (mean quality around 6.3). Further splits are made based on other variables like volatile acidity, free sulfur dioxide, and chlorides. Each terminal node represents a group of wines with similar characteristics and predicted quality.

#install.packages("rpart.plot")

# use the rpart.plot package to create a visualization
library(rpart.plot)

# a basic decision tree diagram
rpart.plot(m.rpart, digits = 3)

# a few adjustments to the diagram
rpart.plot(m.rpart, digits = 4, fallen.leaves = TRUE, type = 3, extra = 101)

Step 4: Evaluate model performanc

# generate predictions for the testing dataset
p.rpart <- predict(m.rpart, wine_test)

# compare the distribution of predicted values vs. actual values
summary(p.rpart)
summary(wine_test$quality)

#Both distributions have similar medians and 3rd quartiles, suggesting predictions tend to be close to actual values for a significant portion of the data.
#The actual values range wider (3.000 to 9.000) compared to predicted values (4.545 to 6.597), indicating the model might not capture the full range of quality.

# compare the correlation
cor(p.rpart, wine_test$quality)

# function to calculate the mean absolute error
MAE <- function(actual, predicted) {
  mean(abs(actual - predicted))  
}

# mean absolute error between predicted and actual values
MAE(p.rpart, wine_test$quality)

#The resulting MAE value represents the average absolute error made by your model on the test set. It reflects the typical magnitude of difference between predictions and actual quality on unseen data.
#on average every prediction is 0.5872 units of quality off the actual value.

# mean absolute error between actual values and mean value
mean(wine_train$quality) # result = 5.87
MAE(5.87, wine_test$quality)  #result = 0.6722474

Step 5: Improving model performance

#install.packages("plyr")
#install.packages("Cubist")

# train a Cubist Model Tree
library(Cubist)
m.cubist <- cubist(x = wine_train[-12], y = wine_train$quality)

# display basic information about the model tree
m.cubist

# display the tree itself
summary(m.cubist)

# generate predictions for the model
p.cubist <- predict(m.cubist, wine_test)

# summary statistics about the predictions
summary(p.cubist)

# correlation between the predicted and true values
cor(p.cubist, wine_test$quality)

#The magnitude of 0.62 suggests a moderately strong linear relationship.

# mean absolute error of predicted and true values
# (uses a custom function defined above)
MAE(wine_test$quality, p.cubist)