(a)

Fit a random forest model to all of the predictors, then estimate the variable importance scores:

library(randomForest)
library(caret)
model1 <- randomForest(y ~ ., data = simulated,
                       importance = TRUE,
                       ntree = 1000)
rfImp1 <- varImp(model1, scale = FALSE)

Did the random forest model significantly use the uninformative predictors ( V6 – V10 )?

Answer: Yes, see below.

First, let’s take a look at the correlations graph.

Plot: Simulated correlations.

Interesting to note the existence of some correlated variables but not considered significant.

The variable importance scores were computed even though they are time consuming; importance = TRUE generated the following values.

Let’s have a visualization of the %IncMSE importance.

Plot: randomForest fit %IncMSE importance.

Let’s have a visualization of the %IncNodePurity importance.

Plot: randomForest fit %IncNodePurity importance.

The reason why the random Forest used the uninformative predictors ( V6 – V10 ) was to keep reducing the MSE as noted on the above table and graphs.

(b)

Now add an additional predictor that is highly correlated with one of the informative predictors. For example:

simulated$duplicate1 <- simulated$V1 + rnorm(200) * .1
cor(simulated$duplicate1, simulated$V1)

Plot: Highly correlated predictor added.

The correlated value in between duplicate1 and V1 is 0.9497025.

Fit another random forest model to these data.

model2 <- randomForest(y ~ ., data = simulated, 
                       importance = TRUE, 
                       ntree = 1000)
rfImp2 <- varImp(model2, scale = FALSE)

The variable importance scores were computed even though they are time consuming; importance = TRUE generated the following values.

As noted on the above table, it is important to note that the predictors importance changed by having a highly correlated predictor to an important predictor.

Let’s have a visualization of the %IncMSE importance.

Plot: randomForest fit %IncMSE importance.

Let’s have a visualization of the %IncNodePurity importance.

Plot: randomForest fit %IncNodePurity importance.

Did the importance score for V1 change? Yes.

What happens when you add another predictor that is also highly correlated with V1 ?

Uninformative predictors with high correlations to informative predictors have abnormally large importance values. In some cases, their importance can be greater than or equal to weakly important variables. Another impact in between-predictor correlations is to dilute the importance of key predictors.

Theory tells us that if we have a critical predictor of importance of X. If another predictor is just as critical but is almost perfectly correlated as the first, the importance of these two predictors will be roughly X/2.

This can be seeing since in the highly correlated value, the V1 new value shows to have %IncMSE = 6.0070978 and %IncNodePurity = 741.9267 which is almost half from the first observed table in which the predictors were not correlated.

(c)

Use the cforest function in the party package to fit a random forest model using conditional inference trees. The party package function varimp can calculate predictor importance. The conditional argument of that function toggles between the traditional importance measure and the modified version described in Strobl et al. (2007).

library(party)
# Calculating original simulation
model3_O <- cforest(y ~ ., data = simulated_Original,
                  controls = cforest_unbiased(ntree = 1000))

cfImp3_O <- varimp(model3_O, conditional = TRUE)

# Calculating Simulated with Correlated values
model3_S <- cforest(y ~ ., data = simulated,
                  controls = cforest_unbiased(ntree = 1000))

cfImp3_S <- varimp(model3_S, conditional = TRUE)

Let’s take a look at the comparison:

From the above table, we can also notice how the values change considerable when a highly correlated value is added to the model.

Plot: Original cforest importance fit.

Plot: Simulated cforest importance fit.

Do these importances show the same pattern as the traditional random forest model? Yes, it does show similar patterns, only part of it.

(d)

Repeat this process with different tree models, such as boosted trees and Cubist. Does the same pattern occur?

Cubist

library(Cubist)

# Calculating Original Simulation
model4_O <- cubist(x = simulated_Original[,-11], 
                 y = simulated_Original[,11],
                 committees = 100)

cubistImp4_O <- varImp(model4_O)

# Calculating Simulated with Correlated values
model4_S <- cubist(x = simulated[,-12], 
                 y = simulated[,12],
                 committees = 100)

cubistImp4_S <- varImp(model4_S)

Let’s take a look at the below table to identify some possible changes:

From the above table, we notice how the values for the selected variables and order has changed in terms of importance.

Boosted trees

library(gbm)
model5_O <- gbm(y ~ ., data = simulated_Original, distribution = "gaussian")
model5_S <- gbm(y ~ ., data = simulated, distribution = "gaussian")

Let’s take a look and compare results:

Let’s take a look at the below table to identify some possible changes:

Banging

library(ipred)

# Calculating Original Simulation
model6_O <- bagging(y ~ ., data = simulated_Original)

banggingImp6_O <- varImp(model6_O)

# Calculating Simulated with Correlated values
model6_S <- bagging(y ~ ., data = simulated)

banggingImp6_S <- varImp(model6_S)

Let’s take a look at the below table to identify some possible changes:

Now, responding to the original question: Does the same pattern occur? we could answer that the same pattern (meaning the order of importance change) occur, not necessarily in the same order or with the same predictors.

Solution

From the text book:

Trees suffer from selection bias: predictors with a higher number of distinct values are favored over more granular predictors (Loh and Shih 1997; Carolin et al. 2007; Loh 2010). Loh and Shih (1997) remarked that

“The danger occurs when a data set consists of a mix of informative and noise variables, and the noise variables have many more splits than the informative variables. Then there is a high probability that the noise variables will be chosen to split the top nodes of the tree. Pruning will produce either a tree with misleading structure or no tree at all.”

Also, as the number of missing values increases, the selection of predictors becomes more biased (Carolin et al. 2007). It is worth noting that the variable importance scores for the solubility regression tree (Fig. 8.6) show that the model tends to rely more on continuous (i.e., less granular) predictors than the binary fingerprints. This could be due to the selection bias or the content of the variables.

Let’s create a simulated experiment in this case as follows:

Let’s create a data set as follows:

• Home Value.
• State (only 2 categories).
• Square feet (random values taken as 200 possible outcomes, not related to the home Price).

# Defining granular data
set.seed(123)
State <-  rep(0:1,each=100)
# Defining relationsships
Price <- State + rnorm(200,mean=0,sd=4)
# Defining non-granular data
set.seed(231)
SFeet <- rnorm(200,mean=0,sd=2)

SData <- data.frame(Price,State,SFeet)

Example of relationship in data:

Boxplot: Simulated home values by State.

Boxplot: Simulated home values by State.

Let’s create our simulation to run 25 times using the rpart function with 1 level.

library(rpart)
# Need to Store variables with highest importance
HImportance <- c()

for (i in 1:25){

    #print(i)
    # Defining granular data
    State <-  rep(1:2,each=100)
    # Defining relationsships
    set.seed(123 + i)
    Price <- State + rnorm(200,mean=0,sd=4)
    # Defining non-granular data
    set.seed(200 + i)
    SFeet <- rnorm(200,mean=0,sd=2)
    
    # Defining simulated Data Frame
    SData <- data.frame(Price,State,SFeet)
    
    # Finding New Model
    Smodel <- rpart(Price ~ ., data = SData, control = rpart.control(maxdepth=1))
    
    # Finding importance
    Imp <- varImp(Smodel)
    
    # Find Highest Importance
    HImportance[i] <- rownames(Imp)[which.max(Imp$Overall)]
} 

Let’s see our test results from the simulation:

## HImportance
## SFeet State 
##    18     7