Project Introduction

In this project you will be tasked to use machine learning and use the methods we have seen on the course on a realistic data set prepared for this course. We will guide you all along with some elements and provide some code already prepared. We propose a workflow with several guided step where you will fill-in some blanks and interpret the results.

Since fitting the Random Forest model with a large amount of observations and variables can be very computationally expensive, in the first part of the project you will explore the data and fit a LASSO model. Using these results you’ll have the possibility to select the variables to include in the Random Forest model.

We will provide you with some hyperparameters that you can use to prevent the training taking too long, as the main goal of the project is to become comfortable in running the code and interpreting the results.

The general workflow for this analysis follows these main steps:

All along this analysis we will guide you with some chunks of code and help you with the choice of hyperparameters and useful functions.

The data set can be used directly from the SIAP’s server. If you need, you can also download it an use it locally.

# Reading DHS survey data from SIAP's website
df <- read.csv("https://www.unsiap.or.jp/on_line/ML/M6-clean_data.csv")

If you feel like taking the exercise a bit further, you are of course encouraged to try to play around with some of the default parameters we set to see if you can improve the results!

Data exploration

The data set is a record of renting prices for different types of housing options. It contains 48751 observations and 14 variables.

Summary statistics

Since some are categorical, and other are numerical, you can visualize their properties and statistical summaries separately. All variables are self-explanatory and their meaning can be understood quite easily.

Numerical variables

datasummary_skim(df, type = "numeric")
Unique (#) Missing (%) Mean SD Min Median Max
price 649 0 163.7 421.8 0.0 105.0 10000.0
latitude 18964 0 40.7 0.1 40.5 40.7 40.9
longitude 14790 0 -74.0 0.0 -74.2 -74.0 -73.7
accommodates 19 0 2.9 1.9 1.0 2.0 22.0
bedrooms 12 0 1.2 0.8 0.0 1.0 21.0
beds 17 0 1.5 1.1 0.0 1.0 16.0
cleaning_fee 214 0 56.2 66.3 0.0 40.0 1200.0
minimum_nights 110 0 7.9 21.6 1.0 3.0 1250.0
availability_365 366 0 120.7 140.4 0.0 63.0 365.0
review_scores_rating 62 0 94.2 8.0 20.0 96.0 100.0

Categorical variables

datasummary_skim(df, type = "categorical" )
N %
host_is_superhost False 39055 80.1
True 9696 19.9
neighborhood A-Zone 19604 40.2
B-Zone 1149 2.4
M-Zone 21778 44.7
Q-Zone 5859 12.0
S-Zone 361 0.7
property_type Appartment 41746 85.6
House 5702 11.7
PrivateRoom 1303 2.7
cancellation_policy flexible 15540 31.9
moderate 11323 23.2
strict 21888 44.9

Correlations

It is always a good practice to see whether some variables are correlated.

numeric <- select_if(df, is.numeric)
# We compute the correlation matrix of the covariates
corr_coef<-cor(numeric ,use = "p")
#And then plot it with nice options 
ggcorrplot(corr_coef, 
           type = "lower",         # lower triangle of the matrix only
           hc.order = TRUE,        # variable sorted from highest to lowest
           outline.col = "white",  #Color options
           lab = TRUE) + ggtitle("Correlation between numerical variables")

To do: Interpret the results of the data exploration, is there anything that stands out in some of the plots? did you learn anything about the data that gives you an insight into if some variables will be more useful or less practical to include in the model?

Preparing the analysis

With 14 variables and considering the correlation between the numerical variables, it may be interesting to try to reduce the number of regressors using a penalization method such as the LASSO.

Hint

Since the data set is quite large we only use half the observations (p = 0.5 in createDataPartition() controls this) for training the models, this will reduce the time you need to train. This can of course be increased or lowered if you want to experiment on the effect this has on your models.

# Splits data into training and testing sets
set.seed(777)
trainIndex <- createDataPartition(df$price, p = 0.5, list = FALSE, times = 1)

One must also prepare our Machine Learning framework and prepare our train data set separately from the one used for validation

train_data <- df[trainIndex,]
validation_data  <- df[-trainIndex,]

Hint

As we have seen in the course, the LASSO method will be affected by the scale of the variables, so we need to scale the data set before we start. Of course this should apply to both our train and validation data sets.

# Scale the training and test data based on the training data mean and variance.
ScalingValues <- preProcess(train_data, method = c("center", "scale"))
train_data <- predict(ScalingValues, train_data)
validation_data <- predict(ScalingValues, validation_data)

Variable selection

Now we can fit the a regression model with LASSO regularization and interpret the results, this may give us some information about which variables might be useful in the Random Forest model later. We fit the regression model using a using cross-validation to ensure it is giving robust results.

Selecting the right value of lambda for the LASSO

As in many Machine Learning exercise, one must select hyperparameters to adjust the procedure and to ensure the best accuracy and performance. Here we want to select the regression model with a good balance of accuracy and a minimal number of regressors to avoid collinearity and over-fitting.

Hint

The grid search for the lambda parameter in the LASSO is tuned here between 0 and 0.003, in 5 regular increments. We also limited the number of models to 10. You may want to play with these parameters.

# Control variables
numbers <- 5
repeats <- 20
rcvTunes <- 10 # tune number of models
seed <- 123
# repeated cross validation
rcvSeeds <- setSeeds(method = "repeatedcv", 
                     numbers = numbers, repeats = repeats, 
                     tunes = rcvTunes, seed = seed)


# Controls for the CV 
rcvControl <- trainControl(method = "repeatedcv", 
                        number = numbers, repeats = repeats,
                        seeds = rcvSeeds)


set.seed(123)
lasso_fit <- train(price ~ .,
                   data = train_data, 
                   method = "glmnet",
                   tuneGrid = expand.grid(alpha = 1, 
                                          # grid to search
                                          lambda = seq(from =0,
                                                       to=0.003,
                                                       length = 5)),
                   trControl = rcvControl)

It always a good practice to visualize the outcome of the grid search to check whether the value of the hyperparameter (lambda here) affects the outcome and if there exist a value that is optimal.

ggplot(lasso_fit)   +
  ggtitle("Lasso Penalization") +
  labs(x = "Regularization parameter (Lambda)")+
  theme_minimal()

cbind(lasso_fit$bestTune$lambda)  %>% 
  kable(digits=3, col.names = c("lambda (exp)")) %>%
  kable_styling()
lambda (exp)
0.001

It appears that the best value for lambda is here 7.5^{-4}. (This result is automatically updated when you knit your notebook.)

Since our goal is to feed a Random Forest model, we can use the previous results and identify the variables and features selected by the LASSO procedure (when used with the right lambda). This can also be done by visualizing the feature importance of the LASSO regression:

Hint

We can first focus on the most important features (here selecting only features importance greater than 3) and refine the analysis later. Be careful that this may require some time.

theme_models <-  theme_minimal()+ theme(plot.title = element_text(hjust = 0.5),
                legend.position = "none") 

lasso_varImp <- data.frame(variables = row.names(varImp(lasso_fit)$importance), varImp(lasso_fit)$importance)
# Below we set that we only show feature importance with a value larger than 3
# You can lower this if you want to see more variables, or increase it if you want to see fewer.
threshold = 2
lasso_varImp <- lasso_varImp[lasso_varImp$Overall > threshold,]
ggplot(data = lasso_varImp, mapping = aes(x=reorder(variables, Overall),
                                        y=Overall,
                                        fill=variables)) +
  coord_flip() + geom_bar(stat = "identity", position = "dodge") +
  theme_models +
  labs(x = "", y = "") +
  ggtitle("Feature Importance Lasso Regression")

To do: Interpret the results of the data exploration, is there anything that stands out in some of the plots? did you learn anything about the data that gives you an insight into if some variables will be more useful or less practical to include in the model?

We can also look at the models out of sample Root Mean Squared Error (RMSE), and compare this later to the results of the Random Forest:

lasso_preds <- predict(lasso_fit, validation_data)
rmse(actual = validation_data$price, predicted = lasso_preds)
## [1] 0.9991186

Fitting a Random Forest model

We are now ready to give it a try and fit the Random Forest, but for this we need to specify some variables.

Hint

So far we have only run with the formula y ~ ., this means that we give the algorithm access to all variables in the data set to fit the model. Based on the results so far in the project, it would be good to select a formula which you think could give good results (see below).1

To do: Select your variables for the model!

Hint

We have set the number of trees to 10 only, which is way too restrictive. However, this allows to let you try and keep the training relatively fast (it may still take some time though!). Based on the variable importance in the LASSO regression and which variables had the highest correlations with price, a good start is to use the model: \[ price = property\_type + neighborhood + accommodates + bedrooms + cancellation\_policy + cleaning\_fee \]

as this contains what seems to be the most important variables, and it will not be too complex to fit.

Remember that finding a good model can be an iterative process, don’t be discouraged if you don’t immediately find one that performs as well as you hoped.

# Modify this formula (keep the syntax!) to choose your regressors

MyRegressors <- "property_type + neighborhood + accommodates +
bedrooms +cancellation_policy + cleaning_fee"

Now, you can experiment with some other hyperparameters such as the number of trees used in the Random Forest model, as we have seen that this may be crucial in the performances of a Random Forest model.

To do: Change the number of trees (let’s say to 50 or 100), and run more experiments.

# Training this model may take some time!!! 
Ntree <-10  # The number of trees affect greatly the duration of the process

#Formula using the regressors' list defined above 
MyFormula <- as.formula(paste('price ~', MyRegressors))

# Change the formula below: 
rf_fit <- train(MyFormula,
                data = train_data,
                method = "rf",
                ntree = Ntree)
rf_fit
## Random Forest 
## 
## 24376 samples
##     6 predictor
## 
## No pre-processing
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 24376, 24376, 24376, 24376, 24376, 24376, ... 
## Resampling results across tuning parameters:
## 
##   mtry  RMSE       Rsquared   MAE      
##    2    0.9370730  0.1369970  0.2084868
##    6    0.9300933  0.1491148  0.2105905
##   11    0.9393542  0.1419820  0.2162355
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was mtry = 6.
rf_preds <- predict(rf_fit, validation_data)
rmse(actual = validation_data$price, predicted = rf_preds)
## [1] 0.986717

With this model we find some slight improvement over the LASSO in terms of the RMSE.

It is now interesting to see if the feature importance of the two different approaches are similar or not:

theme_models <-  theme_minimal()+ theme(plot.title = element_text(hjust = 0.5),
                legend.position = "none") 

rf_varImp <- data.frame(variables = row.names(varImp(rf_fit)$importance), varImp(rf_fit)$importance)

ggplot(data = rf_varImp, mapping = aes(x=reorder(variables, Overall),
                                        y=Overall,
                                        fill=variables)) +
  coord_flip() + geom_bar(stat = "identity", position = "dodge") +
  theme_models +
  labs(x = "", y = "") +
  ggtitle("Feature Importance Random Forest")

To do: Comment on your models performance, and its feature importance. Was the feature importance what you expected? Did the results of your model match what the results that you saw from the LASSO? Which is the best model?

Hint

You can also use a graphical comparison of the performances of the two models!

# This graphics allows the comparison of results (on the SAME scale)
# even if number of CV samples are different

lasso <- as.data.frame(lasso_fit$resample$RMSE)
rf <- as.data.frame(rf_fit$resample$RMSE)

y.min <- min(lasso, rf)
y.max <- max(lasso, rf)

p.lasso <- ggplot(lasso) +
          aes(x = "", y = `lasso_fit$resample$RMSE`) +
          geom_boxplot(fill = SIAP.color) +
          labs( x = "LASSO", y= "RMSE") +
          coord_cartesian(ylim = c(y.min, y.max))+
          theme_minimal()

rf <- as.data.frame(rf_fit$resample$RMSE)
p.rf <- ggplot(rf) +
          aes(x = "", y = `rf_fit$resample$RMSE`) +
          geom_boxplot(fill = orange.color) +
          coord_cartesian(ylim = c(y.min, y.max))+
          labs( x = "Random Forest", y ="")+
          theme_minimal()


grid.newpage()
grid.draw(cbind(ggplotGrob(p.lasso), ggplotGrob(p.rf), size = "last"))

Corresponding functions if you use Python for this project

  1. If you want to run with a specific formula for the model you can provide it in the following way: price ~ bedrooms + cleaning_fee + more variables…, etc.↩︎