Ch. 1 - Regression models: fitting them and evaluating their performance

Welcome to the Toolbox

In-sample RMSE for linear regression

In-sample RMSE for linear regression on diamonds

Out-of-sample error measures

Out-of-sample RMSE for linear regression

Randomly order the data frame

Try an 80/20 split

Predict on test set

Calculate test set RMSE by hand

Comparing out-of-sample RMSE to in-sample RMSE

Cross-validation

Advantage of cross-validation

10-fold cross-validation

5-fold cross-validation

5 x 5-fold cross-validation

Making predictions on new data

Ch. 2 - Classification models: fitting them and evaluating their performance

Logistic regression on sonar

Why a train/test split?

Try a 60/40 split

Fit a logistic regression model

Confusion matrix

Confusion matrix takeaways

Calculate a confusion matrix

Calculating accuracy

Calculating true positive rate

Calculating true negative rate

Class probabilities and predictions

Probabilities and classes

Try another threshold

From probabilites to confusion matrix

Introducing the ROC curve

What’s the value of a ROC curve?

Plot an ROC curve

Area under the curve (AUC)

Model, ROC, and AUC

Customizing trainControl

Using custom trainControl

Ch. 3 - Tuning model parameters to improve performance

Random forests and wine

Random forests vs. linear models

Fit a random forest

Explore a wider model space

Advantage of a longer tune length

Try a longer tune length

Custom tuning grids

Advantages of a custom tuning grid

Fit a random forest with custom tuning

Introducing glmnet

Advantage of glmnet

Make a custom trainControl

Fit glmnet with custom trainControl

glmnet with custom tuning grid

Why a custom tuning grid?

glmnet with custom trainControl and tuning

Interpreting glmnet plots

Ch. 4 - Preprocessing your data

Median imputation

Median imputation vs. omitting rows

Apply median imputation

KNN imputation

Comparing KNN imputation to median imputation

Use KNN imputation

Compare KNN and median imputation

Multiple preprocessing methods

Order of operations

Combining preprocessing methods

Handling low-information predictors

Why remove near zero variance predictors?

Remove near zero variance predictors

preProcess() and nearZeroVar()

Fit model on reduced blood-brain data

Principle components analysis (PCA)

Using PCA as an alternative to nearZeroVar()

Ch. 5 - Selecting models: a case study in churn prediction

Reusing a trainControl

Why reuse a trainControl?

Make custom train/test indices

Reintroducing glmnet

glmnet as a baseline model

Fit the baseline model

Reintroducing random forest

Random forest drawback

Random forest with custom trainControl

Comparing models

Matching train/test indices

Create a resamples object

More on resamples

Create a box-and-whisker plot

Create a scatterplot

Ensembling models

Summary

About Michael Mallari

Michael is a hybrid thinker and doer—a byproduct of being a StrengthsFinder “Learner” over time. With 20+ years of engineering, design, and product experience, he helps organizations identify market needs, mobilize internal and external resources, and deliver delightful digital customer experiences that align with business goals. He has been entrusted with problem-solving for brands—ranging from Fortune 500 companies to early-stage startups to not-for-profit organizations.

Michael earned his BS in Computer Science from New York Institute of Technology and his MBA from the University of Maryland, College Park. He is also a candidate to receive his MS in Applied Analytics from Columbia University.

LinkedIn | Twitter | www.michaelmallari.com/data | www.columbia.edu/~mm5470