Should we impute missing data? (Python)

Python Machine Learning Series

Motivations

The real-world data often has a lot of missing values. The cause of missing values can be data corruption or failure to record data. Handling missing data is important as many machine learning algorithms do not support data with missing values. However, in the case of XGBoost we may not need to impute missing data before training XGboost.

Findings

Experimental results show that XGboost with original (with missing) dataset has an AUC 3% higher than that of obtained from using imputed dataset.

Python Codes

Python codes for above findings with hmeq.csv dataset:

# ==============================================
#       Load data and impute missing data
# ==============================================

# Load data:
import pandas as pd

df = pd.read_csv("http://www.creditriskanalytics.net/uploads/1/9/5/1/19511601/hmeq.csv")

# Convert categories to dummies:
df = pd.get_dummies(df)

# Impute missing data (https://academic.oup.com/bioinformatics/article/28/1/112/219101,
#                      https://academic.oup.com/aje/article/179/6/764/107562,
#                      https://github.com/epsilon-machine/missingpy):

from missingpy import MissForest

imputer = MissForest()
df_imputed = imputer.fit_transform(df)

# Convert to data frame:
df_imputed = pd.DataFrame(df_imputed)

# Rename for columns:
df_imputed.columns = df.columns

# Prepare data:
from sklearn.model_selection import train_test_split

X = df_imputed.drop(labels=["BAD"], axis=1)
Y = df_imputed["BAD"]

# ======================================
#  Train XGboost with imputed dataset
# ======================================

# Train XGBClassifier with cross-validation:
from xgboost import XGBClassifier
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=29)
xgb1 = XGBClassifier(random_state=29)
auc_scores1 = cross_val_score(xgb1, X, Y, cv=cv, scoring="roc_auc", n_jobs=-1)

# =====================================
# Train XGboost with missing dataset
# =====================================

X_o = df.drop(labels=["BAD"], axis=1)
auc_scores2 = cross_val_score(xgb1, X_o, Y, cv=cv, scoring="roc_auc", n_jobs=-1)

# Compare results:
print(auc_scores1.mean(), auc_scores2.mean())