Financial fraud detection & predictive modeling with machine learning

Author

Charles Wainright

Published

July 28, 2026

Code

library(reticulate)
library(tidyverse)
library(gt)
library(DataExplorer)
set.seed(1)
options(scipen=999)
fig_counter <- 0
tbl_counter <- 0

fmt <- function(x) format(x, big.mark = ",", scientific = FALSE)

Code

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import random
import seaborn as sns
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.preprocessing import StandardScaler
from imblearn.over_sampling import SMOTE
from xgboost import XGBClassifier, Booster, DMatrix
from sklearn import metrics
from datetime import datetime
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from sklearn.ensemble import RandomForestClassifier
import plotly.express as px

Abstract

Using historical transaction data, I built a predictive machine learning system to stop credit card fraud while preventing unnecessary blockages on legitimate purchases. By deploying XGBoost with cost-sensitive weighting, the model successfully captures nearly 80% of fraudulent transactions while reducing false alarms by 99% compared to traditional linear baselines—delivering maximum fraud recovery with minimal customer friction.

Introduction

This project combines several data science programming languages, principles, and tools. I used both the R and Python programming languages throughout this analysis. I used R (specifically tidyverse and patchwork) to process, summarize, and visualize data and model performance metrics. I used Python (specifically scikit-learn, imbalanced-learn, and xgboost) to clean features, train the classification models, and extract their evaluation metrics. Finally, I used Quarto to write this document because it seamlessly integrates multilingual code execution, dynamic outputs, and LaTeX into a single, reproducible report.

Methods

I created three machine learning models—Logistic Regression, Random Forest, and XGBoost—to predict whether credit card transactions were fraudulent or legitimate. First, I trained the models on a subset of historical transactions using feature scaling and cost-sensitive class weighting to account for severe class imbalance. Next, I evaluated model performance using Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves to determine how accurately each model identified true fraud without generating excessive false alarms. Finally, I selected the most effective model architecture and compared its performance against random-chance and class prevalence baselines.

Research Question

Which machine learning model—Logistic Regression, Random Forest, or XGBoost—is most effective at accurately identifying fraudulent credit card transactions while minimizing false positive alerts?

Inputs

A tabular dataset of historical credit card transactions containing continuous time/amount features, 28 anonymized PCA components, and binary fraud labels.

Outputs

Binary classification predictions (legitimate vs. fraudulent) and predicted fraud probability scores for each credit card transaction.

Data

Code

fname <- 'data/creditcard.csv'
df <- read.csv(fname)

Code

untransformed_columns = [x for x in r.df.columns if "V" not in x]
transformed_columns = [x for x in r.df.columns if "V" in x]
n_transformed = len(transformed_columns)
n_untransformed = len(untransformed_columns)

# basic attributes of a dataframe
# df.shape
# df.head(5)
# df.isnull().sum()
# df.info()
# df.describe()

I used the Credit Card Fraud Detection dataset from Kaggle for this project, which contains \(284,807\) credit card transactions made by European cardholders in September 2013. The dataset includes 31 features—3 anonymized Principal Component Analysis (PCA) variables (\(V1\)–\(V28\)), alongside untransformed Time, Amount, and the binary target class (Class).

Code

# a glance at the first and last few rows of the dataset
# with an emphasis on the un-transformed columns
n_rows = 5
n_cols = 3
r.df[untransformed_columns + transformed_columns[:n_cols]].head(n_rows)

   Time  Amount  Class        V1        V2        V3
0   0.0  149.62      0 -1.359807 -0.072781  2.536347
1   0.0    2.69      0  1.191857  0.266151  0.166480
2   1.0  378.66      0 -1.358354 -1.340163  1.773209
3   1.0  123.50      0 -0.966272 -0.185226  1.792993
4   2.0   69.99      0 -1.158233  0.877737  1.548718

Un-transformed variables

Time: The elapsed time in seconds between each transaction and the very first transaction in the dataset. Spanning from 0 to 172,792 seconds (approximately 48 hours), the dataset captures a continuous two-day window averaging one transaction every 1.6 seconds and most transactions took place during daytime hours (Fig. 1A).
Amount: The monetary value of the credit card transaction (currency unspecified). Transaction amounts range from 0 to 25,691.16, with a median of 22 and a mean of 88.35, reflecting a heavily right-skewed distribution where the majority of transactions are small (Fig. 1B).
Class: The binary target response variable indicating whether a transaction is legitimate (\(0\)) or fraudulent (\(1\)). The variable contains zero missing values, confirming that all 284,807 transactions in the dataset are fully labeled.

Response-variable balancing

Code

legit_count = r.df['Class'].value_counts()[0]
fraud_count = r.df['Class'].value_counts()[1]
fraud_percent = round(100*(r.df['Class'].value_counts()[1]/len(r.df)),2)

The Class variable was highly imbalanced (Fig. 1C), containing 284315 legitimate transactions and only 492 (0.17%) fraudulent cases. Evaluating response variable balance is essential because traditional classification algorithms, like Logistic Regression, Decision Trees, and Support Vector Machines, operate under the mathematical assumption of an equal class distribution (Ali et al., 2013). Passing heavily skewed data to these models leads them to heavily favor the majority class (i.e., Lazy Classifier), resulting in high rates of false negatives. Because my Class variable is severely imbalanced, I will need to use mitigation strategies like cost-sensitive class weighting (class_weight='balanced', XGBoost scale_pos_weight), over-sampling via SMOTE, or shifting the probability decision threshold away from the \(0.5\) default to reliably predict true fraudulent transactions.

Code

library(tidyverse)
library(patchwork)

# 1. Histogram / Density Plot 1
p1 <- ggplot(df %>% dplyr::mutate(Time = Time/3600), aes(x = .data[[py$untransformed_columns[1]]])) +
  geom_histogram(bins = 30, fill = "steelblue", color = "white") +
  labs(x = paste(py$untransformed_columns[1],"(hours)"), y = "Frequency", title='A.') +
  theme_minimal()

# 2. Histogram / Density Plot 2
p2 <- ggplot(df, aes(x = .data[[py$untransformed_columns[2]]])) +
  geom_histogram(bins = 30, fill = "steelblue", color = "white") +
  labs(x = py$untransformed_columns[2], y = NULL, title='B.') +
  theme_minimal()

# 3. Bar Plot (Categorical Class)
p3 <- ggplot(df, aes(x = factor(Class))) +
  geom_bar(fill = "steelblue", width = 0.5) +
  labs(x = "Class", y = NULL, title='C.') +
  theme_minimal()

# Combine plots and add left-aligned main title
p1 + p2 + p3 + 
  plot_layout(ncol = 3) +
  plot_annotation(
    title = paste0("Fig. ", fig_counter, ". Distribution of Untransformed Features and Target Class"),
    theme = theme(
      plot.title = element_text(size = 14, face = "bold", hjust = 0), # Left-align title
      plot.title.position = "plot" # Align relative to outer plot boundary
    )
  )

Although Principal Component Analysis (PCA) centers features around a mean of zero (\(\mu \approx 0\)), a closer look at the distributions of these variables revealed that the 28 PCA features deviated from a normal distribution. Shapiro-Wilk testing rejected \(H_0\) across all 28 variables (\(p < 0.001\); Table 1). Features such as V28 and V8 displayed severe skewness (11.2, -8.52, respectively; Table 1, Fig. 2), indicating heavy-tailed distributions driven by rare anomalous events. In contrast, features like V13 had low skewness (0.065; Table 1, Fig. 2) and behaved near-symmetrically. Because algorithms like tree ensembles (e.g., Random Forest, XGBoost) do not assume feature normality, they are better suited for these heavy-tailed PCA features than parametric models unless transformations are applied.

Code

library(tidyverse)
library(e1071)  # For skewness and kurtosis
library(gt)

# Calculate normality summary for PCA variables (V1 to V28)
pca_summary <- df %>%
  dplyr::select(starts_with("V")) %>%
  map_df(~ tibble(
    Mean = mean(.x, na.rm = TRUE),
    SD = sd(.x, na.rm = TRUE),
    Skewness = skewness(.x),
    Kurtosis = kurtosis(.x),
    Shapiro_P = shapiro.test(sample(.x, min(length(.x), 5000)))$p.value
  ), .id = "Variable") %>%
  dplyr::arrange(desc(abs(Skewness)))

# Display as a clean GT table

tbl_title <-paste0(
    "Table ",
    tbl_counter,
    ". ",
    "Normality & Distribution Metrics for PCA Features"
)
pca_summary %>%
  gt() %>%
  fmt_number(columns = 2:6, decimals = 3) %>%
  tab_header(title = tbl_title)

Variable	Mean	SD	Skewness	Kurtosis	Shapiro_P
Table 1. Normality & Distribution Metrics for PCA Features
V28	0.000	0.330	11.192	933.375	0.000
V8	0.000	1.194	−8.522	220.582	0.000
V23	0.000	0.624	−5.875	440.078	0.000
V2	0.000	1.651	−4.625	95.771	0.000
V17	0.000	0.849	−3.845	94.797	0.000
V21	0.000	0.735	3.593	207.282	0.000
V1	0.000	1.959	−3.281	32.486	0.000
V7	0.000	1.237	2.554	405.597	0.000
V5	0.000	1.380	−2.426	206.899	0.000
V12	0.000	0.999	−2.278	20.241	0.000
V3	0.000	1.516	−2.240	26.619	0.000
V20	0.000	0.771	−2.037	271.009	0.000
V14	0.000	0.959	−1.995	23.879	0.000
V6	0.000	1.332	1.827	42.641	0.000
V10	0.000	1.089	1.187	31.987	0.000
V27	0.000	0.404	−1.170	244.983	0.000
V16	0.000	0.876	−1.101	10.419	0.000
V4	0.000	1.416	0.676	2.635	0.000
V26	0.000	0.482	0.577	0.919	0.000
V9	0.000	1.099	0.555	3.731	0.000
V24	0.000	0.606	−0.552	0.619	0.000
V25	0.000	0.521	−0.416	4.290	0.000
V11	0.000	1.021	0.357	1.634	0.000
V15	0.000	0.915	−0.308	0.285	0.000
V18	0.000	0.838	−0.260	2.578	0.000
V22	0.000	0.726	−0.213	2.833	0.000
V19	0.000	0.814	0.109	1.725	0.000
V13	0.000	0.995	0.065	0.195	0.000

Code

# Visualizing Skewness Across All 28 PCA Features
fig_title <- paste0(
    "Fig. ",
    fig_counter,
    ". ",
    "Skewness Breakdown Across PCA Features (V1 - V28)"
)
ggplot(pca_summary, aes(x = reorder(Variable, abs(Skewness)), y = Skewness)) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  geom_hline(yintercept = 0, linetype = "dashed", color = "red") +
  labs(
    title = fig_title,
    x = "PCA Feature",
    y = "Skewness (0 = Perfectly Symmetric)"
  ) +
  theme_minimal()

Correlation matrix

Code

fig_counter <- fig_counter +1

To evaluate feature redundancy and identify key signals for fraud, I computed a correlation matrix across all predictors and Class (Fig. 3) . While the PCA components (\(V_1\)–\(V_28\)) remain strictly uncorrelated with each other (\(r \approx 0\)), evaluating their relationships with non-PCA features reveals meaningful patterns. Most notably, transaction Amount exhibits a moderate negative correlation with V2 (\(r = -0.50\)) and V7 (\(r = 0.40\)), indicating that these variables capture information related to the size of a transaction. Similarly, several PCA components showed moderate relationships with Class, which provides an initial indicator of which features may distinguish fraudulent transactions from legitimate ones.

Code

model.matrix(~0+., data=df) %>% 
  cor(use="pairwise.complete.obs") %>% 
  ggcorrplot::ggcorrplot(show.diag = F, type="lower", lab=TRUE, lab_size=2) + labs(title= paste("Fig.", fig_counter, ". Correlation Plot for all predictors and Class."))

Data preparation

Splitting data into train and test sets

To evaluate model performance on unseen data, I partitioned the dataset into a 70% training set (X_train, y_train) and a 30% testing set (X_test, y_test). A 70/30 split provides ample training records—approximately 199,364 transactions—for complex models like tree ensembles to learn feature patterns, while holding back a robust test sample of 85,443 transactions to reliably evaluate rare fraud events without overfitting. Additionally, I used stratified sampling (stratify=y) to ensure both splits preserve the exact 0.17% class prevalence ratio of the original dataset.

Code

# Separate features and target variable
X = r.df.drop('Class', axis=1)
y = r.df['Class']

# Split the dataset into training and testing sets (70% train, 30% test)
# Stratify=y ensures the class ratio is maintained in both splits
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

Addressing scale differences among variables

Because predictors in this dataset span drastically different scales—such as Time exceeding \(170,000\) seconds while PCA variables are centered around \(0.00\)—unscaled features can severely skew how models learn. Models like Logistic Regression look at numerical sizes directly, meaning larger unscaled values can incorrectly appear far more important to the model simply because their numbers are bigger. To put all variables on an equal footing, I standardized the features so that every predictor shares a common average of zero and the same relative spread. Crucially, I calculated these scaling adjustments using only the training set (X_train) before applying them to the test set, ensuring the test set remained completely unseen and untainted during model preparation.

Code

from sklearn.preprocessing import StandardScaler
# 1. Scale the features (fit on train, transform both train and test)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)  # Transform test using training statistics!

Addressing class imbalance

Because legitimate transactions make up over \(99.8\%\) of the dataset, standard models would naturally default to predicting everything as legitimate to maximize overall accuracy. To fix this, I applied class-weighting across all models for consistency. Class-weighting works by placing a much heavier mathematical penalty on missed fraud cases during model training, forcing the algorithm to treat capturing rare fraudulent transactions as its top priority. While I could have generated synthetic fraudulent data to artificially balance the classes (e.g., using SMOTE), class-weighting achieves the same goal without altering or inflating the original dataset.

Results

Logistic Regression

I began my modeling pipeline with Logistic Regression to establish a linear baseline performance floor. This simple model provided a benchmark against which I can compare more complex non-linear algorithms (e.g., Random Forest, XGBoost) to justify any added computational complexity.

Code

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
r.tbl_counter +=1

# 4. Train Logistic Regression on scaled data
lr_model = LogisticRegression(
    max_iter=1000,
    class_weight='balanced'  # Penalizes fraud-detection errors more heavily
)
lr_model.fit(X_train_scaled, y_train)  # Fixed: changed y_train_res to y_train

LogisticRegression(class_weight='balanced', max_iter=1000)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Code

# 5. Predict using scaled test features
y_pred_lr = lr_model.predict(X_test_scaled)

# 6. Evaluate model performance
print(f"Table {int(r.tbl_counter)}. Classification Report for Logistic Regression:\n")

Table 2. Classification Report for Logistic Regression:

Code

print(classification_report(y_test, y_pred_lr))

              precision    recall  f1-score   support

           0       1.00      0.98      0.99     85295
           1       0.07      0.88      0.12       148

    accuracy                           0.98     85443
   macro avg       0.53      0.93      0.56     85443
weighted avg       1.00      0.98      0.99     85443

Code

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.metrics import confusion_matrix
r.fig_counter +=1

# 1. Compute confusion matrix
cm = confusion_matrix(y_test, y_pred_lr)
total_samples = cm.sum()

# 2. Create custom text labels for each cell (Count + Percentage of Total)
group_names = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
group_counts = [f"{value:,}" for value in cm.flatten()]
group_percentages = [f"({value / total_samples:.1%})" for value in cm.flatten()]

labels = [
    f"{name}\n{count} {percent}" 
    for name, count, percent in zip(group_names, group_counts, group_percentages)
]
labels = np.asarray(labels).reshape(2, 2)

# 3. Plot confusion matrix with custom annotations
plt.figure(figsize=(7, 5))
sns.heatmap(
    cm, 
    annot=labels, 
    fmt='', 
    cmap='Blues', 
    cbar=False,
    annot_kws={"size": 11, "weight": "bold"},
    xticklabels=['Predicted Legit (0)', 'Predicted Fraud (1)'], 
    yticklabels=['Actual Legit (0)', 'Actual Fraud (1)']
)

plt.title(f'Fig. {int(r.fig_counter)}. Logistic Regression Confusion Matrix', fontsize=12, pad=12)
plt.tight_layout()
plt.show()

Figure 4 highlights the Logistic Regression model’s biggest trade-off: false positives.

Evaluating the baseline Logistic Regression model demonstrates that while a simple linear model catches the majority of fraud, it generates far too many false alarms to be operationally viable. The model achieved an \(88\%\) recall rate, successfully flagging \(130\) of the \(148\) actual fraudulent transactions in the test set. However, its precision was only \(7\%\), meaning that for every legitimate fraud alert, the model generated roughly \(13\) false alarms (misidentifying \(1,727\) legitimate transactions as fraud). Finally, while overall accuracy reached \(98\%\), this figure is deceptive because legitimate transactions dominate the test set (\(85,295\) out of \(85,443\)). These baseline results establish a clear objective for subsequent models: maintain high recall while drastically improving precision to eliminate false positive alerts.

Random forest

I next trained a Random Forest classifier to evaluate how a tree-based ensemble method handles complex non-linear feature patterns and heavy-tailed distributions without requiring manual feature transformations.

Code

from sklearn.ensemble import RandomForestClassifier
r.tbl_counter +=1

rf_model = RandomForestClassifier(
    n_estimators=100,
    class_weight='balanced',
    random_state=42,
    n_jobs=-1 # uses all available CPU cores
)
rf_model.fit(X_train_scaled, y_train)

RandomForestClassifier(class_weight='balanced', n_jobs=-1, random_state=42)

Code

# Predict on the original test set
y_pred_rf = rf_model.predict(X_test_scaled)

print(f"Table {int(r.tbl_counter)}. Classification Report for Random Forest:\n")

Table 3. Classification Report for Random Forest:

Code

print(classification_report(y_test, y_pred_rf))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00     85295
           1       0.97      0.71      0.82       148

    accuracy                           1.00     85443
   macro avg       0.99      0.85      0.91     85443
weighted avg       1.00      1.00      1.00     85443

Code

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.metrics import confusion_matrix
r.fig_counter +=1

# 1. Compute confusion matrix for Random Forest
cm = confusion_matrix(y_test, y_pred_rf)
total_samples = cm.sum()

# 2. Create custom text labels for each cell (Count + Percentage of Total)
group_names = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
group_counts = [f"{value:,}" for value in cm.flatten()]
group_percentages = [f"({value / total_samples:.2%})" for value in cm.flatten()]

labels = [
    f"{name}\n{count} {percent}" 
    for name, count, percent in zip(group_names, group_counts, group_percentages)
]
labels = np.asarray(labels).reshape(2, 2)

# 3. Plot styled confusion matrix
plt.figure(figsize=(7, 5))
sns.heatmap(
    cm, 
    annot=labels, 
    fmt='', 
    cmap='Blues', 
    cbar=False,
    annot_kws={"size": 11, "weight": "bold"},
    xticklabels=['Predicted Legit (0)', 'Predicted Fraud (1)'], 
    yticklabels=['Actual Legit (0)', 'Actual Fraud (1)']
)

plt.title(f'Fig. {int(r.fig_counter)}. Random Forest Confusion Matrix', fontsize=12, pad=12)
plt.tight_layout()
plt.show()

Figure 5 illustrates the Random Forest confusion matrix. The model correctly identified 105 fraudulent transactions (True Positives) while missing 43 (False Negatives), while misclassifying only 3 legitimate transactions as fraud (False Positives).

This performance represents a dramatic improvement in precision compared to the Logistic Regression baseline. While Logistic Regression suffered from an overwhelming false alarm rate (7% precision; Table 2), Random Forest achieved an outstanding precision of 97% (Table 3), virtually eliminating false alarms across 85,295 legitimate test transactions. This precision boost came with a trade-off in sensitivity: recall dropped from 88% down to 71% (catching 105 of 148 actual fraud cases; Fig. 5). However, because false alarms incur heavy operational costs and customer friction, the overall balance of the model improved significantly, driving the F1-score up from 0.12 to 0.82 (Table 3).

XGBoost boosting ensemble

I next trained an XGBoost classifier to evaluate whether gradient boosting could improve upon Random Forest’s recall rate while maintaining high precision, leveraging sequential tree optimization and cost-sensitive scale weighting (scale_pos_weight) to refine decision boundaries on hard-to-classify fraud cases.

Code

import xgboost as xgb
from sklearn.metrics import classification_report
r.tbl_counter +=1

# 1. Calculate class weight ratio for XGBoost
ratio = (y_train == 0).sum() / (y_train == 1).sum()

# 2. Initialize and train XGBoost Classifier
xgb_model = xgb.XGBClassifier(
    scale_pos_weight=ratio,
    eval_metric='logloss',
    random_state=42,
    n_jobs=-1
)

xgb_model.fit(X_train_scaled, y_train)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=None, device=None, early_stopping_rounds=None,
              enable_categorical=False, eval_metric='logloss',
              feature_types=None, gamma=None, grow_policy=None,
              importance_type=None, interaction_constraints=None,
              learning_rate=None, max_bin=None, max_cat_threshold=None,
              max_cat_to_onehot=None, max_delta_step=None, max_depth=None,
              max_leaves=None, min_child_weight=None, missing=nan,
              monotone_constraints=None, multi_strategy=None, n_estimators=None,
              n_jobs=-1, num_parallel_tree=None, random_state=42, ...)

Code

# 3. Predict on scaled test set
y_pred_xgb = xgb_model.predict(X_test_scaled)

# 4. Print evaluation results
print(f"Table {int(r.tbl_counter)}. Classification Report for XGBoost:\n")

Table 4. Classification Report for XGBoost:

Code

print(classification_report(y_test, y_pred_xgb))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00     85295
           1       0.88      0.79      0.83       148

    accuracy                           1.00     85443
   macro avg       0.94      0.90      0.92     85443
weighted avg       1.00      1.00      1.00     85443

Code

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.metrics import confusion_matrix
r.fig_counter +=1

# 1. Compute confusion matrix for XGBoost
cm = confusion_matrix(y_test, y_pred_xgb)
total_samples = cm.sum()

# 2. Create custom text labels for each cell
group_names = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
group_counts = [f"{value:,}" for value in cm.flatten()]
group_percentages = [f"({value / total_samples:.2%})" for value in cm.flatten()]

labels = [
    f"{name}\n{count} {percent}" 
    for name, count, percent in zip(group_names, group_counts, group_percentages)
]
labels = np.asarray(labels).reshape(2, 2)

# 3. Plot styled confusion matrix
plt.figure(figsize=(7, 5))
sns.heatmap(
    cm, 
    annot=labels, 
    fmt='', 
    cmap='Blues', 
    cbar=False,
    annot_kws={"size": 11, "weight": "bold"},
    xticklabels=['Predicted Legit (0)', 'Predicted Fraud (1)'], 
    yticklabels=['Actual Legit (0)', 'Actual Fraud (1)']
)

plt.title(f'Fig. {int(r.fig_counter)}. XGBoost Confusion Matrix', fontsize=12, pad=12)
plt.tight_layout()
plt.show()

Figure 6 illustrates the XGBoost model’s balanced performance trade-off. The model correctly caught 117 fraudulent transactions (True Positives) while missing 31 (False Negatives), and generated only 16 false alarms on legitimate transactions (False Positives).

Evaluating the XGBoost model demonstrates the effectiveness of gradient boosting for this imbalanced classification task. By applying cost-sensitive scale weighting to prioritize fraud detection during training, XGBoost achieved a recall rate of 79% (Table 4), catching 117 of 148 actual fraudulent transactions (Fig. 6)—an 8 percentage point improvement over Random Forest (Fig. 5). Crucially, this gain in sensitivity was achieved with minimal impact on precision, maintaining an 88% precision rate (Table 4) with only 16 false alarms out of 85,295 legitimate transactions (Fig. 6). By recovering 12 additional fraud cases compared to Random Forest while keeping false alarms to a fraction of Logistic Regression’s rate, XGBoost achieves the best overall operational balance.

Conclusions

Model evaluation

Code

import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
from sklearn.metrics import (
    auc,
    precision_recall_curve,
    roc_curve,
)

font_scale = 2
r.fig_counter += 1

# 1. Obtain predicted probabilities for the Fraud class (Class 1)
y_proba_lr = lr_model.predict_proba(X_test_scaled)[:, 1]
y_proba_rf = rf_model.predict_proba(X_test_scaled)[:, 1]
y_proba_xgb = xgb_model.predict_proba(X_test_scaled)[:, 1]

# High-contrast, colorblind-friendly palette: Black, Blue, Red
models = [
    ('Logistic Regression', y_proba_lr, '#000000'),  # Black
    ('Random Forest', y_proba_rf, '#0072B2'),  # Blue
    ('XGBoost', y_proba_xgb, '#D55E00'),  # Red
]

# 2. Set up side-by-side subplots with adjusted top margin
fig, axes = plt.subplots(1, 2, figsize=(15, 7))

handles_roc = []

# --- LEFT PLOT: ROC Curves ---
for name, y_proba, color in models:
  fpr, tpr, _ = roc_curve(y_test, y_proba)
  roc_auc = auc(fpr, tpr)
  (line,) = axes[0].plot(
      fpr, tpr, lw=2, color=color, label=f'{name} (ROC-AUC: {roc_auc:.3f})'
  )
  handles_roc.append(line)

(roc_base,) = axes[0].plot([0, 1], [0, 1], 'k--', lw=1.5)

axes[0].set_title(
    'A. Receiver Operating Characteristic (ROC)',
    fontsize=11 * font_scale,
    pad=15,
)

Code

axes[0].set_xlabel('False Positive Rate', fontsize=10 * font_scale)

Code

axes[0].set_ylabel('True Positive Rate (Recall)', fontsize=10 * font_scale)

Code

axes[0].grid(alpha=0.3)

# --- RIGHT PLOT: Precision-Recall Curves ---
for name, y_proba, color in models:
  precision, recall, _ = precision_recall_curve(y_test, y_proba)
  axes[1].plot(recall, precision, lw=2, color=color)

Code

baseline = y_test.mean()
pr_base = axes[1].axhline(y=baseline, color='k', linestyle='--', lw=1.5)

axes[1].set_title(
    'B. Precision-Recall (PR)', fontsize=11 * font_scale, pad=15
)

Code

axes[1].set_xlabel('Recall', fontsize=10 * font_scale)

Code

axes[1].set_ylabel('Precision', fontsize=10 * font_scale)

Code

axes[1].grid(alpha=0.3)

# 3. Main Centered Figure Title positioned cleanly above subplots
fig.suptitle(
    f'Fig. {int(r.fig_counter)}. Model Evaluation',
    fontsize=12 * font_scale,
    y=0.98,
)

Code

# 4. Explicitly adjust top to 0.82 (reserves top ~18% for main title)
fig.subplots_adjust(
    left=0.08, right=0.95, bottom=0.32, top=0.82, wspace=0.35
)

# 5. Interleave list items with updated intuitive baseline labels
empty_patch = mpatches.Patch(color='none', label='')

all_handles = [
    handles_roc[0],
    roc_base,
    handles_roc[1],
    pr_base,
    handles_roc[2],
    empty_patch,
]

all_labels = [
    'Logistic Regression',
    'Random-Chance Baseline (0.500)',
    'Random Forest',
    f'Prevalence Baseline ({baseline:.4f})',
    'XGBoost',
    '',
]

fig.legend(
    handles=all_handles,
    labels=all_labels,
    loc='lower center',
    bbox_to_anchor=(0.5, 0.02),
    ncol=3,
    frameon=True,
    fontsize=9.5 * font_scale,
)

Code

plt.show()

Figure 7 evaluates overall model performance by comparing Logistic Regression, Random Forest, and XGBoost against random-chance and class prevalence baselines.

In Fig. 7A (ROC), all three models perform significantly better than the diagonal random-chance baseline (\(0.500\)). Random Forest and XGBoost rapidly climb to a True Positive Rate (Recall) near \(0.90\) at very low False Positive Rates (\(<0.05\)), capturing the vast majority of fraud with almost no legitimate transactions flagged. In contrast, Logistic Regression climbs much more gradually across intermediate false positive rates before eventually converging with the tree ensembles.

In Fig. 7B (Precision-Recall), models are evaluated against the class prevalence baseline (\(0.0017\))—the expected precision of a naive classifier guessing fraud strictly at its natural occurrence rate. Because fraud represents only \(0.17\%\) of transactions, this baseline sits near zero, demonstrating how difficult high precision is to sustain under extreme class imbalance. While all three models exceed this benchmark, Random Forest and XGBoost maintain near-perfect precision (\(1.00\)) across recall levels up to roughly \(0.80\). Conversely, Logistic Regression suffers an immediate, steep drop in precision as recall exceeds \(0.10\), confirming that tree-based ensemble methods are vastly superior at isolating rare fraudulent transactions without flooding operations with false alarms.

Model selection

Code

library(gt)
library(dplyr)
tbl_counter <- tbl_counter + 1

# 1. Create the detailed comparison data frame
model_comp <- tibble(
  Metric = c(
    "True Negative (Legit → Legit)",
    "False Positive (Legit → Fraud)",
    "False Negative (Fraud → Legit)",
    "True Positive (Fraud → Fraud)",
    "Precision",
    "Recall",
    "F1-Score"
  ),
  `Logistic Regression` = c(
    "83,568 (97.81%)",
    "1,727 (2.02%)",
    "18 (0.02%)",
    "130 (0.15%)",
    "0.07 (7%)",
    "0.88 (88%)",
    "0.12"
  ),
  `Random Forest` = c(
    "85,292 (99.82%)",
    "3 (0.00%)",
    "43 (0.05%)",
    "105 (0.12%)",
    "0.97 (97%)",
    "0.71 (71%)",
    "0.82"
  ),
  `XGBoost` = c(
    "85,279 (99.81%)",
    "16 (0.02%)",
    "31 (0.04%)",
    "117 (0.14%)",
    "0.88 (88%)",
    "0.79 (79%)",
    "0.83"
  )
)

# Style list helper for best cells
highlight_style <- list(
  cell_fill(color = "#EBF5FB"),
  cell_text(weight = "bold")
)

# 2. Build the scorecard gt table
gt_table <- model_comp %>%
  gt() %>%
  tab_header(
    title = md(paste0("**Table ",tbl_counter,". Head-to-Head Model Performance Comparison**")),
    subtitle = "Confusion matrix counts and classification metrics on the test dataset (N = 85,443)"
  ) %>%
  cols_align(
    align = "center",
    columns = c(`Logistic Regression`, `Random Forest`, `XGBoost`)
  ) %>%
  cols_align(
    align = "left",
    columns = Metric
  ) %>%
  # Group rows into Confusion Matrix vs. Higher-Level Metrics
  tab_row_group(
    label = md("**Classification Metrics (Class 1)**"),
    rows = 5:7
  ) %>%
  tab_row_group(
    label = md("**Confusion Matrix Counts**"),
    rows = 1:4
  ) %>%
  # --- HIGHLIGHT BEST VALUES (SCORECARD STYLE) ---
  # True Negatives: Random Forest (Highest: 85,292)
  tab_style(style = highlight_style, locations = cells_body(columns = `Random Forest`, rows = 1)) %>%
  # False Positives: Random Forest (Lowest / Best: 3)
  tab_style(style = highlight_style, locations = cells_body(columns = `Random Forest`, rows = 2)) %>%
  # False Negatives: Logistic Regression (Lowest: 18)
  tab_style(style = highlight_style, locations = cells_body(columns = `Logistic Regression`, rows = 3)) %>%
  # True Positives: Logistic Regression (Highest: 130)
  tab_style(style = highlight_style, locations = cells_body(columns = `Logistic Regression`, rows = 4)) %>%
  # Precision: Random Forest (Highest: 0.97)
  tab_style(style = highlight_style, locations = cells_body(columns = `Random Forest`, rows = 5)) %>%
  # Recall: Logistic Regression (Highest: 0.88)
  tab_style(style = highlight_style, locations = cells_body(columns = `Logistic Regression`, rows = 6)) %>%
  # F1-Score: XGBoost (Highest: 0.83)
  tab_style(style = highlight_style, locations = cells_body(columns = XGBoost, rows = 7)) %>%
  # Table formatting options
  tab_options(
    table.width = pct(95),
    heading.title.font.size = px(14),
    heading.subtitle.font.size = px(11),
    column_labels.font.weight = "bold",
    row_group.font.weight = "bold",
    table.border.top.color = "black",
    table.border.bottom.color = "black"
  )

# Display table
gt_table

Metric	Logistic Regression	Random Forest	XGBoost
Table 5. Head-to-Head Model Performance Comparison
Confusion matrix counts and classification metrics on the test dataset (N = 85,443)
Confusion Matrix Counts
True Negative (Legit → Legit)	83,568 (97.81%)	85,292 (99.82%)	85,279 (99.81%)
False Positive (Legit → Fraud)	1,727 (2.02%)	3 (0.00%)	16 (0.02%)
False Negative (Fraud → Legit)	18 (0.02%)	43 (0.05%)	31 (0.04%)
True Positive (Fraud → Fraud)	130 (0.15%)	105 (0.12%)	117 (0.14%)
Classification Metrics (Class 1)
Precision	0.07 (7%)	0.97 (97%)	0.88 (88%)
Recall	0.88 (88%)	0.71 (71%)	0.79 (79%)
F1-Score	0.12	0.82	0.83

Table 5 presents the final model evaluation scorecard, with bold, highlighted cells indicating the top-performing model for each metric across the test set (\(N = 85,443\)).

Despite achieving the highest True Positive count (\(130\)) and lowest False Negative count (\(18\)), Logistic Regression is not a viable deployment choice. Its low precision (\(0.07\)) produced \(1,727\) false positives, which would create overwhelming operational review costs and severe customer friction.

In contrast, Random Forest and XGBoost performed exceptionally well, presenting a classic business trade-off between customer experience and financial loss prevention. I recommend presenting business leadership with these two options, framing the final deployment decision around their operational risk tolerance:

Option A — Random Forest (Minimal Customer Friction): If the primary objective is to eliminate false alarms and prevent cardholder inconvenience, Random Forest is optimal. It achieved a near-zero false positive rate (\(0.00\%\), with only \(3\) false alarms across \(85,295\) legitimate transactions) and an exceptional precision of \(0.97\).

Option B — XGBoost (Maximum Fraud Recovery): If the primary objective is to minimize direct financial loss from uncollected fraud, XGBoost is optimal. By accepting a minor increase of \(13\) false alarms (\(16\) vs. \(3\)), XGBoost recovered \(12\) additional fraudulent transactions (Recall of \(0.79\) vs. \(0.71\)), achieving the highest overall F1-score (\(0.83\)).

Recommendation: Assuming the financial liability of missed fraud outweighs the minor operational cost of \(13\) additional alerts, XGBoost is the recommended model for production deployment. However, if preventing cardholder friction and minimizing manual review overhead are paramount, Random Forest serves as a highly robust alternative.

Glossary

Precision: The proportion of flagged transactions that are actually fraudulent, showing how trustworthy the model’s fraud alerts are.
Recall: The proportion of total fraudulent transactions that the model successfully catches, showing how effectively it prevents missed fraud.
F1 Score: A single summary metric that balances precision and recall to measure overall accuracy on imbalanced datasets.
Receiver Operating Characteristic (ROC): A graph that illustrates how well a model detects true fraud cases while keeping false alarms low across different confidence thresholds.
Precision-Recall (PR): A graph that measures how accurately a model identifies rare fraud cases without generating excessive false alarms, making it ideal for severely imbalanced data.

References

Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015

Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon

Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE

Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)

Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-Aël; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier

Carcillo, Fabrizio; Le Borgne, Yann-Aël; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing

Bertrand Lebichot, Yann-Aël Le Borgne, Liyun He, Frederic Oblé, Gianluca Bontempi Deep-Learning Domain Adaptation Techniques for Credit Cards Fraud Detection, INNSBDDL 2019: Recent Advances in Big Data and Deep Learning, pp 78-88, 2019

Fabrizio Carcillo, Yann-Aël Le Borgne, Olivier Caelen, Frederic Oblé, Gianluca Bontempi Combining Unsupervised and Supervised Learning in Credit Card Fraud Detection Information Sciences, 2019

Yann-Aël Le Borgne, Gianluca Bontempi Reproducible machine Learning for Credit Card Fraud Detection - Practical Handbook

Bertrand Lebichot, Gianmarco Paldino, Wissam Siblini, Liyun He, Frederic Oblé, Gianluca Bontempi Incremental learning strategies for credit cards fraud detection, IInternational Journal of Data Science and Analytics

Ali, Aida, Siti Mariyam Shamsuddin, and Anca L. Ralescu. “Classification with Class Imbalance Problem: A Review.” International Journal of Advances in Soft Computing and Its Applications, vol. 5, no. 3, 2013, pp. 1–30.