Bagging

Bagging: A Study Guide with Mathematical and Coding Representation

1. Introduction to Bagging

Definition

Bagging (Bootstrap Aggregating) is an ensemble learning technique that improves model stability and accuracy by training multiple models on different random subsets of data and then aggregating their predictions.

Bagging is widely used in algorithms like Random Forests, where multiple decision trees are trained independently and combined for improved performance.

---

2. Mathematical Representation of Bagging

Bagging reduces variance by training models on multiple bootstrapped subsets and averaging their outputs.

Bootstrap Sampling

Given a dataset \(D\) with \(N\) samples, bagging creates multiple subsets \(D_b\) by randomly sampling with replacement: \[ D_b = \{ x_i | x_i \in D, \text{sampled with replacement} \} \]

Each model is trained on a different subset \(D_b\) and their predictions \(f_b(x)\) are aggregated:

• For Regression (Averaging Outputs): \[ F(x) = \frac{1}{B} \sum_{b=1}^{B} f_b(x) \]

• For Classification (Majority Voting): \[ F(x) = ext{mode} \{ f_b(x) \} \]

where: - \(B\) is the total number of bootstrapped models - \(f_b(x)\) is the prediction from the \(b\)-th model

Bagging reduces overfitting and enhances robustness by minimizing individual model biases.

---

3. Bagging in Decision Trees

Bagging is particularly effective in decision trees, where high variance models benefit from aggregation.

• Bootstrap Sample Creation: Each tree is trained on a different randomly sampled subset.
• Independent Model Training: Each tree makes its own predictions without interaction.
• Aggregation: The final output is obtained via majority voting (classification) or averaging (regression).

Bagging is the core technique behind Random Forests, where additional randomness is introduced by selecting feature subsets at each split.

---

4. Python Implementation of Bagging

Training a Bagging Classifier

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load dataset
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, test_size=0.2, random_state=42)

# Train a single Decision Tree (without bagging)
dt = DecisionTreeClassifier(random_state=42)
dt.fit(X_train, y_train)
y_pred_dt = dt.predict(X_test)
accuracy_dt = accuracy_score(y_test, y_pred_dt)

# Train a BaggingClassifier with 50 bootstrap samples
bagging = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=50, random_state=42)
bagging.fit(X_train, y_train)
y_pred_bagging = bagging.predict(X_test)
accuracy_bagging = accuracy_score(y_test, y_pred_bagging)

# Print results
print(f"Accuracy of single Decision Tree: {accuracy_dt:.4f}")
print(f"Accuracy with Bagging (50 estimators): {accuracy_bagging:.4f}")

Visualizing the Bagging Process

import matplotlib.pyplot as plt
import numpy as np

# Example of bootstrap sampling
original_data = np.arange(1, 11)
bootstrap_sample = np.random.choice(original_data, size=10, replace=True)

plt.hist(bootstrap_sample, bins=np.arange(1, 12)-0.5, edgecolor='black', alpha=0.7)
plt.xticks(np.arange(1, 11))
plt.xlabel("Sample Values")
plt.ylabel("Frequency")
plt.title("Bootstrap Sampling Example")
plt.show()

---

5. Key Takeaways

1. Bagging reduces variance by training multiple models on bootstrapped subsets and aggregating predictions.
2. Bootstrap sampling ensures diversity in training, making the final model more robust.
3. Bagging works best with high-variance models like decision trees, enhancing their generalization ability.
4. Random Forests extend bagging by adding feature randomness, further improving performance and stability.

By leveraging bagging, we can significantly improve predictive performance while minimizing overfitting in machine learning models.