7333 Module 14 - Ensembling - QTW - Dr Slater Spring 2025

Here’s your detailed Module 14 Study Guide: Ensembling — Quantifying the World with Dr. Slater
Includes full coverage: visuals, math, coding, examples, and end-of-module questions.

---

Title: Module 14 – Ensembling: Voting, Stacking, and Real-World Performance

---

Section 1: What Is Ensembling?

Definition:
Ensembling is a technique where multiple models (weak learners) are combined to create a stronger, more accurate model.

Two main types:

1. Voting (Bagging) – Combines outputs of independent models. Majority or weighted vote decides the output.
2. Stacking (Stacked Generalization) – A meta-model learns to combine the predictions of base models.

---

Section 2: Voting Ensemble — Concept and Math

From Slide 1:

• Models: A, B, C
• Each model gives binary predictions
• Final decision = majority vote

Example Voting Table Recap: Model A: [1, 0, 1, 1, 0, 1, 1, 0, 0, 0] (80%)
Model B: [1, 1, 0, 1, 1, 0, 1, 0, 1, 1] (60%)
Model C: [1, 0, 0, 0, 1, 1, 1, 1, 0, 1] (70%)
Vote Winner: [1, 0, 0, 1, 1, 1, 1, 0, 0, 1]
True Values: [1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
Vote Accuracy = 90%

Mathematical Notation:
Let \(M_i(x)\) be prediction of model i on input x.
Voting ensemble prediction:
\[ \hat{y} = \text{mode}(M_1(x), M_2(x), ..., M_n(x)) \]

---

Section 3: Stacking — Ensemble Diagram

From Slide 2:

• Models A, B, and C are trained on original data.
• Their predictions are collected as features for training a new model φ (meta-model).
• Out-of-Fold (OOF): Prevents data leakage by training base models on k-1 folds and generating predictions on the k-th.

Mathematical Flow: Let: - \(h_i(x)\): Output of base model i
- Meta-model φ gets vector \([h_1(x), h_2(x), ..., h_n(x)]\)
- φ is trained to predict true label y from this vector.

\[ \phi([h_1(x), h_2(x), ..., h_n(x)]) \approx y \]

---

Section 4: Real-World Impact of Ensembling

From Slide 3:

• SQuAD2 Leaderboard (Nov 2020):
  • Top 6 models were all ensembles.
  • Best single model: Rank 7.
  • Ensembles increase EM score, e.g. 89.55 (single) → 90.72 (ensemble)

---

Section 5: Code Examples

VotingClassifier (Sklearn)

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

clf1 = LogisticRegression(max_iter=1000)
clf2 = DecisionTreeClassifier()
clf3 = SVC(probability=True)

ensemble = VotingClassifier(estimators=[
    ('lr', clf1), ('dt', clf2), ('svc', clf3)], voting='soft')

scores = cross_val_score(ensemble, X, y, cv=5)
print(f'Voting Ensemble Accuracy: {scores.mean():.4f}')

StackingClassifier (Sklearn)

from sklearn.ensemble import StackingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier

base_learners = [
    ('gnb', GaussianNB()),
    ('knn', KNeighborsClassifier())
]
meta = LogisticRegression()

stack_model = StackingClassifier(estimators=base_learners, final_estimator=meta)
scores = cross_val_score(stack_model, X, y, cv=5)
print(f'Stacking Ensemble Accuracy: {scores.mean():.4f}')

---

Section 6: Why It Works

Bias-Variance Tradeoff:
- Voting lowers variance. - Stacking lowers both bias and variance if base models are diverse.

Theorem (Simplified): If each classifier has error < 0.5 and errors are independent, then ensembling reduces total error.

---

Final Takeaways:

• Voting works well when models are independent and diverse.
• Stacking is more powerful but requires careful cross-validation.
• Ensembles always outperform a single model if implemented properly.
• Real-world systems (e.g., top NLP systems) use ensembles for SOTA performance.

---

Questions for Dr. Slater:

1. When stacking, how do we avoid overfitting with the meta-model?
2. Are there known limits to ensemble size before it becomes computationally inefficient?
3. How does ensemble diversity affect final accuracy in non-binary classification?
4. Can ensemble models be interpreted easily for explainability?
5. Are there practical techniques for ensembling deep learning models like ConvNets or RNNs?