7333 QTW - Module 2

Study Guide: Understanding Overfitting and Regularization in Machine Learning (integrating theory, mathematical foundations, and practical examples to regularization techniques to help understand their impact on overfitting in machine learning.

---

1. Overfitting

• Definition: Overfitting occurs when a model is overly tailored to the training data, leading to poor performance on unseen data.
• Symptoms:
  • High accuracy on training data but poor performance on validation/test data.
  • The model captures noise and specific patterns of the training data.
• Key Issue: Overfitted models have low bias but high variance.

2. Tools to Combat Overfitting

• Validation Set: Monitor performance during training.
• Regularization: Introduce penalties to the loss function.

---

3. Regularization: Core Concept

• Definition: Adding a penalty term to the loss function to discourage complex models.
• Mathematical Representation: \[ \text{Loss Function} = \underbrace{\sum (y_i - \hat{y}_i)^2}_{\text{Squared Loss}} + \underbrace{\lambda \cdot \text{Penalty}}_{\text{Regularization Term}} \]
  • \(\lambda\): Regularization strength (hyperparameter).
  • \(\text{Penalty}\): Function of model parameters to constrain them.

---

4. Types of Regularization

• L1 Regularization (LASSO):

  • Penalty: \(\lambda \sum |w_i|\) (absolute values of coefficients).
  • Use Case: Feature selection (some coefficients are driven to 0, creating sparse models).
  • Interpretation: Encourages sparsity by setting weak feature coefficients to 0.

  \[ \text{Loss} = \sum (y_i - \hat{y}_i)^2 + \lambda \sum |w_i| \]

• L2 Regularization (Ridge):

  • Penalty: \(\lambda \sum w_i^2\) (squared values of coefficients).
  • Use Case: Generalized overfitting prevention without sparsity.
  • Interpretation: Penalizes large coefficients more strongly, but all features remain non-zero.

  \[ \text{Loss} = \sum (y_i - \hat{y}_i)^2 + \lambda \sum w_i^2 \]

---

5. Comparison: L1 vs. L2

Aspect	L1 (LASSO)	L2 (Ridge)
Penalty Function	Absolute (\(|w_i|\))	Squared (\(w_i^2\))
Behavior	Sparse coefficients	Uniform coefficient shrinkage
Use Case	Feature selection	Prevent overfitting
Geometry of Penalty	Diamond-shaped exclusion zone	Circular exclusion zone

---

6. Practical Implementation: Python Examples

• Data Preprocessing:

  from sklearn.preprocessing import StandardScaler
  scaler = StandardScaler()
  X_scaled = scaler.fit_transform(X)  # Normalize features

• L1 Regularization (LASSO):

  from sklearn.linear_model import Lasso
  lasso = Lasso(alpha=0.1)  # Set regularization strength
  lasso.fit(X_scaled, y)
  print("Coefficients:", lasso.coef_)

• L2 Regularization (Ridge):

  from sklearn.linear_model import Ridge
  ridge = Ridge(alpha=1.0)  # Set regularization strength
  ridge.fit(X_scaled, y)
  print("Coefficients:", ridge.coef_)

• Tuning Regularization Strength (\(\lambda\)):

  from sklearn.model_selection import cross_val_score
  alphas = [0.01, 0.1, 1, 10, 100]
  for alpha in alphas:
      model = Ridge(alpha=alpha)
      scores = cross_val_score(model, X_scaled, y, cv=5, scoring='neg_mean_squared_error')
      print(f"Alpha: {alpha}, Score: {-scores.mean()}")

---

7. Bias-Variance Tradeoff

• High Bias (Underfitting): Model is too simple, failing to capture data patterns.
• High Variance (Overfitting): Model is too complex, capturing noise in the data.
• Regularization: Balances bias and variance to achieve a generalizable model.

---

8. Key Takeaways

1. Regularization Strength (\(\lambda\)):
   • Controls the impact of the penalty.
   • Needs to be tuned experimentally.
2. Interpret Coefficients:
   • Importance of scaling features to ensure comparability.
3. Validation:
   • Use cross-validation to determine optimal \(\lambda\).
4. L1 vs. L2:
   • L1 for feature selection, L2 for preventing overfitting.

---

9. Visualization and Tools

• Visualize penalties and data fit:

  import matplotlib.pyplot as plt
  plt.plot(alphas, scores)
  plt.xlabel('Alpha (λ)')
  plt.ylabel('Cross-Validation Loss')
  plt.title('Tuning Regularization Strength')
  plt.show()

Questions from the Lecture

1. Conceptual Question:
   What is the key difference between L1 (LASSO) and L2 (Ridge) regularization, and how does it affect model coefficients?

2. Analytical Question:
   How does increasing the regularization strength (\(\lambda\)) affect the bias-variance tradeoff in a machine learning model?

3. Practical Question:
   Why is it important to scale features before applying regularization, and how does it impact the interpretation of model coefficients?

---

Takeaways from the Lecture

1. Understanding Overfitting:
   Overfitting occurs when a model performs exceptionally well on training data but poorly on unseen data. Regularization is a critical tool to combat overfitting by constraining model complexity.

2. Role of Regularization Techniques:

   • L1 (LASSO) regularization introduces sparsity by driving weak feature coefficients to zero, making it ideal for feature selection.
     
   • L2 (Ridge) regularization uniformly shrinks coefficients, preventing overfitting without eliminating features.

3. Importance of Hyperparameter Tuning:
   The regularization strength (\(\lambda\)) must be carefully tuned (e.g., using cross-validation) to balance model bias and variance, ensuring optimal performance on unseen data.