73333 Module 3 Trasncript and Summaries

Logistic Regression Study Guide

Key Concepts

1. Categorical Data

• Definition: Data based on classes (e.g., red/green/blue, true/false).
• Transformation: Use one-hot encoding to convert non-numeric data into numeric format.
  • Example: Red = [1, 0, 0], Green = [0, 1, 0], Blue = [0, 0, 1].
  • Each category is represented by a binary vector.
• Discrete Data: Numeric values without fractional parts (e.g., number of rooms, children).
  • Do not one-hot encode discrete data; treat it as numeric.

2. Linear to Logistic Regression

• Linear regression predicts continuous values, but logistic regression is used for categorical targets.
• Logistic regression transforms linear outputs into probabilities using the sigmoid function.

3. The Sigmoid Function

• Formula: \(\sigma(x) = \frac{1}{1 + e^{-x}}\)
• Squeezes input values into the range (0, 1), representing probabilities.
• Default classification threshold is 0.5 but can be adjusted based on the use case (e.g., fraud detection).

4. Log Loss (Logarithmic Loss)

• Measures the distance between predicted probabilities and actual binary targets.
• Formula:
  \(\text{Log Loss} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]\)
  • \(y_i\): Actual class (0 or 1).
  • \(p_i\): Predicted probability for class 1.
• Penalizes predictions farther from the actual target.

5. Minimizing Log Loss

• Uses gradient descent to adjust weights (\(m\)) and minimize loss.
• Partial derivatives of log loss guide weight updates: \(m_{new} = m_{old} - \alpha \frac{\partial J}{\partial m}\), where \(J\) is the log loss and \(\alpha\) is the learning rate.

6. Multiclass Classification

• Extends binary logistic regression to multiple categories.
• Two approaches:
  • One-vs-All (OvA): Train a separate classifier for each class.
  • One-vs-One (OvO): Compare every pair of classes; assign the class with the most wins.
• Use libraries (e.g., sklearn) for built-in multiclass implementation.

---

Demonstration of Logistic Regression

Part I: Binary Classification (Breast Cancer Dataset)

Steps:

1. Data Preparation:
   • Import dataset from sklearn.datasets.
   • Convert data to a DataFrame and add column names.
   • Separate features (X) and targets (y).
2. Logistic Regression:
   • Use LogisticRegressionCV for cross-validation.
   • Fit the model: model.fit(X, y).
   • Retrieve model coefficients with model.coef_.
3. Cross-Validation:

   • For unbiased performance metrics, use cross_val_score.

   • Example:

     from sklearn.model_selection import cross_val_score
     scores = cross_val_score(model, X, y, scoring='accuracy', cv=5)
     print("Accuracy:", scores.mean())

Part II: Cross-Validation and Regularization

• Use LogisticRegressionCV to optimize regularization parameters.
• Interpret the results and evaluate metrics.

---

Case Study: Hospital Readmission Prediction

Problem Overview

• Predict patient readmission within 30 days using logistic regression.
• Challenges:
  • Missing data must be imputed.
  • Ethical considerations in using sensitive features (e.g., race).
  • Imbalanced classes.

Assignment Steps:

1. Data Preprocessing:
   • Handle missing data using imputation techniques (e.g., mean, median).
   • Normalize/scale features for better model performance.
2. Model Development:
   • Build a logistic regression model for each target class (multiclass setup).
   • Use cross-validation to evaluate performance.
3. Feature Importance:
   • Analyze the top 5 important features contributing to predictions using model coefficients.

---

Mathematical and Coding Representations

Mathematical Representation

1. Sigmoid Function:
   \(\sigma(x) = \frac{1}{1 + e^{-x}}\)
2. Log Loss:
   \(J = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]\)
3. Gradient Update:
   \(m_{new} = m_{old} - \alpha \frac{\partial J}{\partial m}\)

Python Code Representation

1. One-Hot Encoding:

   import pandas as pd
   from sklearn.preprocessing import OneHotEncoder
   
   data = {'Color': ['Red', 'Blue', 'Green']}
   df = pd.DataFrame(data)
   encoder = OneHotEncoder()
   encoded = encoder.fit_transform(df[['Color']]).toarray()
   print(encoded)

2. Logistic Regression:

   from sklearn.datasets import load_breast_cancer
   from sklearn.model_selection import train_test_split
   from sklearn.linear_model import LogisticRegressionCV
   
   # Load data
   data = load_breast_cancer()
   X, y = data.data, data.target
   
   # Split data
   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
   
   # Logistic Regression with CV
   model = LogisticRegressionCV(cv=5, max_iter=1000).fit(X_train, y_train)
   print("Accuracy:", model.score(X_test, y_test))

3. Cross-Validation:

   from sklearn.model_selection import cross_val_score
   
   scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
   print("Cross-Validation Accuracy:", scores.mean())

Visualization of Sigmoid Function

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-10, 10, 100)
sigmoid = 1 / (1 + np.exp(-x))

plt.plot(x, sigmoid)
plt.title('Sigmoid Function')
plt.xlabel('x')
plt.ylabel('Sigmoid(x)')
plt.grid()
plt.show()

Study Guide: Logistic Regression with Mathematical and Coding Representations

---

1. Categorical Data

• Definition: Data representing classes or categories (e.g., red, green, blue; true/false).
• Challenges: Categorical data must be transformed into numerical data to be used in most machine learning models.
• Transformation:
  • One-Hot Encoding: Converts categorical features into binary columns, where each unique value becomes a column. E.g., for colors red, green, blue:
    • Red: [1, 0, 0]
    • Green: [0, 1, 0]
    • Blue: [0, 0, 1]

Python Example:

import pandas as pd

data = {'Color': ['Red', 'Green', 'Blue']}
df = pd.DataFrame(data)
df_encoded = pd.get_dummies(df, columns=['Color'])

Mathematical Representation: If \(C\) represents the categories and \(x\) is the input, the transformation is: \[ \text{One-hot encoded vector: } \mathbf{x}_{\text{encoded}} = [x_1, x_2, \ldots, x_n], \text{where } x_i = 1 \text{ if } x = C_i, \text{ else } 0. \]

---

2. Linear to Logistic Regression

• Linear regression predicts continuous values. Logistic regression transforms this to predict probabilities for binary outcomes.
• Key Equation: \[ z = \mathbf{w}^T\mathbf{x} + b, \quad p = \sigma(z), \quad \sigma(z) = \frac{1}{1 + e^{-z}} \] Where \(\sigma(z)\) (sigmoid function) squashes \(z\) to range [0, 1].

Python Example:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train, y_train)

---

3. The Sigmoid Function

• Equation: \(\sigma(z) = \frac{1}{1 + e^{-z}}\)
• Properties:
  • \(z \to +\infty\): \(\sigma(z) \to 1\)
  • \(z \to -\infty\): \(\sigma(z) \to 0\)
  • Outputs probabilities.

Python Example:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-10, 10, 100)
sigmoid_values = sigmoid(z)

---

4. Log Loss

• Measures the distance between predicted probabilities and actual class labels.
• Equation: \[ \text{Log Loss: } L(y, \hat{y}) = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right] \] Where:
  • \(y_i\): Actual label.
  • \(\hat{y}_i\): Predicted probability.

Python Example:

from sklearn.metrics import log_loss
loss = log_loss(y_true, y_pred)

---

5. Minimizing Log Loss

• Achieved through optimization (e.g., gradient descent).
• Gradient Descent:
  • Update rule: \(w = w - \alpha \nabla L(w)\), where \(\alpha\) is the learning rate.
• Python Example:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=100)
model.fit(X_train, y_train)

---

6. Multiclass Classification

• Extends binary logistic regression to multiple classes.
• Approaches:
  • One-vs-Rest (OvR): Train separate classifiers for each class.
  • Softmax Regression: Directly models all classes using probabilities.
• Softmax Function: \[ \sigma(z_j) = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}} \]

Python Example:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(multi_class='multinomial', solver='lbfgs')
model.fit(X_train, y_train)

---

Case Study: Predicting Diabetes Readmission

Objective

• Predict hospital readmission within 30 days using logistic regression.
• Handle missing data via imputation.

Steps:

1. Data Preprocessing:
   • Handle missing values (e.g., mean/mode imputation).
   • Encode categorical variables (e.g., one-hot encoding).
2. Model Building:
   • Train logistic regression for three classes of readmission:
     • No readmission.
     • Readmission in less than 30 days.
     • Readmission in more than 30 days.
   • Evaluate using metrics like log loss or accuracy.
3. Feature Importance:
   • Extract coefficients to identify top 5 predictive features.

Python Code:

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

# Data preprocessing
imputer = SimpleImputer(strategy='mean')
X = imputer.fit_transform(X)

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Logistic regression model
model = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=500)
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)

# Feature importance
importance = model.coef_
print("Top 5 Features:", importance.argsort()[-5:])

# Accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy}")

Variable Importance Interpretation

• Coefficients reflect the importance of each feature.
• Analyze top variables to derive insights for readmission patterns.

---

Assignment

• Train and evaluate logistic regression on the diabetes dataset.
• Report top 5 features and their significance.
• Submit results with a clear explanation of findings.

Deliverable: - Code file (e.g., FirstName_LastName_LogReg_Assignment.py). - Written report on model performance and feature importance.

Key Takeaways

1. Logistic Regression Bridges Linear Models and Probabilistic Predictions:

   • Logistic regression extends linear regression for categorical target variables by employing the sigmoid function to output probabilities. This allows for binary or multiclass predictions by interpreting probabilities as class memberships.

2. Log Loss as a Metric:

   • Unlike linear regression’s mean squared error, logistic regression uses log loss to measure the model’s performance. It penalizes predictions further from the true class, ensuring probabilities are accurate.

3. Handling Multiclass Problems:

   • Multiclass classification can be addressed using methods like one-vs-rest (OvR) or softmax regression. While OvR is computationally efficient for a few classes, it scales poorly with many classes due to class imbalance issues.

4. Importance of Sigmoid Function in Logistic Regression: The sigmoid function transforms the linear regression output into probabilities, making it possible to handle binary classification tasks effectively. This function ensures that predictions fall within the range [0, 1], which can then be used to classify data into distinct categories.

5. Log Loss for Classification: Logarithmic loss (log loss) quantifies the error in probabilistic predictions by penalizing predictions that deviate significantly from the true labels. It forms a convex function with a well-defined minimum, facilitating optimization and convergence.

6. Multiclass Classification Challenges: Logistic regression can be extended to multiclass problems through techniques like One-vs-All and One-vs-One. While these methods allow logistic regression to handle more than two classes, they come with scalability and bias challenges as the number of classes increases.

---

Questions for Class Discussion

1. Threshold Adjustment in Logistic Regression:

   • How can adjusting the classification threshold (e.g., moving it from 0.5 to 0.2 for fraud detection) impact the balance between false positives and false negatives? What real-world examples highlight the importance of this?

2. Log Loss vs. Accuracy:

   • In what scenarios might log loss be a more appropriate evaluation metric than accuracy? Can you provide examples where accuracy might be misleading?

3. Ethical Considerations in Categorical Variables:

   • When including sensitive variables like race or gender in logistic regression, how can we ensure the model is ethically sound and avoids discriminatory practices while leveraging potentially predictive information?

4. On Sigmoid Threshold Adjustment: In real-world applications like fraud detection, how do we decide on the optimal threshold for classification beyond the default value of 0.5? What strategies or metrics should guide this decision?

5. On Handling Missing Data: When dealing with missing values in a dataset, as mentioned in the diabetes case study, what factors should influence the choice of an imputation strategy? How do we ensure the imputation does not introduce bias into the model?

6. On Ethical Considerations in Feature Selection: In cases where sensitive features like race are included in the dataset, how do we balance the potential utility of such features with ethical concerns and the risk of perpetuating bias in predictions?

Best Takeaways

1. Logistic Regression Bridges Linear Models and Probabilistic Predictions:
   • Logistic regression extends linear regression for categorical target variables by employing the sigmoid function to output probabilities. This allows for binary or multiclass predictions by interpreting probabilities as class memberships.
2. Log Loss for Classification:
   • Logarithmic loss (log loss) quantifies the error in probabilistic predictions by penalizing predictions that deviate significantly from the true labels. It forms a convex function with a well-defined minimum, facilitating optimization and convergence.

---

Best Questions for Class Discussion

1. Threshold Adjustment in Logistic Regression:
   • How can adjusting the classification threshold (e.g., moving it from 0.5 to 0.2 for fraud detection) impact the balance between false positives and false negatives? What real-world examples highlight the importance of this?
2. Ethical Considerations in Categorical Variables:
   • When including sensitive variables like race or gender in logistic regression, how can we ensure the model is ethically sound and avoids discriminatory practices while leveraging potentially predictive information?