73333 Module 3 Trasncript and Summaries

Logistic Regression Study Guide

Key Concepts

1. Categorical Data

• Definition: Data based on classes (e.g., red/green/blue, true/false).
• Transformation: Use one-hot encoding to convert non-numeric data into numeric format.
  • Example: Red = [1, 0, 0], Green = [0, 1, 0], Blue = [0, 0, 1].
  • Each category is represented by a binary vector.
• Discrete Data: Numeric values without fractional parts (e.g., number of rooms, children).
  • Do not one-hot encode discrete data; treat it as numeric.

2. Linear to Logistic Regression

• Linear regression predicts continuous values, but logistic regression is used for categorical targets.
• Logistic regression transforms linear outputs into probabilities using the sigmoid function.

3. The Sigmoid Function

• Formula: \(\sigma(x) = \frac{1}{1 + e^{-x}}\)
• Squeezes input values into the range (0, 1), representing probabilities.
• Default classification threshold is 0.5 but can be adjusted based on the use case (e.g., fraud detection).

4. Log Loss (Logarithmic Loss)

• Measures the distance between predicted probabilities and actual binary targets.
• Formula:
  \(\text{Log Loss} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]\)
  • \(y_i\): Actual class (0 or 1).
  • \(p_i\): Predicted probability for class 1.
• Penalizes predictions farther from the actual target.

5. Minimizing Log Loss

• Uses gradient descent to adjust weights (\(m\)) and minimize loss.
• Partial derivatives of log loss guide weight updates: \(m_{new} = m_{old} - \alpha \frac{\partial J}{\partial m}\), where \(J\) is the log loss and \(\alpha\) is the learning rate.

6. Multiclass Classification

• Extends binary logistic regression to multiple categories.
• Two approaches:
  • One-vs-All (OvA): Train a separate classifier for each class.
  • One-vs-One (OvO): Compare every pair of classes; assign the class with the most wins.
• Use libraries (e.g., sklearn) for built-in multiclass implementation.

---

Demonstration of Logistic Regression

Part I: Binary Classification (Breast Cancer Dataset)

Steps:

1. Data Preparation:
   • Import dataset from sklearn.datasets.
   • Convert data to a DataFrame and add column names.
   • Separate features (X) and targets (y).
2. Logistic Regression:
   • Use LogisticRegressionCV for cross-validation.
   • Fit the model: model.fit(X, y).
   • Retrieve model coefficients with model.coef_.
3. Cross-Validation:

   • For unbiased performance metrics, use cross_val_score.

   • Example:

     from sklearn.model_selection import cross_val_score
     scores = cross_val_score(model, X, y, scoring='accuracy', cv=5)
     print("Accuracy:", scores.mean())

Part II: Cross-Validation and Regularization

• Use LogisticRegressionCV to optimize regularization parameters.
• Interpret the results and evaluate metrics.

---

Case Study: Hospital Readmission Prediction

Problem Overview

• Predict patient readmission within 30 days using logistic regression.
• Challenges:
  • Missing data must be imputed.
  • Ethical considerations in using sensitive features (e.g., race).
  • Imbalanced classes.

Assignment Steps:

1. Data Preprocessing:
   • Handle missing data using imputation techniques (e.g., mean, median).
   • Normalize/scale features for better model performance.
2. Model Development:
   • Build a logistic regression model for each target class (multiclass setup).
   • Use cross-validation to evaluate performance.
3. Feature Importance:
   • Analyze the top 5 important features contributing to predictions using model coefficients.

---

Mathematical and Coding Representations

Mathematical Representation

1. Sigmoid Function:
   \(\sigma(x) = \frac{1}{1 + e^{-x}}\)
2. Log Loss:
   \(J = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]\)
3. Gradient Update:
   \(m_{new} = m_{old} - \alpha \frac{\partial J}{\partial m}\)

Python Code Representation

1. One-Hot Encoding:

   import pandas as pd
   from sklearn.preprocessing import OneHotEncoder
   
   data = {'Color': ['Red', 'Blue', 'Green']}
   df = pd.DataFrame(data)
   encoder = OneHotEncoder()
   encoded = encoder.fit_transform(df[['Color']]).toarray()
   print(encoded)

2. Logistic Regression:

   from sklearn.datasets import load_breast_cancer
   from sklearn.model_selection import train_test_split
   from sklearn.linear_model import LogisticRegressionCV
   
   # Load data
   data = load_breast_cancer()
   X, y = data.data, data.target
   
   # Split data
   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
   
   # Logistic Regression with CV
   model = LogisticRegressionCV(cv=5, max_iter=1000).fit(X_train, y_train)
   print("Accuracy:", model.score(X_test, y_test))

3. Cross-Validation:

   from sklearn.model_selection import cross_val_score
   
   scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
   print("Cross-Validation Accuracy:", scores.mean())

Visualization of Sigmoid Function

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-10, 10, 100)
sigmoid = 1 / (1 + np.exp(-x))

plt.plot(x, sigmoid)
plt.title('Sigmoid Function')
plt.xlabel('x')
plt.ylabel('Sigmoid(x)')
plt.grid()
plt.show()

Study Guide: Logistic Regression with Mathematical and Coding Representations

---

1. Categorical Data

• Definition: Data representing classes or categories (e.g., red, green, blue; true/false).
• Challenges: Categorical data must be transformed into numerical data to be used in most machine learning models.
• Transformation:
  • One-Hot Encoding: Converts categorical features into binary columns, where each unique value becomes a column. E.g., for colors red, green, blue:
    • Red: [1, 0, 0]
    • Green: [0, 1, 0]
    • Blue: [0, 0, 1]

Python Example:

import pandas as pd

data = {'Color': ['Red', 'Green', 'Blue']}
df = pd.DataFrame(data)
df_encoded = pd.get_dummies(df, columns=['Color'])

Mathematical Representation: If \(C\) represents the categories and \(x\) is the input, the transformation is: \[ \text{One-hot encoded vector: } \mathbf{x}_{\text{encoded}} = [x_1, x_2, \ldots, x_n], \text{where } x_i = 1 \text{ if } x = C_i, \text{ else } 0. \]

---

2. Linear to Logistic Regression

• Linear regression predicts continuous values. Logistic regression transforms this to predict probabilities for binary outcomes.
• Key Equation: \[ z = \mathbf{w}^T\mathbf{x} + b, \quad p = \sigma(z), \quad \sigma(z) = \frac{1}{1 + e^{-z}} \] Where \(\sigma(z)\) (sigmoid function) squashes \(z\) to range [0, 1].

Python Example:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train, y_train)

---

3. The Sigmoid Function

• Equation: \(\sigma(z) = \frac{1}{1 + e^{-z}}\)
• Properties:
  • \(z \to +\infty\): \(\sigma(z) \to 1\)
  • \(z \to -\infty\): \(\sigma(z) \to 0\)
  • Outputs probabilities.

Python Example:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-10, 10, 100)
sigmoid_values = sigmoid(z)

---

4. Log Loss

• Measures the distance between predicted probabilities and actual class labels.
• Equation: \[ \text{Log Loss: } L(y, \hat{y}) = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right] \] Where:
  • \(y_i\): Actual label.
  • \(\hat{y}_i\): Predicted probability.

Python Example:

from sklearn.metrics import log_loss
loss = log_loss(y_true, y_pred)

---

5. Minimizing Log Loss

• Achieved through optimization (e.g., gradient descent).
• Gradient Descent:
  • Update rule: \(w = w - \alpha \nabla L(w)\), where \(\alpha\) is the learning rate.
• Python Example:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=100)
model.fit(X_train, y_train)

---

6. Multiclass Classification

• Extends binary logistic regression to multiple classes.
• Approaches:
  • One-vs-Rest (OvR): Train separate classifiers for each class.
  • Softmax Regression: Directly models all classes using probabilities.
• Softmax Function: \[ \sigma(z_j) = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}} \]

Python Example:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(multi_class='multinomial', solver='lbfgs')
model.fit(X_train, y_train)

---

Case Study: Predicting Diabetes Readmission

Objective

• Predict hospital readmission within 30 days using logistic regression.
• Handle missing data via imputation.

Steps:

1. Data Preprocessing:
   • Handle missing values (e.g., mean/mode imputation).
   • Encode categorical variables (e.g., one-hot encoding).
2. Model Building:
   • Train logistic regression for three classes of readmission:
     • No readmission.
     • Readmission in less than 30 days.
     • Readmission in more than 30 days.
   • Evaluate using metrics like log loss or accuracy.
3. Feature Importance:
   • Extract coefficients to identify top 5 predictive features.

Python Code:

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

# Data preprocessing
imputer = SimpleImputer(strategy='mean')
X = imputer.fit_transform(X)

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Logistic regression model
model = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=500)
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)

# Feature importance
importance = model.coef_
print("Top 5 Features:", importance.argsort()[-5:])

# Accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy}")

Variable Importance Interpretation

• Coefficients reflect the importance of each feature.
• Analyze top variables to derive insights for readmission patterns.

---

Assignment

• Train and evaluate logistic regression on the diabetes dataset.
• Report top 5 features and their significance.
• Submit results with a clear explanation of findings.

Deliverable: - Code file (e.g., FirstName_LastName_LogReg_Assignment.py). - Written report on model performance and feature importance.

Key Takeaways

1. Logistic Regression Bridges Linear Models and Probabilistic Predictions:

   • Logistic regression extends linear regression for categorical target variables by employing the sigmoid function to output probabilities. This allows for binary or multiclass predictions by interpreting probabilities as class memberships.

2. Log Loss as a Metric:

   • Unlike linear regression’s mean squared error, logistic regression uses log loss to measure the model’s performance. It penalizes predictions further from the true class, ensuring probabilities are accurate.

3. Handling Multiclass Problems:

   • Multiclass classification can be addressed using methods like one-vs-rest (OvR) or softmax regression. While OvR is computationally efficient for a few classes, it scales poorly with many classes due to class imbalance issues.

4. Importance of Sigmoid Function in Logistic Regression: The sigmoid function transforms the linear regression output into probabilities, making it possible to handle binary classification tasks effectively. This function ensures that predictions fall within the range [0, 1], which can then be used to classify data into distinct categories.

5. Log Loss for Classification: Logarithmic loss (log loss) quantifies the error in probabilistic predictions by penalizing predictions that deviate significantly from the true labels. It forms a convex function with a well-defined minimum, facilitating optimization and convergence.

6. Multiclass Classification Challenges: Logistic regression can be extended to multiclass problems through techniques like One-vs-All and One-vs-One. While these methods allow logistic regression to handle more than two classes, they come with scalability and bias challenges as the number of classes increases.

---

Questions for Class Discussion

1. Threshold Adjustment in Logistic Regression:

   • How can adjusting the classification threshold (e.g., moving it from 0.5 to 0.2 for fraud detection) impact the balance between false positives and false negatives? What real-world examples highlight the importance of this?

2. Log Loss vs. Accuracy:

   • In what scenarios might log loss be a more appropriate evaluation metric than accuracy? Can you provide examples where accuracy might be misleading?

3. Ethical Considerations in Categorical Variables:

   • When including sensitive variables like race or gender in logistic regression, how can we ensure the model is ethically sound and avoids discriminatory practices while leveraging potentially predictive information?

4. On Sigmoid Threshold Adjustment: In real-world applications like fraud detection, how do we decide on the optimal threshold for classification beyond the default value of 0.5? What strategies or metrics should guide this decision?

5. On Handling Missing Data: When dealing with missing values in a dataset, as mentioned in the diabetes case study, what factors should influence the choice of an imputation strategy? How do we ensure the imputation does not introduce bias into the model?

6. On Ethical Considerations in Feature Selection: In cases where sensitive features like race are included in the dataset, how do we balance the potential utility of such features with ethical concerns and the risk of perpetuating bias in predictions?

Best Takeaways

1. Logistic Regression Bridges Linear Models and Probabilistic Predictions:
   • Logistic regression extends linear regression for categorical target variables by employing the sigmoid function to output probabilities. This allows for binary or multiclass predictions by interpreting probabilities as class memberships.
2. Log Loss for Classification:
   • Logarithmic loss (log loss) quantifies the error in probabilistic predictions by penalizing predictions that deviate significantly from the true labels. It forms a convex function with a well-defined minimum, facilitating optimization and convergence.

---

Best Questions for Class Discussion

1. Threshold Adjustment in Logistic Regression:
   • How can adjusting the classification threshold (e.g., moving it from 0.5 to 0.2 for fraud detection) impact the balance between false positives and false negatives? What real-world examples highlight the importance of this?
2. Ethical Considerations in Categorical Variables:
   • When including sensitive variables like race or gender in logistic regression, how can we ensure the model is ethically sound and avoids discriminatory practices while leveraging potentially predictive information?
3. The correct answer is:

Discrete data has specific values that can be numeric, while categorical should always be one hot encoded.

Explanation:

• Categorical Data: Represents categories or classes without an inherent order (e.g., colors, types of animals). It must be one-hot encoded for numerical representation in most machine learning models.
• Discrete Data: Represents countable, distinct values that are numeric in nature (e.g., number of rooms in a house, number of cars a family owns). Discrete data typically does not require one-hot encoding and can often be left as is.

---

Why Not the Other Choices?

1. “Only discrete data should be one-hot encoded”: Incorrect. Discrete data is usually numeric and does not need one-hot encoding, while categorical data typically does.

2. “Categorical data cannot be used in regression”: Incorrect. Categorical data can be used in regression after proper encoding.

3. “Categorical data can have multiple values, while discrete data only have two”: Incorrect. Both categorical and discrete data can have multiple values.

The correct answer is:

Discrete data has specific values that can be numeric, while categorical should always be one hot encoded.

Explanation:

• Categorical data: Represents labels or classes that do not have an inherent numeric order (e.g., “red”, “blue”, “green”). These must be one-hot encoded to be used in machine learning models.
• Discrete data: Represents numeric values that take on specific, distinct numbers (e.g., the number of rooms, cars owned). Discrete data does not need one-hot encoding because the numeric values are meaningful and preserve order.

2. The correct answer is:

Is the same as linear regression

Explanation:

• Slope Update Rule: The slope update rule in logistic regression is mathematically similar to linear regression. The key difference lies in how the error is calculated due to the use of the sigmoid function and log loss. However, the gradient descent procedure (used to minimize the loss function) remains the same.

---

Why Not the Other Choices?

1. “Involves the sigmoid”: While the sigmoid function is integral to logistic regression, it is part of the prediction mechanism, not the slope update rule itself.

2. “Uses cross-entropy loss”: Logistic regression can be associated with cross-entropy loss in multiclass classification, but this isn’t specific to the slope update rule.

3. “Uses the log loss”: Log loss is the cost function minimized in logistic regression, but it is not the slope update rule itself. The rule is derived using the gradient of the log loss.

The correct answer is:

Is the same as linear regression

Explanation:

• The slope update rule for logistic regression is mathematically the same as in linear regression because it is derived using gradient descent. The update rule adjusts weights (slopes) iteratively to minimize the loss function.

Why the other options are incorrect:

1. Involves the sigmoid: The sigmoid function is used to transform outputs into probabilities, but it is not directly part of the slope update rule.

2. Uses cross-entropy loss: While cross-entropy loss (log loss) is the loss function for logistic regression, it is separate from the actual slope update rule.

3. Uses the log loss: Log loss is minimized during training, but the slope update rule itself remains identical to that in linear regression.

4. The correct answer is:

Can produce negative outputs

---

Explanation:

• Outputs always between 0 and 1: True. The sigmoid function \(\sigma(z) = \frac{1}{1 + e^{-z}}\) always produces outputs in the range \((0, 1)\).
  
• Can take negative inputs: True. The sigmoid function accepts any real number \(z\), including negative values.
  
• A simulated step function: True. For very large positive or negative inputs, the sigmoid approximates a step function with outputs close to 1 or 0, respectively.
  
• Can produce negative outputs: False. By definition, the sigmoid function only outputs values between 0 and 1, so it cannot produce negative outputs.

The correct answer is:

Can produce negative outputs

Explanation:

The sigmoid function has the following properties: 1. Outputs always between 0 and 1: The sigmoid function maps all inputs to values within this range. 2. Can take negative inputs: The function accepts any real number as input, including negative values. 3. A simulated step function: For very large positive or negative inputs, the sigmoid function behaves like a step function.

However: - It cannot produce negative outputs, as its range is strictly between 0 and 1.

4. The correct answer is:

Log loss has two terms.

---

Explanation:

1. Log loss:
   • Used in classification tasks, especially for logistic regression.
     
   • It is defined as:
     \[ \text{Log Loss: } -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right] \]
   • The formula has two terms, one for \(y_i = 1\) and one for \(y_i = 0\), to handle the binary nature of the classification problem.
2. Mean Squared Error (MSE) and Mean Absolute Error (MAE):
   • Used in regression tasks for continuous outputs.
   • Both have a single term that measures the difference between the predicted value and the actual value:
     • MSE: \(\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2\)
       
     • MAE: \(\frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|\)

---

Why Not the Other Choices?

1. Log loss is not continuous: False. Log loss is a continuous function of its inputs.
   
2. Log loss is not convex: False. Log loss is a convex function, which ensures optimization algorithms converge to a global minimum.
   
3. Log loss is not differentiable: False. Log loss is differentiable, and gradient descent relies on this property for optimization.

The correct answer is:

Log loss has two terms.

Explanation:

1. Log loss:
   • Specifically designed for classification problems, particularly binary classification.
   • It has two terms, one for when the actual class is 1 and another for when the actual class is 0.
   • These terms correspond to the negative log of predicted probabilities for the correct class.
2. Why the other options are incorrect:
   • Log loss is not continuous: Incorrect. Log loss is a continuous function.
   • Log loss is not convex: Incorrect. Log loss is convex, which ensures that gradient descent can find the global minimum.
   • Log loss is not differentiable: Incorrect. Log loss is differentiable, allowing for optimization using gradient descent.

Key Difference with MSE/MAE:

• Mean squared error (MSE) and mean absolute error (MAE) measure the distance between predictions and targets for regression tasks.
• Log loss measures the probability assigned to the correct class for classification tasks. It penalizes incorrect predictions more heavily as the predicted probability moves further away from the true label.

5. The correct answer is:

We can use the one vs. rest method.

---

Explanation:

1. One-vs-Rest Method:
   • In multiclass classification with logistic regression, the one-vs-rest (OvR) approach trains a separate binary classifier for each class.
     
   • For a class \(C_i\), the model predicts whether a data point belongs to \(C_i\) (1) or not (0).
     
   • During inference, the model chooses the class with the highest probability.
2. One-vs-One Method:
   • While the one-vs-one (OvO) method is used in some classification tasks, it is not directly associated with logistic regression using the sigmoid function. OvO involves creating a binary classifier for each pair of classes, which can become computationally expensive as the number of classes increases.
3. Multiclass is impossible with log loss:
   • This is false. Multiclass logistic regression can use a generalization of log loss (e.g., softmax cross-entropy) to handle multiple classes directly.

---

Multiclass with Logistic Regression:

For a direct multiclass approach (e.g., softmax regression):
- Use the softmax function instead of the sigmoid function, which generalizes probability assignment to \(K\) classes.
- The log loss in this case is adapted for multiclass problems.

The correct answer is:

We can use the one vs. one method.
We can use the one vs. rest method.

Explanation:

1. One-vs-Rest (One-vs-All):
   • For each class, a separate classifier is trained to distinguish that class from all other classes.
   • The class with the highest probability is selected as the predicted class.
   • Example: For classes Red, Green, and Blue:
     • Red vs. Not Red
     • Green vs. Not Green
     • Blue vs. Not Blue
2. One-vs-One:
   • A classifier is trained for every pair of classes.
   • Each pairwise comparison predicts a winner, and the class with the most wins is selected.
   • Example: For classes Red, Green, and Blue:
     • Red vs. Green
     • Red vs. Blue
     • Green vs. Blue
3. Why “Multiclass is impossible with log loss” is incorrect:
   • Log loss can be extended to multiclass problems using softmax instead of sigmoid. This is common in frameworks like neural networks.

Summary:

Multiclass problems can be handled with sigmoid-based methods like One-vs-Rest or One-vs-One, although other techniques (e.g., softmax) are often more efficient for larger class sizes.