Logistic Regression Notes

Logistic Regression Overview

• Logistic regression is used for predicting the probability of a binary outcome (yes/no, 1/0).

Key Concepts in Logistic Regression

• Logistic Regression Equation: The equation estimates probabilities that the target variable belongs to a particular category based on predictor variables.

• The model can be represented as:
  $P(Y = 1 | X) = \frac{e^{\beta<em>0 + \beta</em>1 x<em>1 + \beta</em>2 x<em>2 + … + \beta</em>p x<em>p}}{1 + e^{\beta</em>0 + \beta<em>1 x</em>1 + \beta<em>2 x</em>2 + … + \beta<em>p x</em>p}}$

• Each coefficient ($\beta_i$) indicates the relationship between the predictor variable and the outcome.

Interpretation of Coefficients

• A coefficient measures the change in log-odds of the outcome for a one-unit increase in the predictor variable.

• The logistic function gives probabilities between 0 and 1 for the binary outcome.

Predictions Using Logistic Regression

• Predictions can be converted into classifications:

  • Default threshold: 0.5

    • If predicted probability > 0.5, classify as 1 (yes).

    • If predicted probability < 0.5, classify as 0 (no).

• Threshold adjustments can control false positives and false negatives:

  • Higher threshold (e.g., 0.9) reduces false positives but increases false negatives.

  • Lower threshold (e.g., 0.1) increases false positives but reduces false negatives.

Performance Assessment of Logistic Regression

• Confusion Matrix: A matrix that summarizes the classification performance, showing true positive, false positive, true negative, and false negative counts.

• Performance Metrics: Evaluating the model's predictions can include:

  • Accuracy: Total correct predictions / total predictions.

  • Sensitivity (Recall): True Positives / (True Positives + False Negatives).

  • Specificity: True Negatives / (True Negatives + False Positives).

  • Precision: True Positives / (True Positives + False Positives).

• ROC Curve: A graphical representation of the true positive rate versus the false positive rate at various threshold settings.

• Maximizing classification accuracy, improving sensitivity, and minimizing false positive rates can be achieved through ROC analysis.

Model Fitting Process

• Maximum Likelihood Estimation (MLE): The method used to estimate the parameters of the logistic regression model.

• The likelihood function is defined as:
  $L(\beta) = \prod<em>{i=1}^{n} P(Y</em>i | X<em>i)^{Y</em>i} (1 - P(Y<em>i | X</em>i))^{(1 - Y_i)}$

• Taking the logarithm for numerical stability leads to:
  $\log L(\beta) = \sum<em>{i=1}^{n} \left( Y</em>i \log(P(Y<em>i | X</em>i)) + (1 - Y<em>i) log(1 - P(Y</em>i | X_i)) \right)$

• Model fitting revolves around maximizing the likelihood, or equivalently, minimizing the negative log-likelihood (this is referred to as cross-entropy loss in classification contexts).

Loss Function

• Loss function in this context is the negative log-likelihood:
  $-\log L(\beta) = -\sum<em>{i=1}^{n} \left( Y</em>i \log(P(Y<em>i | X</em>i)) + (1 - Y<em>i) log(1 - P(Y</em>i | X_i)) \right)$

• This function models penalty for incorrect predictions, guiding the optimization process for models such as neural networks.

Pseudo R-squared in Logistic Regression

• Unlike linear regression, R-squared in logistic regression does not represent variance explained. Instead, it measures improvements relative to a null model (model with no predictors).

• Defined as:
  $R^2 = 1 - \frac{\log L(\text{full model})}{\log L(\text{null model})}$

• Entropy R-squared is a specific term for this measure in logistic regression contexts.

Classification Performance Evaluation

• Lift Chart: Compares the effectiveness of a model versus a random selection.

  • Measure performance based on sorted probabilities of predicted outcomes.

  • Help determine optimal targeting strategies for a marketing campaign based on predicted outcomes.

• Lift Calculation: Shows the improvement in capturing true positive outcomes through model predictions over random guessing.

Conclusion

• This diagnostic discussion integrates various aspects of model fitting, evaluation, and interpretation critical for assessing logistic regression models in a practical setting.