In Depth Notes on Logistic Regression Analysis

Logistic Regression Analysis

Introduction to Logistic Regression

• Logistic Regression is a supervised machine learning algorithm used for binary classification, determining the relationship between a binary dependent variable and independent variables.
• Goal: Identify a well-fitting model that describes the relationship between a binary dependent variable (Y) and a set of independent variables.
  • Y = 1 (event/success) or Y = 0 (no event/failure)

Basic Concepts

• Dependent Variable: A binary outcome (e.g., success/failure).
• Independent Variables: Variables that explain the dependent variable.
• Example Scenario: Examine the relationship between device type and blood pressure measures to classify whether a reading is from a GP or a home device.

Logistic Regression Basics

• Key Ideas:
  • Probability: Instead of predicting 0 or 1, we predict the probability of an event happening (e.g., P(Y=1)).
  • Odds: Odds indicate the likelihood of an event occurring as a function of probability.
  • Computation: Odds (O) = P/(1 - P)
  • Log Odds (Logit): To remove boundaries from probability (0-1) to (-∞, +∞), we use log odds:
  • Logit(P) = log(P/(1-P))

Modeling with Logistic Regression

• Logistic Regression Equation:
  • logit(P(X)) = β0 + β1x1 + β2x2 + … + βpxp
  • P(X) = e^(β0 + β1x1 + β2x2 + … + βpxp) / (1 + e^(β0 + β1x1 + β2x2 + … + βpxp))
• Max Likelihood Estimation (MLE): Used to estimate parameters. It finds the values of parameters that maximize the likelihood of the observed data.
• Log-Likelihood Function: Logarithm of the likelihood function simplifies the estimation task, allowing optimization of parameters.

Interpretation of Outcomes

• Coefficients (β):
  • β0: Log odds of the baseline event.
  • βi: Log odds ratio for each independent variable.
  • e^(βi): Odds ratio indicating how changes in the independent variable influence the odds of the outcome.
• Example: For a one-week increase in gestational age, the increase in the odds of normal birth weight could be expressed through the coefficient (β).

Goodness of Fit in Logistic Regression

• Goodness-of-Fit Tests: Assess how well models predict outcomes compared to observed data. Includes:
  • Chi-Square Tests, Deviance, Hosmer-Lemeshow Tests.
• Pseudo R-Squared: Measures how much the logistic model explains variability in the data, such as McFadden's R².
• Classification Performance Metrics:
  • Accuracy, Sensitivity, Specificity, Precision, F1 Score, and ROC Curve analysis for evaluating model performance.

Conclusion and Further Reading

• Logistic regression is a powerful tool for predicting binary outcomes. It balances interpretability and predictive ability effectively.
• Recommended Reading: Washington et al. (2020). Statistical and Econometric Methods for Transportation Data Analysis.
• Final Note: Understanding logistic regression involves conceptualizing how probability transforms into odds and log odds, making it a robust method for classification tasks.