Logistic Regression

Definition and Overview

Logistic Regression is a supervised machine learning algorithm predominantly utilized for classification problems. It serves to predict the probability that a given input belongs to a specific class. A key feature of logistic regression is its ability to model the probability of a categorical dependent variable based on one or more predictor variables.

Types of Logistic Regression

There are three primary types of logistic regression, each suitable for different scenarios based on the nature of the dependent variable (𝑌):

1. Binomial Logistic Regression

Characteristics: The dependent variable (𝑌) is binary, having only two possible categories.
Examples of categories:
- Yes/No
- True/False
- Pass/Fail

2. Multinomial Logistic Regression

Characteristics: The dependent variable (𝑌) has three or more categories that are not ordered.
Examples of categories:
- Animals: Cat, Dog, Sheep
- Colors: Brown, Red, Blue
- Names: Bryant, Coleman, PM West

3. Ordinal Logistic Regression

Characteristics: The dependent variable (𝑌) has three or more categories that have a natural order.
Examples of categories:
- Satisfaction levels: High, Medium, Low
- Feedback: Satisfied, Neutral, Unsatisfied, Very Unsatisfied

Focus in AQM 2000

In this course, we will concentrate specifically on Binomial Logistic Regression. This involves predicting the likelihood of a customer responding to a marketing campaign based on various factors such as:

Age
Income
Prior purchase frequency
Website visits
Loyalty status

Real-World Examples

Predicting whether a loan applicant will default based on variables such as:
- Credit score
- Income
- Loan amount
- Employment length
- Debt-to-income ratio
Detecting fraudulent transactions based on factors like:
- Transaction amount
- Time of day
- Merchant category
- Cardholder location
- Number of prior attempts
Identifying whether a neighborhood in Boston has a high median value based on the average number of rooms (RM):
- Probability of ISHIGHVAL

The Limitations of Linear Regression

Linear regression is ineffective for these types of classification problems. This is primarily due to its assumption of a linear relationship between the dependent and independent variables, which does not hold true for binary outcomes.

Deriving Logistic Regression

Understanding Odds

The odds of an event occurring can be defined mathematically. For instance, the probability of flipping a coin and obtaining heads is 𝑝 = 0.5. Hence, the odds can be calculated as:
$\text{odds} = \frac{p}{1 - p} = \frac{0.5}{0.5} = 1$
In a group of 4 Babson students, 1 is a freshman, 1 is a sophomore, 1 is a junior, and 1 is a senior. The probability of randomly selecting a senior is 𝑝 = 0.25. Thus, the odds of picking a senior are given by:
$\text{odds} = \frac{0.25}{1 - 0.25} = \frac{0.25}{0.75} = \frac{1}{3}$
If an event occurs with probability 𝑝 = 0.8, the corresponding odds are:
$\text{odds} = \frac{0.8}{1-0.8} = \frac{0.8}{0.2} = 4$
The formula to generalize odds for any event can be expressed as:
$\text{odds} = \frac{p}{1 - p}$

Logistic Regression Model Structure

To convert raw data into probabilities and subsequently odds, we use the relationship defined by the odds calculation from the previous section. The odds can be modeled in logistic regression as:
$\text{odds} \approx e^{\beta<em>0 + \beta</em>1 \text{RM}}$
Logistic regression models the log of the odds as a linear function, allowing us to apply understandings from linear regression. The relationship is expressed as:
$\log\left(\frac{p(\text{RM})}{1 - p(\text{RM})}\right) = \beta<em>0 + \beta</em>1 \text{RM}$
This indicates:

If 𝛽1 > 0, as RM increases, the odds increase.
If 𝛽1 < 0, as RM increases, the odds decrease.
If 𝛽1 = 0, the odds remain unchanged with an increase in RM.

Analyzing Changes in Odds

To analyze how the odds change with a change in RM, we can express:
$\text{Odds at RM+1} = \text{Odds at RM} \times \text{Odds Ratio}$
Thus, if RM increases by 1 unit, the odds change multiplicatively based on the model. For instance, in a univariate logistic model fitted to data:
$\log(\text{odds of ISHIGHVAL}) = 0.1 \times \text{RM}$
This can be interpreted to assess how each additional room affects the odds of the neighborhood being classified as a high median value neighborhood (ISHIGHVAL).

Implementing Logistic Regression in R

In R, logistic regression can be executed using the glm() function, which is included in base R, meaning no additional library or package is required. It is important to use family = binomial as a parameter to specify the nature of the outcome variable. The output from this function resembles that of linear regression.

Key Points for Implementation:

The model estimates log(odds) of the outcome, so each coefficient (𝛽-value) needs exponentiation to interpret its impact on odds.
The type = "response" argument signals R to return predictions in terms of the outcome category (TRUE/FALSE).

Choosing the Cutoff Threshold

Choosing the cutoff threshold for predictions is crucial and debated. The cutoff defines how predicted probabilities are translated into binary outcomes, with various strategies based on the specific application.

Evaluating Logistic Regressions

Sensitivity and Specificity

Sensitivity:
The proportion of actual positives (TRUE outcomes) correctly identified by the model.
Specificity:
The proportion of actual negatives (FALSE outcomes) correctly identified by the model.
Typically, for any given modeling issue, either sensitivity or specificity is often deemed more critical depending on the context.

Impact of Changing Cutoff Value

Modifying the cutoff from 0.5 to 0 leads to predicting all instances as TRUE, resulting in:

Sensitivity = 1
Specificity = 0
Conversely, increasing the cutoff to 1 results in predicting all instances as FALSE, yielding:
Sensitivity = 0
Specificity = 1
This adjustment illustrates that shifting the cutoff from 0 to 1 inversely affects sensitivity and specificity.

Receiver Operating Characteristic (ROC) Charts

ROC charts are utilized to visualize model performance across various cutoff values ranging from 0 to 1.
The ideal scenario is achieving sensitivity and specificity both equal to 1, represented by the upper right point in the chart.
Each point in the ROC chart reflects the model's sensitivity and specificity for the selected cutoff.
The area under the ROC curve (AUC-ROC) is a vital metric; a model should aim for a larger AUC, indicating better performance.

Lift Charts

An alternative visualization for evaluating logistic regression performance involves Lift Charts:

The predictions are sorted from highest to lowest, aiming for true observations nearer to 1 and false observations nearer to 0.
The Lift Chart compares cumulative observed TRUE counts versus a baseline performance of a random model, thereby evaluating how well the model differentiates between outcomes.

ROC Chart vs. Lift Charts

Comparison

ROC Chart:
- Main Goal: Assess overall model accuracy.
- Main Use: Diagnosing and comparing potential models.
- Requires Cutoff Value: No.
- Typical Metric: Area Under the Curve (AUC).
- Queries Answered: "Which model classifies better?"
Lift Chart:
- Main Goal: Evaluate performance of TRUE vs. FALSE predictions.
- Main Use: Ranking prediction results for marketing targeting and prioritization.
- Requires Cutoff Value: No fixed cutoff is used, but it informs decision on cutoff.
- Typical Metric: Lift at top decile, % gain.
- Queries Answered: "How many TRUE/FALSE outcomes do we reach if we target the top/bottom x% in our data?"

Sensitivity and Specificity Calculations in R

Both metrics can be implemented in R to quantitatively measure the performance of logistic regression models. The functions necessary for ROC Charts and Lift Charts can be sourced from the BabsonAnalytics.R script, to enable comparison of model predictions against testing data effectively.