Data Science Lecture 7

Which statement best describes a binary classification model?

A model that assigns each observation to one of two possible classes.

In a binary classification problem, the response variable usually takes which form?

A variable with two possible outcomes, often coded as 0 and 1.

A logistic regression model for binary classification most directly estimates:

The probability that an observation belongs to the positive class.

Why are predicted probabilities useful in binary classification?

They allow decision makers to select different thresholds based on costs and objectives.

Which example is most naturally a binary classification problem?

Predicting whether an email is spam or not spam.

In a medical screening model, which class is often treated as the positive class?

The condition of interest, such as disease present.

A confusion matrix compares:

Predicted classes with actual classes

A true positive occurs when:

The actual class is positive and the predicted class is positive.

A false negative occurs when:

The model predicts negative for an actual positive case.

Suppose TP = 40, TN = 50, FP = 10, and FN = 20. How many observations are in the sample?

120

For an actual negative observation, which two confusion matrix outcomes are possible?

TN or FP

Changing the probability cut-off affects the confusion matrix because:

The predicted labels may change when probabilities are converted into classes.

Accuracy is defined as:

(TP + TN)/(TP + TN + FP + FN)

Sensitivity, also called recall or true positive rate, is:

TP/(TP + FN)

Specificity is:

TN/(TN + FP)

Precision, also called positive predictive value, is:

TP/(TP + FP)

Which metric is usually most directly concerned with minimizing missed positive cases?

Sensitivity

If the threshold is lowered, which result is generally expected?

More predicted positive cases.

A high false positive rate means:

Many actual negative cases are incorrectly classified as positive.

The F1-score is most useful when the analyst wants to balance:

Precision and recall.

For highly imbalanced data, why can accuracy be misleading?

A model can achieve high accuracy by predicting the majority class most of the time.

Balanced accuracy is defined as:

(Sensitivity + Specificity)/2

Suppose TP = 30 and FN = 10. What is sensitivity?

0.75

Suppose TP = 30 and FP = 20. What is precision?

0.60

An ROC curve plots:

Sensitivity against false positive rate across thresholds

The false positive rate is equal to:

1 - Specificity

A point near the upper-left corner of the ROC plot generally indicates:

High sensitivity and low false positive rate

Why is an ROC curve considered threshold-independent?

It summarizes performance over many possible thresholds.

Which classifier would have an ROC curve close to the diagonal line from (0,0) to (1,1)?

A classifier with random ranking ability.

In ROC analysis, moving the threshold from high to low usually causes:

Both TPR and FPR to increase.

A lift chart is primarily used to evaluate:

How well a model ranks observations by likelihood of the positive class.

If the top 10% of ranked customers contains 30% of all responders, the cumulative lift at 10% is:

3.0

A cumulative lift chart compares the model with:

A random selection baseline.

A model with strong ranking performance should have a cumulative lift curve that:

Rises above the random baseline, especially in early deciles

Which statement about lift is correct?

Lift is most useful when ranking and targeting cases are important

AUC is best interpreted as:

The probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case.

An AUC of 0.5 usually indicates:

Random ranking performance.

An AUC close to 1 indicates:

Strong ability to rank positive cases above negative cases.

The Gini coefficient used in binary classification is commonly computed as:

2 x AUC - 1

If AUC = 0.80, what is the Gini coefficient?

0.60

AUC is threshold-independent because:

It summarizes ranking performance across thresholds.

Log loss evaluates:

The quality of predicted probabilities for the actual class.

For binary classification, log loss is smallest when:

The model assigns high probability to the true class for each observation.

What happens when a model predicts probability near 0 for an observation whose true class is 1?

The log loss contribution is very large.

Which statement best compares log loss and accuracy?

Log loss uses probabilities, while accuracy uses predicted class labels after selecting a threshold.

Suppose y = 1 and the model predicts p-hat = 0.80. What is the log loss contribution?

-log(0.80)

Why can log loss be sensitive to outlying or unusual observations?

Because a few extremely confident wrong predictions can create very large loss values.

The Brier Score for binary classification is:

The mean squared difference between actual outcomes and predicted probabilities.

A smaller Brier Score indicates:

Better probability predictions.

Suppose y = 1 and p-hat = 0.70. What is the squared error contribution to the Brier Score?

0.09

Which statement best compares Brier Score and log loss?

Brier Score uses squared probability errors, while log loss uses logarithmic penalties

A Brier Score of 0 means:

Predicted probabilities match the binary outcomes perfectly.

A multi-class classification problem has:

Three or more possible class labels.

Macro-averaged recall is computed by:

Calculating recall separately for each class and then taking the unweighted average

Micro-averaged performance metrics are computed by:

Pooling the counts across all classes before computing the metric.

Which averaging method is usually more sensitive to poor performance on small classes?

Macro averaging

In a multi-class confusion matrix, the diagonal entries represent:

Correct classifications for each class

Why may overall accuracy be insufficient for multi-class imbalanced data?

It may be dominated by performance on the largest classes.

A predictive model is called a binary classification model when the response variable has exactly ______ possible classes.

two

In a binary classification model, the class often coded as 1 is usually called the ______ class.

positive

A logistic regression model for binary classification commonly estimates the probability that Y = ______.

1

Before computing accuracy or sensitivity, estimated probabilities must usually be converted into predicted ______.

classes

The probability value used to convert estimated probabilities into class labels is called the probability ______ or cut-off probability.

threshold

For a binary target with classes “Default” and “No Default,” predicting whether a borrower will default is a ______ classification problem.

binary

A confusion matrix compares the actual class labels with the predicted ______ labels

class

A true positive occurs when the actual class is positive and the predicted class is ______.

positive

A false positive occurs when the actual class is negative but the predicted class is ______.

positive

A false negative occurs when the actual class is positive but the predicted class is ______.

negative

If TP = 40, TN = 50, FP = 10, and FN = 5, the total sample size is ______

105

If an observation has actual class 0 and predicted probability 0.70 with p_cut = 0.50, this observation is a ______ positive.

false

Accuracy is defined as (TP + TN) divided by ______.

TP + TN + FP + FN

Sensitivity is also called recall or the true ______ rate.

positive

Specificity is also called the true ______ rate.

negative

Precision is also called the positive predictive ______.

value

The false positive rate is equal to FP divided by ______.

FP + TN

The false negative rate is equal to FN divided by ______.

FN + TP

The F1-score is the harmonic mean of precision and ______.

recall

Balanced accuracy is the average of sensitivity and ______.

specificity

For TP = 30, FN = 10, TN = 50, and FP = 10, sensitivity is ______.

0.75

For TP = 30, FP = 10, TN = 50, and FN = 10, precision is ______.

0.75

When the cut-off probability changes, threshold-dependent metrics usually ______

change

In a highly imbalanced dataset, accuracy can be misleading because it may be dominated by the ______ class.

majority

The ROC curve plots the true positive rate against the false ______ rate.

positive

The true positive rate is the same as ______.

sensitivity

The false positive rate is equal to 1 minus ______.

specificity

Each point on an ROC curve corresponds to a different probability ______.

threshold

A classifier with no discrimination ability is expected to have an ROC curve close to the ______ line.

diagonal

A better ROC curve is generally closer to the upper ______ corner.

left

A lift chart evaluates how well a model concentrates positive cases among observations with high predicted ______.

probabilities

Lift is commonly defined as the positive response rate in a selected group divided by the overall positive response ______.

rate

If the overall positive rate is 10% and the top decile positive rate is 30%, the lift for the top decile is ______.

3

A cumulative lift chart evaluates performance after accumulating observations from the highest predicted probability group downward to lower ______ groups.

probability

In marketing, lift charts are useful because they identify groups most likely to ______.

respond

A lift value of 1 indicates performance equivalent to ______ selection.

random

AUC stands for Area Under the ______ curve.

ROC

AUC can be interpreted as the probability that a randomly chosen positive case receives a higher predicted score than a randomly chosen ______ case.

negative

An AUC value of 0.5 indicates approximately ______ discrimination ability.

no

An AUC value closer to 1 indicates ______ discrimination performance.

better

The Gini coefficient for binary classification is commonly calculated as 2 × AUC minus ______.

1