CS3001 - Section 3 - Confusion Matrix & Evaluation

Use HIGH SENSITIVITY when...

Missing a real case is dangerous (e.g. disease screening)

False positives are acceptable — they lead to more tests

Use HIGH SPECIFICITY when...

A false alarm causes harm (e.g. selecting patients for risky surgery)

False positives are NOT acceptable — they cause unnecessary harm

Bootstrapping

Sample WITH REPLACEMENT to build training set

All unselected data (~36.8%) becomes test set

Repeat many times and average results

Slightly pessimistic (conservative) estimate

Cross-Validation (k-fold)

Split data into K equal folds

Train on K-1 folds, test on 1 fold

Rotate K times, average all test results

Unbiased estimate — every point tested once

Both methods

Estimate the TRUE error rate, NEVER evaluate on training data — the model already 'knows' it!

Predicted Positive and Actual Positive equal...

True Positive (TP)

Predicted Negative and Actual Positive equal..

False Negative (FN)

Predicted Positive and Actual Negative equal...

False Positive (FP)

Predicted Negative and Actual Negative equal...

True Negative (TN)