Machine Learning Model Evaluation: Performance & Fairness Metrics

Purpose Performance metrics

Measure how well the model predicts

Purpose Fairness Metrics

Measure how fairly the model treats different groups

Focus Performance metrics

Overall accuracy, precision, recall, error, etc.

Focus Fairness Metrics

Equality of performance or outcomes across sensitive groups (e.g., gender, race, age).

Question Asked Performance Metrics

"Is my model good?

Question Asked Fairness Metrics

"Is my model fair?"

Level of Analysis Performance Metrics

Global

Level of Analysis Fairness Metrics

Group-level

Example Performance Metrics

Accuracy, F1-score, RMSE, AUC, R²

Example Fairness Metrics

Demographic Parity, Equalized Odds, Mean Residual Difference

what is performance metrics used in?

Model evaluation/tuning

what is Fairness Metrics used in?

Ethical analysis, bias detection, compliance

Mean Residual Difference (MRD)

Average difference in errors between groups (should be close to 0)

R² by group

Checks whether the model fits one group better than another.

Statistical Parity in Predictions

Compares average predicted values between groups.

Group Mean Error/RMSE

Compares average prediction error across groups.

Demographic Parity

Equal probability of positive prediction across groups.

Equal Opportunity:

Equal True Positive Rate (TPR) across groups.

Equalized Odds

Equal TPR and FPR across groups.

Predictive Parity

Equal Positive Predictive Value (precision) across groups.

Calibration

Predictions reflect equal probabilities for all groups.