Fairness Metrics vs. Performance Metrics

what does Mean Residual Difference measure?

Average difference in residuals (errors) between protected and unprotected groups.

example of Mean Residual Difference

Ideally ≈ 0 → both groups have similar prediction errors.

what does R² by Group measure

Checks whether the model fits one group better than another.

what does Statistical Parity in Predictions mean?

Compares average predicted values between groups.

example of R² by Group

Lower R² for one group → possible bias in performance.

example of Statistical Parity in Predictions

Mean(ŷ₁) ≈ Mean(ŷ₂) → fair outcomes.

what does Group Mean Error / RMSE measure

Compares average prediction error across groups.

Example of Group Mean Error / RMSE

Identifies whether one group consistently gets higher or lower predictions.

what does Demographic Parity measure

Equal probability of positive prediction across groups.

example of demographic parity measure

P(ŷ=1

what does equal opportunity measure

Equal True Positive Rate (TPR) across groups.

example of equal opportunity

P(ŷ=1

what does equalized odds measure

Equal TPR and FPR across groups.

example of equalized odds

Ensures fairness in both correct and incorrect predictions.

what does Predictive Parity measure

Equal Positive Predictive Value (precision) across groups.

what does calibration measure

Predictions reflect equal probabilities for all groups.

example of calibration

For same predicted score, actual outcome rates should match.

linear regression fairness metrics

mean residual difference, r squared by group, statistical parity in predictions, group mean error/RMSE

logarithmic regression fairness metrics

demographic parity, equal opportunity, equalized odds, predictive parity, calibration