Data Science Quiz 2

Mean Residual Difference

Average difference in residuals (errors) between protected and unprotected groups. Ideally=0.

R² by Group

Check whether the model fits one group better than another. Lower R² for a group = possible bias in performance

Statistical Parity in Predictions

Compares average predicted values by groups. Means need to be equal.

Group Mean Error/RMSE

Compares average prediction errors. Identifies whether one group gets consistently lower/higher predictions. 

Demographic Parity

Equal probability of positive prediction across groups.

Equal Opportunity

Equal True Positive Rate (TPR) across groups.

Equalized Odds

Equal TPR and FPR across groups.

Predictive Parity

Equal Positive Predictive Value (precision) across groups.

Calibration

Predictions reflect equal probabilities for all groups. For same predicted score, actual outcome rates should match. No one from one group should have a lower chance with an equal score.

Demographic Groups

Categories of people defined by sensitive attributes/variables in the dataset.

Sensitive Attributes

The attributes related to fairness and equity. Any attributes that could be a basis for bias or discrimination.

What happens when we measure a fairness metric?

We compare the models performance outcomes across different demographic groups. 

Performance metrics: purpose, focus, question, level, used in

Measure how well model predicts, overall accuracy (etc.), Is model GOOD, global, model evaluation and training.

Fairness Metrics

Measure how fair model treats groups, equality of performance across sensitive groups, is model fair, group-level, ethical analysis bias detection and compliance.

SHAP

Shapley Additive Explanations

LIME

Local Interpretable Model-Agnostic Explanations

What are SHAP and LIME

Explainability tools, evaluation category of model interpretability.

Why do we care about local explanation/how the model reasons?

Fairness/Accountability (explain for one person), Identify anomalies (see if there are outliers where the model is behaving differently), Personalization/User Experience (Give guidance to one person), Regulatory/Ethical (rules for explaining down to individual)

Trust, debugging bias, Regulatory transparency