1/22
Looks like no tags are added yet.
what does ‘fair’ even mean
“similar treatment of similar people”
2 dimensies van fairness met confsion matrix
equal representation : proportional selection across groups, independent of true class
equal opportunity : equally accurate in identifying true positives across groups
fairness metrics : demographic parity
(TP + FP) /N1 = (TP2 + FR2) / N2
equal representation across groups → as many (proportionally) men as women selected, independently of their ‘real’ class membership

metrics : equal opportunity
TP1 /( TP1 + FN1) = TP2/ (TP2+FN2)
equally accurate in identifying true positives across groups (percentage of identified true positives)
equal opportunity for qualified individuals, regardless of the group

external impact : mitigation strategies : fairness through unawareness
just remove the protected features A from the data and everything is fine right ?
BUT : the feature X still contain a lot of information about A
when can we intervene to engineer fairness
pre-processing
in-procesing (optimization)
post-processing
external impact : mitigation strategies : pre-processing
from sepcific transformatioons on the features X and/ or outcome Y with the aim to remove bias from the training data
the most flexible part of the data science, as it makes no assumptions with respect to the choice of subsequently applied modeling techniques
→ eg reweighting, re sampling, relabelling and perturbations, adversial learning
what does (external impact) : mitigation strategies : pre-processing requires
access to the training data
external impact : mitigation strategies : in-processing
incorporating one or more fairness notions into the model optimization functions
the joint optimization of performance and fairness may lead to the best result on multiple metrics
eg : constraint optimization, regularization, adversial learning, bandits
what does (external impact ): mitigation strategies : in-processing requires
access to the training data and model
external impact : mitigation strategies : post-processing
sometimes you deal wit black-boc scenarios and only have access to the predictions y^and your own input X, including the sensitive attribute A information
apply transformtaions to model output Y^ to improve prediction fairness with resoect to A
But it may worsen the performance of the original model
eg thresholding, calibrations, transormation, constraint optimization
what types of bias might occur during modeling
historical bias
representation bias
measurement bias
learning bias
aggregation bias
evaluation bias
deployment bias
Why are LLMs stochastic
bc the base model predicts word by word autoregressively until it hits a < stop> token
the model predicts each token with a certain probability therefore LLMS are stochastic
How LLMs adapt : what is adaptation
the process of specializing a general language model for a specific task
two ways of adapting LLMs
finetuning : updating the model’s parameters with a small, focused dataset
prompting (in context learning) : guiding the model by providing examples within the input prompt itself
LLM adaptation : by fine-tuning
starting with the langauge model and (partly) updating its parameters based on a limited set of training instances
LLMs concepts : adaptation : prompting
in context learning : varying the input prompt to achieve better modleing results
eg giving examples of a task in the prompt
defining dangers
hallucinantions
alignment
defining dangers : hallucinations
a confidence response by an AI that does not seem to be justified by its training data
defining dangers : alignment
behavior in accordance with what a human wants
alignment strategies :
supervised finetuning
RLHF
DPO : direct preferences optimization
dangers of LLMs
reliability + unexplainable
social bias
Locked-in/ monopoly forming
security
disingformation
legal considerations
cost and environmental impact
LLMs in businesses of the futiure
LLms will be used for various tasks, extent and impact highly uncertain
made available through API-based subscriptions, owned and sustained by large corporations
end-users : fine-tuning, prompt engineering, chaining of tools, agentic AI, human in the loop
the majority of projects will fail
agentic AI
is a system typically prowered by a language model, that can use tools to complete multi-step tasks