Ethics Quiz 16

Gaming the system

Clickbait thumbnails, hooks in the first three seconds, Uber somewhere nearby to avoid surge pricing. Hidden resume text → once we know how an algorithm works, we adapt.

US News and World Report College Ranking

Feedback loop where the higher ranked schools get more applicants and can select the best ones, to further increase their performance.

Arms race: larger endowments, which aren’t actually related to education quality but can improve other categories.

How do we measure changes in the model? What is the outcome, when do we say it’s no longer good enough?

How do we improve it once we know it changed? Is it retaining with updated dataset seasonally. Was it ever measuring the right thing?

Construct validity

A construct: something you want to measure that cannot be directly measured.

Fed by indicators, latent variables, proxies. Something you can measure, but we have issues when the proxies don’t relate to the construct.

Construct drift

The construct itself has changed over time. Ex. engagement in videos.

Model drift

Data drift: input features change but the relationship stays the same.

Concept drift: the relationship between features and target shift. SAT

Sample drift: sampled data drifts while population does not. Phone surveys.

Measuring drift

Compare distribtions from sample data. Rolling average, prediction error, accuracy. Then we set a threshold → cloudwatch, Eventbridge, Lambda