Mercor SPL

Generation

From scratch, no starting text

Open QA

No single fixed correct answer

Closed QA

One specific, checkable correct answer

Classification

Predefined category/label

Brainstorming

Multiple ideas, not one final answer

Chat

Multi-turn conversation; earlier turns affect later ones

Rewrite

Form/tone/style changed, meaning and length kept

Summarization

Text condensed, key info kept, shorter

Verifiable answer → data type

SFT, ideally with an automated checker (math, code, closed QA)

Judgment call → data type

RLHF / rubric-graded preference data (tone, reasoning quality, open-ended writing)

SFT: how it works

Ideal answer shown, model copies it. Only type with no grader

RLHF / Preference Labeling

Two outputs compared, human picks better, trains a reward model

Rubric-Based Evaluation

Criteria written once by experts and then judge model at scale. The dominant method

RL Environments

Agent acts in a simulated/real setting, rewarded on outcomes, not just text

Wrong format/structure/steps

SFT (demonstrate the right answer)

Capable, but wrong tone/style/character

RLHF (learn human preferences)

Inconsistent quality on complex, subjective reasoning

Rubrics (pinpoint where reasoning fails)

Can't interact with tools/systems beyond a chatbox

RL Environments (measure outcomes of actions)

pass@k

Capability metric: probability of at least one success in k tries

pass^k

Reliability metric: probability all k tries succeed

Hill-climbing sweet spot

Ideal prompt difficulty: ~20–40% model score, not too easy or too hard

Model-as-judge

Rubric written once by humans; model applies it to score at scale

Single-turn eval

One prompt, one response, one grade

Multi-turn eval

Back-and-forth conversation; early mistakes compound

Agent eval

Tool use across many turns, changing state; highest cascading-failure risk

Why evals matter (3 reasons)

Product, scoreboard, directs experts

APEX

Tests high-value professional knowledge work

GDPval

Real tasks across 44 occupations, top 9 US GDP sectors

SWE-bench Verified

Real GitHub issues resolved, patch passes tests

Humanity's Last Exam

Extremely hard, probes near/superhuman capability

HealthBench

Medical/clinical reasoning benchmark, safety-grounded

τ-Bench

Multi-turn simulated agents, retail/airline settings

Benchmark saturation

Top scores near 100%, no longer useful. SWE-bench Verified: ~30% to >80% in months

Open Box: 3 premises

Data is core IP; annotator quality is the biggest lever; speed is critical

Open Box: 5 steps

Propose, source/vet, design pipeline, measure, swap talent

Prompt: the 6 requirements (CFUM SORT)

Critical failure, multi-step, unambiguous, objective, relevant, timeless

Rubric: the 8 requirements (TF SC P1 UTEREA)

T/F, unambiguous, self-contained, one check, expert agreement, timeless, matches prompt, error ranges

Pipeline hierarchy

Campaign > batch > task

IAA (Inter-Annotator Agreement)

Consistency of reviewers grading the same work; Cohen's Kappa = 2 raters, Fleiss' Kappa = more than 2

Stalled batch / failed gate diagnosis

Instructions gap, task difficulty, low performers, pause wave

QC steps & fixes

Writer self-check, calibration session, gold-set spot checks, consolidated feedback, targeted rubric fix, gold-example reference

Causes of bad data

Unrepresentative data, inconsistent labels/low IAA, reward hacking, instructions gap, task difficulty, low performers

Foody's blog thesis

Models are smart but not trained to do the job

Macrosoft: 6 required elements

Data type, domain+why, expert archetype/sourcing, quality mgmt, ramp plan, billing