Evals, Monitoring, Gateways, Skills & Rules best practices

What do evals test, and why can the same agent give different results on two runs?

Evals test the harness (rules, skills, AGENTS.md lines). Agents are non-deterministic, so even a fixed setup gives varying results.

How should you treat a line in the harness?

As a guess: measure it before you keep it. The agent following a line does not mean the agent improves.

Why can harness assumptions go wrong over time?

Models improve, so assumptions about what the agent can't do go stale.

Which harness lines are probably helping?

Short, specific lines about something the agent can't know unless you tell it, but they should still be tested.

What do a unit test, a fitness function, and an eval each check?

Code, architecture, and the harness.

What did Gloaguen et al. (February 2026, arXiv 2602.11988) find about AGENTS.md files?

No shown improvement in success rate, with cost up by around 20%. The agents did follow the instructions, so context files should hold only minimal requirements.

What does Expo say to test for a skill, and where do the cues go?

Test discovery, because a skill only matters if an agent finds it. Put the cues in the title and the opening sentence.

Which way do LLM judges tend to be biased?

They overrate results, especially from the same model family, so calibrate them before relying on them.

How should you phrase the question you give an LLM judge?

Situation-based ("does this follow our principles for good design?"), not subjective ("is it beautiful?").

What does an eval compare, and what is the baseline?

Two sets of runs that differ in one thing: with the rule vs without it. The runs without the rule are the baseline.

What must an eval case's action name?

One specific behavior, e.g. "put the new file under a given place", or descriptive behaviors that must be tested.

What is a negative control?

A case where the rule should not affect the output, used to make the comparison fairer.

What is pass@k, and what do you compare between the two arms?

pass@k is the chance that at least one of k attempts passed. Compare pass rate, pass@k, cost, and time with vs without the rule.

Is an eval result the verdict?

No, it is evidence. The programmer makes the verdict.

When is "just try it and remove it" enough, and when do you run a small eval?

Try and remove when the line is narrow, rare, and easy to delete. Run a small eval when it fires on every task or you must show the result to others.

What is a gateway and what is it used for?

A server between requests and the model provider: devs get a gateway key, the gateway holds the real provider key and budget configs. It enforces budgets, revokes keys, and keeps team credentials in one place.

What are the risks of a gateway, and how do Anthropic's docs treat LiteLLM?

If an attacker gets it, it is over; if it goes down, it is also over. The docs treat LiteLLM as an example gateway, unaffiliated and unaudited.

What do /usage, /insights, and ccusage do?

They estimate session cost, give an HTML report of your sessions, and let you compare token usage with Claude Code's own numbers.

What does OpenTelemetry give you for Claude Code, and what is the catch?

Cost, tokens, sessions, skills, and tools per team, so you can see what a skill costs. You set it up yourself, and account identifiers are still sent. Check Anthropic's first-party dashboard first.

When would a small team adopt evals, OpenTelemetry, or a gateway?

Evals when several people share a harness and a change could break many projects. OpenTelemetry when spend becomes a real line item. A gateway when you need hard per-person caps or runaway loops are a real risk.