Designing an evaluation suite from scratch

Start smaller than you think and weight it harder than feels comfortable.

Step 1: define what a failure costs

Before writing a single case, list the ways this system can hurt the organisation, and rank them.

For a claims system that might be: settling a fraudulent claim, refusing a valid one, breaching a regulatory timescale, exposing another claimant's data, or producing a rationale that will not survive a complaint.

This list determines everything about the suite. A suite that scores all errors equally will report an average that hides the one you built it for.

Step 2: source cases from reality

In priority order:

Historical failures. Every incident, complaint, overturned decision and near-miss the organisation already has. These are known-real failure modes and they are free.

Expert adversarial cases. Ask three domain experts to spend a day trying to break it. An underwriter who has spent twenty years finding the edge of a policy wording will out-perform any automated red-team.

Boundary cases. Just inside and just outside every policy threshold. This is where automated systems produce their most expensive errors.

Representative sample. Ordinary cases, as a regression floor.

A hundred cases across these four beats a thousand random ones.

Step 3: write rubrics, not reference answers

Do not grade on similarity to a golden answer — that measures phrasing.

For each case, write what a good response must do and must not do. Typically five to eight criteria, some pass/fail, some graded:

• Reaches the correct disposition.
• Every factual claim is supported by the retrieved evidence.
• Applies the correct policy version.
• Declines where the evidence is insufficient.
• Does not assert anything about parties not in scope.
• Rationale would survive a complaint.

The domain owner writes these. Not engineering — the person who will be accountable when it goes wrong.

Step 4: score four outcomes, not two

Correct answer, incorrect answer, correct refusal, incorrect refusal.

Collapsing refusals into failure produces a system tuned to always answer, which is precisely the behaviour that ends clinical and financial deployments.

Step 5: weight and set a threshold

Apply the cost ranking from step 1. A catastrophic failure might be worth twenty ordinary ones.

Then set a promotion threshold and write it down before you run anything, because thresholds set after seeing results are not thresholds.

Include hard gates: any case in the catastrophic class that fails blocks promotion regardless of the aggregate.

Step 6: run it on the whole system

Not the model in isolation. The deployed path: retrieval, prompt assembly, deterministic checks, model, post-processing, policy gates.

Most production failures come from retrieval returning the wrong thing or a gate being misconfigured. Model-only evaluation cannot see either.

Step 7: keep it alive

Accrue. Every production failure, human override and sampled disagreement becomes a candidate case. Review monthly.

Audit for staleness. Annually, check that each case still reflects current policy. A case encoding a superseded rule punishes correct behaviour, which is worse than no case at all.

The property that matters most

The suite must be able to block a release, blocking must be a normal outcome rather than an incident, and the person who owns it must not report to the person shipping the model.

A suite that has never failed anything is a report wearing a gate's job title.