Evaluation suites that can actually fail a release

Summary

Public benchmarks answer a question no enterprise is asking. They measure general capability on tasks chosen by researchers, and they are increasingly contaminated by their own popularity.

The question an enterprise is asking is narrower and more consequential: does this version of this model, on our data, in our workflow, fail in ways we cannot accept?

Answering it requires a suite the business owns, weighted toward the firm's own failure modes, with the authority to block a release. This paper covers how to build one.

The property that matters most

An evaluation suite that has never blocked a promotion is not a gate. It is a report with a gate's job title.

Before anything else, establish that the suite can stop a release, that stopping a release is a normal outcome rather than an incident, and that the person who owns the suite does not report to the person shipping the model.

Everything else in this paper is detail by comparison.

Build the set from failures, not from coverage

The instinct is to sample representatively — take 500 random cases and measure accuracy. This produces a number that is stable, comforting and nearly useless, because the cases that matter are rare by construction.

Build instead from:

• Historical failures. Every incident, complaint, overturned decision and near-miss the firm already has. These are the failure modes you know are real.
• Adversarial cases. Written by domain experts who are trying to break it. An underwriter who has spent twenty years finding the edge of a policy wording is a better adversary than any red-team framework.
• Boundary cases. Just inside and just outside every policy threshold. This is where automated systems produce their most expensive errors.
• A representative sample. Still needed — as a regression floor, not as the headline.

Weight them explicitly. A suite where a catastrophic failure counts the same as a formatting error will report an average that hides the thing you built it to catch.

Grade with rubrics, not with similarity

String similarity against a reference answer measures whether the model phrased something the way the annotator did. For most enterprise tasks that is not the property under test.

Use graded rubrics per case, authored by the domain owner, scoring the dimensions that matter: is it factually supported by the retrieved evidence, does it apply the correct policy, does it hedge where it should, does it refuse where it must, is any claim present that the evidence does not support.

Rubric grading can be model-assisted, and should be spot-checked by humans on a fixed sample rate. The spot-check rate is a control, not an optimisation target.

Measure refusal as a first-class outcome

Most suites score correct answers and treat everything else as failure. This produces models tuned to always answer, which is exactly the behaviour that ends clinical and financial pilots.

Score four outcomes separately: correct answer, incorrect answer, correct refusal, incorrect refusal. An incorrect answer and an incorrect refusal have very different costs, and collapsing them hides the trade you are actually making.

Refresh the suite, and watch it drift

A suite built in January measures January's world. Products change, policies change, and the distribution of what arrives changes.

Two mechanisms:

• Continuous accrual. Every production failure, every human override and every sampled disagreement becomes a candidate case. Review the queue monthly.
• Staleness audit. Annually, sample the suite and ask whether each case still reflects current policy. Cases that encode a superseded rule are worse than no case, because they punish correct behaviour.

Run it on the whole system, not the model alone

The unit under test is the deployed path: retrieval, prompt assembly, the deterministic checks, the model, the post-processing and the policy gates.

Model-only evaluation misses the majority of production failures, which come from retrieval returning the wrong thing, a gate misconfigured, or a prompt template that silently truncates. Test what you ship.

Report it in a form a risk committee can use

Not a leaderboard. A short document: what changed, what the suite says, which cases regressed, what the failure-weighted score is against the promotion threshold, and an explicit recommendation with a name attached.

The suite produces evidence. A person makes the call. Automating the call away is how organisations end up unable to explain why a release went out.

The reference implementation — suite scaffolding, rubric schemas, the grading harness and the promotion-gate integration — is in the developer portal, along with worked examples for financial services, clinical and industrial workloads.