Cost-aware routing, and the failure mode it introduces

The configuration

Requests are served first by a small tuned model. An escalation decision — based on the small model's uncertainty, retrieval grounding scores and task-type priors — sends a subset to a frontier model.

The saving is real and arrives quickly, because the majority path becomes the cheap one. Median latency improves for the same reason. P99 gets slightly worse, since escalated requests pay for both models.

That much is arithmetic. The part worth writing down is what it costs you that does not show up in a dashboard.

The failure mode

Escalation decisions have two error types.

Over-escalation sends an easy request to the expensive model. It costs money and harms nothing.

Under-escalation keeps a hard request on the small model, which answers it — plausibly, fluently, and worse.

Under-escalation is invisible in aggregate metrics. Overall accuracy barely moves, because under-escalated requests are a small fraction of volume. But they are not randomly distributed. They concentrate exactly where the small model is confidently wrong, which is a systematic population rather than noise.

The mechanism is worth being precise about: a router trained or tuned on request distribution learns that unusual inputs are rare. Rare and hard are different properties, and nothing in the frequency signal distinguishes them.

The controls that address it

Never route on the model's own confidence alone. Combine it with retrieval grounding quality and task-type priors. Self-reported confidence is a property of the output distribution, not of task difficulty — the same argument as in the evidence quorum note.

Force-escalate a random sample. A small percentage of eligible traffic goes to the frontier model regardless, and the two outputs are compared. This is what converts the under-escalation gap from an assumption into a measurement.

Build this on day one. Retrofitting it means having no baseline for exactly the period you most want to understand.

Escalate on rarity, not just uncertainty. Unusual entity types, unusual document types, out-of-distribution vocabulary — escalate by policy rather than by learned judgement.

Escalate consequential decisions by policy. Above a value threshold or touching a flagged party, cost is not the relevant consideration.

What to expect

The router will be the component you tune most, and most of the tuning will be about when to escalate rather than which model to escalate to.

Budget for that. A routing layer is not a thing you configure once; it is a thing you operate, and the control sample is what tells you whether it is still doing its job.