Routing across providers is a policy problem wearing an engineering costume

Model routing gets pitched as a cost optimisation. Send the easy requests to the cheap model, the hard ones to the expensive one, save 40%.

That works, and it is the least interesting thing routing does.

Residency is a routing decision

An estate operating across jurisdictions cannot send every request to the same place. A request touching EU personal data may need to be served in-region. A request from a regulated entity in one country may be prohibited from leaving it entirely.

This is not a preference to express in a config file somewhere and hope. It has to be enforced at the routing layer, from attributes carried on the request, with a hard failure when no compliant route exists.

The failure mode to design against is a fallback that silently degrades residency. If the in-region endpoint is unavailable, the correct behaviour is usually to fail — not to quietly serve the request somewhere the policy prohibits. That is a decision to make deliberately, in advance, with the risk owner in the room.

Capability floors, not capability preferences

"Route to the cheapest model that can do the task" is a preference. What you need is a floor: this workload requires function calling, a 200k context, and a documented evaluation result above threshold on our suite.

Express the floor as a requirement on the route, not as a model name. Naming models in application code is how estates end up unable to move.

Deprecation is the resilience case nobody plans for

Providers retire models. It is normal product management for them and an outage for you, with a notice period measured in months against enterprise change cycles measured in quarters.

Routing that abstracts the model is what turns that from an incident into a change ticket. But only if two things are true:

1. No model names in application code. Applications ask for a capability class. The route resolves it.
2. Evaluation runs against the replacement before it is routed to. Otherwise the abstraction has simply moved the surprise from deployment to production.

Latency budgets belong in the route

Interactive paths and batch paths want different things. A customer-facing agent needs a P95 it can promise. An overnight document classification job does not care and would rather have the cheap route.

Put the budget on the route and let it choose. Applications should not be reasoning about which model is fast today.

What routing should not do

It should not silently change behaviour. If a request is served by a different model than yesterday, the trace must say so. Debugging an output change without knowing the route changed is miserable.

It should not route around a failing evaluation. If the primary is failing quality checks, the answer is an alert, not a quiet substitution.

It should not be clever without being observable. Every routing decision should be inspectable after the fact, with the reason attached. A router nobody can audit is a source of unexplainable variance.

Build the router for policy first and cost second. The cost saving is real and it arrives anyway. The residency guarantee and the deprecation resilience are the parts you cannot retrofit under pressure.