Cost attribution for AI workloads

The single most useful thing you can do in the first week of an AI programme costs almost nothing: tag every request with its business context.

What to tag

At minimum, on every request:

• Business unit — who pays.
• Use case — which capability this serves.
• Environment — production, evaluation, development.
• Model and route — what actually served it.
• Trigger type — interactive, batch, retry, evaluation.

The last one matters more than it looks. Estates routinely discover that a meaningful share of their spend is retries and evaluation runs, which appear in no business case.

Why day one

Retrofitting attribution across a live estate means threading context through call paths that were not designed to carry it, across services owned by different teams, while the estate is in use.

It is a quarter of work to add later and an afternoon to add at the start. And until it exists, every capacity and value conversation is people trading impressions.

Allocate shared and idle capacity, do not hide it

Two lines get quietly dropped and both distort the picture.

Shared services — retrieval, embedding refresh, the trace store, the control plane — cost real money. Allocate them by usage, not evenly, and not into a central bucket nobody owns.

Idle reserved capacity is the honest one. If you reserve to base load, the trough is paid for. Attributing it to whoever justified the reservation makes the reserve-versus-burst decision visible instead of comfortable.

Report cost per decision

Per-token cost is not decision-grade. It is not comparable to anything the business already measures.

Cost per decision is, because the manual baseline exists: this decision costs 14 units automatically and 90 manually, and here is the distribution.

Compute it as (serving + retrieval + evaluation + amortised retraining + attributed idle + human review) ÷ decisions. The human review term is the one people leave out, and for workloads with mandatory approval it can dominate.

Distributions, not averages

Average cost per decision hides everything interesting. The distribution shows you the long tail — the 3% of cases that consume 40% of the compute because they trigger deep retrieval, long contexts and escalation.

That tail is where optimisation actually pays, and it is invisible in an average.

Make it visible to the people who can change it

A cost dashboard only finance can see changes nothing.

Put cost per decision, by use case, in front of the teams who own those use cases, weekly. In our experience the first four weeks of visibility produce more saving than the following six months of optimisation work — because someone notices a job running hourly that only needed to run daily.

What to do with the number

In rough order of payback:

1. Right-size the model. The largest available is the default and usually wrong.
2. Cache. Enterprise query distributions repeat far more than people expect.
3. Route. Small model for the easy majority, escalate on uncertainty.
4. Schedule the trough with batch and evaluation work.
5. Trim retrieval. Twelve documents where four would do is a cost line disguised as thoroughness.