The total cost of inference

Summary

Most enterprise AI business cases are built on a per-token price. That price is real, and it is typically 20–35% of what the capability actually costs to run.

This paper builds the full stack, including four cost lines that reliably appear in year two and are reliably absent from year-one business cases.

The visible cost

Serving compute. Tokens in, tokens out, at the model's rate — or, on dedicated capacity, the accelerator hours you have reserved.

The first structural decision is between the two. Serverless pricing is linear in usage and excellent at low and spiky volume. Dedicated capacity is fixed and cheaper per unit above a crossover point — plausibly somewhere around a third to a half of sustained utilisation, though that is a figure to derive from your own pricing rather than to inherit.

The mistake is choosing once. Most estates should run both: dedicated capacity sized to the reliable base load, serverless absorbing the peak.

The costs that are usually missed

Retrieval

Every grounded request does work before the model runs: embedding the query, searching the index, assembling and re-ranking context. At scale this is a meaningful compute line of its own, and it grows with corpus size rather than with request volume.

Index maintenance is the part that surprises people. A corpus that changes daily needs continuous re-embedding, and re-embedding the whole corpus after a model change is a project with a bill.

Evaluation

Running a real evaluation suite against a candidate version costs compute, and running it against every candidate — including the ones you reject — costs more.

Budget for evaluation as a standing line rather than a project cost. A firm that spends nothing on evaluation is not saving money; it is deferring an incident.

Retraining

Domain models decay. Not dramatically — a slow drift as the world moves away from the training distribution.

Budget for a retraining cadence from the start: data preparation, training compute, evaluation, and the staged rollout. Quarterly is typical for fast-moving domains, annually for stable ones. A business case with no retraining line is a business case for year one only.

Idle capacity

Dedicated capacity is paid for whether or not it is used, and enterprise load is diurnal. A workload sized for the 09:00–17:00 peak in one time zone is substantially idle for two thirds of the day.

This is recoverable — batch work, evaluation runs, retraining and lower-priority queues all fit into the trough — but only if the platform can schedule across them. If it cannot, the idle time is simply cost.

The costs nobody puts in a model

Integration. Connectors into systems of record, and the maintenance when those systems change. This is ordinary enterprise integration work and it does not become cheaper because AI is involved.

Data preparation. Entity resolution, survivorship rules, quarantine remediation. Front-loaded, substantial, and the highest-leverage money in the programme.

Human review. If policy requires human approval on a proportion of decisions, that is a per-decision operating cost that scales with volume. It belongs in the unit economics, not in a footnote.

Change management. The cost of people learning to work differently. Consistently underestimated and consistently the reason a deployed capability produces no measured benefit.

A unit that survives scrutiny

Per-token cost is not a decision-grade unit. Cost per decision is, because it is comparable to the thing being replaced.

Cost per decision = (serving + retrieval + evaluation + amortised retraining + attributed idle + human review) ÷ decisions.

Computing this requires cost attribution down to the workload, which requires tagging every request with its business context from day one. Retrofitting attribution across a live estate is unpleasant.

Once you have it, the conversation changes. "Inference is expensive" becomes "this decision costs 14 units to make automatically and 90 to make manually, and here is the distribution". That is a conversation a CFO can act on.

Where the savings actually are

In roughly this order:

1. Right-sizing the model. The largest available model is the default and it is usually wrong. Most enterprise tasks are served well by a substantially smaller domain-tuned model, at a fraction of the cost and latency.
2. Caching. Enterprise query distributions are far more repetitive than people expect, and semantic caching on a stable corpus can remove a large share of load. Measure your own repeat rate before sizing the benefit.
3. Routing. Send the easy majority to a small model and escalate on uncertainty. This is where cost-aware routing earns its complexity.
4. Scheduling the trough. Filling idle dedicated capacity with batch and evaluation work.
5. Prompt discipline. Context windows are billed. Retrieval that returns twelve documents when four would do is a cost line disguised as thoroughness.