Serverless or dedicated endpoints: how to actually decide

Both options are correct for different workloads, and most estates should run both. Here is how to decide which goes where.

Serverless inference

You send requests, you are billed per token, and capacity is somebody else's problem.

Good for: unpredictable or spiky volume, early-stage workloads, low sustained utilisation, and anything where you would rather not forecast.

Costs you: a cold-start tail on less common models, less control over the P99, noisy-neighbour variance, and — depending on the provider — less say over exactly where the request is served.

Dedicated endpoints

Reserved capacity, yours, billed by the hour whether used or not.

Good for: predictable base load, latency-sensitive interactive paths, workloads with residency or isolation requirements, custom or fine-tuned models, and anything above the utilisation crossover.

Costs you: the idle hours, and a forecasting obligation.

The crossover

Dedicated becomes cheaper above some sustained utilisation — plausibly around a third to a half, though the exact point falls out of your own pricing and commitment length rather than out of a rule of thumb.

Two things people get wrong:

Peak instead of sustained. A cluster at 90% for two hours and 5% for twenty-two is not a 90% cluster. Measure the sustained figure over at least a fortnight.

Ignoring the trough. If you can fill idle hours with batch work, evaluation runs and embedding refresh, effective utilisation rises substantially and the crossover moves in dedicated's favour. That requires a scheduler that can preempt, and workloads that tolerate it.

Latency

Dedicated is not automatically faster per token. What it is, is more predictable, and predictability is usually what the requirement actually needs.

If you have promised a customer-facing P95, you need a tail you control. Serverless P50 can be excellent while the P99 varies with somebody else's load.

When it is not a choice

Residency. If the workload must be served in a specific jurisdiction or inside your own network boundary, that usually means dedicated.

Isolation. Some regulatory positions do not permit shared inference infrastructure regardless of logical separation.

Custom weights. A model you have fine-tuned generally needs somewhere to live.

Air-gapped. No serverless option exists by definition.

The pattern that works

Reserve to the reliable base load. Burst to serverless above it. Route by workload class: interactive paths to dedicated for the predictable tail, batch and overflow to serverless.

Start every new workload serverless. Move it when its sustained utilisation earns it, not when someone estimates that it will.

What to instrument first

You cannot make this decision from impressions. Before committing to anything, have per-workload request volume over time, sustained utilisation, latency distributions including the tail, and cost attributed to the workload rather than to the platform.

With those four numbers the decision is arithmetic. Without them it is a debate.