Accelerator capacity planning goes wrong in two directions, and both are expensive.
Over-reserve and you own idle silicon. Under-reserve and you are paying spot prices during exactly the hours your business needs the capability, which is when spot prices are worst.
Here is the shape that works.
Find the crossover honestly
Reserved capacity is cheaper per unit above some utilisation. Below it, serverless wins.
We would expect that crossover somewhere around a third to a half of sustained utilisation, depending on the accelerator and the commitment length — but treat that as a starting hypothesis to check against your own pricing rather than as a number to plan on. The important word is sustained. Peak utilisation is the number people quote and it is the wrong one — a cluster that hits 90% for two hours and 5% for twenty-two is a 12% cluster.
Measure sustained utilisation over a fortnight before committing to anything.
Reserve the base, rent the peak
Enterprise load is diurnal and weekly. There is a floor it rarely drops below and a peak it rarely exceeds.
Reserve to the floor. Burst the rest. The mistake is reserving to the peak because that is what capacity planning traditionally means — in an environment with a working serverless path, reserving to peak is buying insurance you already have.
Fill the trough, or stop calling it a saving
If you reserve capacity, you pay for it at 03:00 whether or not anything runs.
Things that fit in a trough: evaluation suite runs, batch document processing, embedding refresh, fine-tuning jobs, synthetic data generation, and any queue with a next-morning SLA.
This requires a scheduler that can preempt low-priority work when interactive load returns, and workloads that can tolerate preemption. It is genuinely more engineering. It is also the difference between 35% and 70% utilisation on the same hardware, which changes the crossover calculation entirely.
Size on memory bandwidth, not FLOPS
For serving, the constraint is almost always memory bandwidth and capacity, not compute. A model that fits in HBM with room for the KV cache at your target concurrency will serve well on modest silicon. One that does not will serve badly on the best available.
Work backwards: model size, quantisation, target concurrency, context length, and therefore KV cache. That gives you the memory requirement, which gives you the accelerator, which gives you the count.
Doing it the other way around — picking the best accelerator and hoping — is how estates end up with expensive hardware running at low utilisation because the model was the wrong shape for it.
Plan the interconnect for training, not for serving
Serving is mostly single-node. Training is not, and multi-node training is bounded by interconnect far more than by accelerator count.
If the roadmap includes training or substantial fine-tuning, the fabric decision is made at procurement time and is very difficult to change later. Plenty of estates have discovered they own a lot of accelerators they cannot train across.
Instrument before you optimise
You cannot plan capacity you cannot see. Per-workload attribution — which team, which use case, which model, which route — should exist before the first optimisation.
Without it, every capacity conversation is a negotiation between people's impressions. With it, it is arithmetic, and arithmetic settles faster.
- Compute
- GPU
- Capacity
- FinOps