Fast enough that people use it, cheap enough to keep
The same model on the same hardware can cost ten times more or less; we find the operating point where latency and bill both fit.
Economics
Bigger batches raise tokens per second and raise time-to-first-token with them. Aggressive quantisation cuts memory and can cut quality. We agree the budgets first, in milliseconds and in currency, then tune against them.
Throughput
2–4×
typical gain from continuous batching over static batching
Latency
sub-300ms
time-to-first-token target for interactive chat workloads
Cache
On a loaded server it is attention memory, not weights, that caps concurrency. Prefix reuse removes the long preamble every agent and RAG request repeats.
Fleet
Send cheap traffic to a small model, serve dozens of LoRA adapters from one base, and scale on queue depth rather than CPU.
Quantisation
FP8 and INT8 are close to free on current hardware. INT4 quarters weight memory but costs quality unevenly, so we re-run your eval set before shipping it.
Serving engines
Orchestration
Quantisation
Cache and decode
Measurement
Who we do this for
1–2 weeks
We benchmark your current stack under replayed traffic and rank the available wins by effort.
Baseline benchmark, cost per million tokens, ranked list of wins
4–10 weeks
Engine tuning, cache and batching, quantisation behind quality gates, routing — each shipped by shadow then canary.
Tuned configuration in version control, quantised artefacts, load-test suite
Ongoing
We run the serving tier under an SLA and re-tune it as traffic and model options change.
Capacity plan, cost reporting, model upgrade path
It depends on where you start. Untuned stacks usually give up the most to three dull things: no continuous batching, no prefix caching on a long shared preamble, and a frontier model serving traffic a small one handles identically. We measure your current cost first, and if the headroom is small we say so.
Below a steady, predictable volume a hosted API is cheaper once you price the engineers, the idle capacity you hold for peak and the on-call rota. Self-hosting wins at sustained high volume, tight latency, or where residency rules leave no choice. Do not hire us to optimise a cluster you should not be running.
Shadow traffic first, so the new configuration answers real requests nobody sees. Then a canary with automatic rollback on latency or quality regression. Load tests use replayed traces, because prompt length distribution is what determines cache behaviour and a uniform test tells you a comfortable lie.
Send us the constraint you are actually up against — budget, latency, regulator, deadline.
