A map of LLM inference optimization
Inference optimization looks like a zoo of techniques, but it decomposes into a small set of recurring moves applied to one underlying situation. This note is the map; every claim links to a note that argues it.
The substrate — two facts everything grows from:
- Attention is the only inter-token operation in a transformer, which is why the KV cache exists and is attention-only.
- Prefill is compute-bound; decode is memory-bound — quantified by the roofline: a kernel is memory-bound below the GPU’s ridge point, and decode sits far below it. The bytes that move are governed by KV-cache traffic scaling with B × context while weight traffic stays fixed.
Five threads — the recurring moves:
- Slack arbitrage — find the idle resource and spend it: FlashAttention, speculative decoding’s use of idle compute.
- Amortization — pay a fixed cost once, spread it: batching below the ridge, microbatching, spec-decode’s one-pass-per-γ.
- Caching + indirection — never redo identical work, and make the reuse shareable: the KV cache, PagedAttention, refcounts + copy-on-write.
- Verified approximation — from quantization’s bounded loss to speculative decoding’s provably exact free lunch.
- Scale-out — when one unit’s budget is exceeded, the problem changes category: TP, PP, DP and their distinct costs.
The scheduling layer, where techniques meet traffic: continuous batching and chunked prefill — where the throughput-vs-TTFT tension gets settled request by request.
The lens on top: agentic workloads are a workload characterization, not a technique — the empirical claim about input shape that decides which threads pay off hardest.
Two meta-observations the map keeps yielding: the most powerful techniques (speculative decoding, PagedAttention) deliberately span multiple threads rather than one; and the whole field is scale forcing AI engineering to import classical hardware-architecture and distributed-systems thinking.