Evergreen Notes
Atomic, densely-linked working notes in the spirit of Andy Matuschak’s evergreen notes: each note argues one claim, its title is the claim, and the links are where the structure lives. Unlike blog posts, these are continually revised.
The first series maps LLM inference optimization. Start at the hub:
A map of LLM inference optimization
Mechanisms
- Attention is the only inter-token operation in a transformer
- Prefill is compute-bound; decode is memory-bound
- A kernel is memory-bound below the GPU’s ridge point — which is ~300 FLOPs/byte, not 1
- Batching is nearly free until the ridge point, then trades latency for throughput ~1:1
- KV-cache traffic scales with batch × context; weight traffic is fixed per step
- FlashAttention keeps the attention matrix out of HBM via fusion, tiling, and online softmax
- PagedAttention is virtual memory for the KV cache, not a new attention algorithm
- Prefix sharing is safe only with reference counts and copy-on-write
- Continuous batching moves scheduling from batch granularity to iteration granularity
- Chunked prefill trades one request’s TTFT for everyone’s inter-token latency stability
- Quantization loss is outlier-driven, not bit-driven
- Speculative decoding is provably exact; draft calibration, not accuracy, is the lever
- TP pays latency in per-layer all-reduces; PP pays in bubbles; DP pays in memory
- Microbatching shrinks the pipeline bubble from (P−1)/P toward (P−1)/M
Threads
- Slack arbitrage: find the idle resource and spend it
- Amortization: pay a fixed cost once, spread it over more work
- Caching avoids redoing identical work; indirection makes the reuse shareable
- Approximation spans a spectrum from bounded-loss to provably-free
- When one unit’s budget is exceeded, the problem changes category