Amortization: pay a fixed cost once, spread it over more work

Wherever a large fixed cost must be paid regardless of how much useful work rides along, the move is to attach more work to each payment. In LLM serving the fixed cost is usually one trip of the model weights across the memory bus:

• Batching is the purest case — weights loaded once per decode step, reused across every sequence in the batch, nearly free until the ridge point.
• Speculative decoding amortizes the target model’s expensive forward pass over $\gamma$ verified tokens instead of paying it per token.
• Prefix sharing amortizes one physical KV allocation across many logical requests.
• Microbatching amortizes the pipeline’s fixed fill/drain bubble over $M$ work-items.

The boundary of the thread matters as much as its members: amortization has a natural limit wherever the cost isn’t fixed — KV-cache traffic scales with B × context and amortizes across nothing, which is exactly where the caching thread has to take over. And TP/DP don’t belong here at all: they aren’t making a fixed cost cheaper per unit, they’re adding units when one unit’s budget is exceeded.

Sibling threads: slack arbitrage, caching + indirection, verified approximation. Hub: a map of LLM inference optimization.