Slack arbitrage: find the idle resource and spend it

When a system is bottlenecked on one resource, every other resource is idle — and idle capacity is free money. The recurring optimization move is to identify which resource the roofline model says is saturated, then deliberately spend the other one to relieve it.

Because decode is memory-bound, the idle resource during generation is almost always compute, which is why the same arbitrage shows up everywhere:

• FlashAttention restructures attention to stop paying the HBM tax — and in the backward pass literally recomputes values rather than store and reload them: compute spent, bandwidth saved.
• Speculative decoding uses the compute sitting idle during every decode step to verify $\gamma$ draft tokens in one parallel pass.

The general form: this is classical systems thinking — bottleneck analysis — imported into ML by scale. The arbitrage is only visible if you know the exchange rate, which is exactly what the ridge-point number (~300 FLOPs/byte on an H100) gives you.

Sibling threads: amortization, caching + indirection, verified approximation, scale-out. Hub: a map of LLM inference optimization.