Harel Lidar

Paper summary · Aug 2026

AgentSpec Speculative Decoding for Batch Inference of LLM Agents

Summary of AgentSpec Speculative Decoding for Batch Inference of LLM Agents

The authors begin by noting the inherent friction that exists in modern speculative decoding - the algorithms are designed to provide speed ups at low batch sizes but modern AI service platforms all operate under high-load. The authors attempt to adapt speculative decoding to the modern agentic AI era, and they do this through two main mechanisms:

  1. Lowering rejection rates - AgentSpec constraing speculation to semantically coherent segments of the agent workflow. By avoiding speculation across heterogeneous execution stages, agentspec reduces the generation and wasted verification of irrelevant tokens.
  2. Dynamic Token Budges - introduces a redundancy metric to guide the allocation of available token budgets across requests.

Relevancy

Questions