Harel Lidar

Paper summary · Sept 2026

ThunderAgent

Summary of ThunderAgent · paper ↗

Provenance

Claim

Current LLM usage patterns are no longer a model-level but rather, agentic inference is a composition of inference engines with tool orchestrators. The existing optimizations are component-specific and don’t have the “entire scope”, without said global view system-level optimization of KV cache utilization (clean up as well as hit rates), tool lifecycles and cross-memory node imbalances (from tail latencies) are impossible. This paper presents an abstraction of agentic workflows as agentic programs which is a fist-class scheduling unit that persists across model innovations and tool executions and by exposing metadata for the workflow allows optimizations from state pausing, migration and tool resource management.

Techniques and evidence

The authors show that current decomposed pipelines neglect optimized memory usage, tool lifecycle management and KV Cache optimizations.

Abstraction of Programs

A program is represented as a tuple of <ID,c,T,L,τ,s><ID,c, \mathcal{T}, \mathcal{L}, \tau, s> where ID represents the unique global identifier. c denotes the number of tokens in the context, corresponding to the KV cache memory footprint during active execution. T tracks the set of tool environments used by the program, enabling garbage collection when no program requires them further. L, τ , and s denote the node placement, execution phase, and scheduling status, respectively, facilitating program-level KV cache thrashing reduction and cross-node transferring.

Cost Model

Their cost model uses the Space-Time product (STP) as the primary metric, which is defined as the integral of the memory footprint (They use token count as a proxy) over time. They further decomose total cost as the sum of decode, prefill, recomputation, unused capacity and idle caching while external tools are processing - only the first two are “effective compute”.

Generalizable vs specific

general

Specific

Cherry-picking

Limitations and flaws

Theirs

Mine

4.3.1 The way they derived the “optimal” evicition strategy assumes all sessions are equally likely to resume idle time simultaneously, this is what begets the optimized solution being least sum of squared context lengths

Concrete output

Anki