AI researcher · LLM inference
I work on making large models cheaper to run (speculative decoding, batching, the KV cache) and I write to find out what I think.
These are my working notes: summaries of what I read, evergreen notes that argue one claim each, and longer essays when a thread is ready. More about me.
Recently added
- Improving Data Efficiency for LLM Reinforcement Fine-tuning through Difficulty-targeted Online Data Selection and Rollout Replay
- Continual Learning for Agents
- The Art of Scaling RL Compute for LLMs
- What I Learned from RL and Agentic RL Interview Questions
- Where Training Data Comes From - Synthesis, Verification and Distillation Across the Training Stack
- Continual LR with Neuroevolution
- Tropical RL
- Loss of Plasticity in Deep Continual Learning
Evergreen notes
Prefill is compute-bound; decode is memory-boundSpeculative decoding is provably exact; draft calibration, not accuracy, is the leverPagedAttention is virtual memory for the KV cache, not a new attention algorithmBatching is nearly free until the ridge point, then trades latency for throughput ~1:1Chunked prefill trades one request's TTFT for everyone's inter-token latency stabilityQuantization loss is outlier-driven, not bit-driven