Archive
Everything here, by kind. Each kind is a different sort of writing, so each is shown differently. Titles without a link were read or attended and logged, but not published.
Essays
Longer pieces in my own words.
Evergreen notes
Atomic notes: each argues one claim, and the title is the claim.
- A kernel is memory-bound below the GPU's ridge point — which is ~300 FLOPs/byte, not 1
- A map of LLM inference optimization
- Agentic workloads are a workload characterization, not a technique
- Amortization: pay a fixed cost once, spread it over more work
- Approximation spans a spectrum from bounded-loss to provably-free
- Attention is the only inter-token operation in a transformer
- Batching is nearly free until the ridge point, then trades latency for throughput ~1:1
- Caching avoids redoing identical work; indirection makes the reuse shareable
- Chunked prefill trades one request's TTFT for everyone's inter-token latency stability
- Continuous batching moves scheduling from batch granularity to iteration granularity
- FlashAttention keeps the attention matrix out of HBM via fusion, tiling, and online softmax
- KV-cache traffic scales with batch × context; weight traffic is fixed per step
- Microbatching shrinks the pipeline bubble from (P−1)/P toward (P−1)/M
- PagedAttention is virtual memory for the KV cache, not a new attention algorithm
- Prefill is compute-bound; decode is memory-bound
- Prefix sharing is safe only with reference counts and copy-on-write
- Quantization loss is outlier-driven, not bit-driven
- Slack arbitrage: find the idle resource and spend it
- Speculative decoding is provably exact; draft calibration, not accuracy, is the lever
- TP pays latency in per-layer all-reduces; PP pays in bubbles; DP pays in memory
- When one unit's budget is exceeded, the problem changes category
Paper summaries
One paper each: the problem, the method, what the results do and don’t show.
- Improving Data Efficiency for LLM Reinforcement Fine-tuning through Difficulty-targeted Online Data Selection and Rollout Replay
- The Art of Scaling RL Compute for LLMs
- Continual LR with Neuroevolution
- Tropical RL
- Loss of Plasticity in Deep Continual Learning
- Reward Respecting Subtasks for Model-Based RL
- Finetuning with Sampling - SFT Learns Better Than You Think
- Razor & Higher Order Pruning of Experts in MoE language Models
- Pedagogical RL - Teaching Models to Teach Temselves from Privileged Information
- Generalization Dynamics of LM Pre-Training
- Mid Training Survey
- CodeMidas - Scaling Agentic Coding RL Environments from Code Itself
- Reinforce-Ada - An Adaptive Sampling Framework under Non-linear RL Objectives
- Breaking the Token Ceiling - Distilling Smaller, Stronger Byte Models
- DeepSeek V4.1-Flash
- ThunderAgent
- AdaptiVocab - Enhancing LLM efficiency in focused domains through lightweight vocabulary adaptation
- Task-Centric Acceleration of SLMs
- Conditional Memory via Scalable Lookup - A New Axis of Sparsity for Large Language Models
- Distillation Scaling Laws
- Speculative Probing - LLM Monitoring at Speculative-Decoding Cost
- DuoDecoding - Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
- AgentSpec Speculative Decoding for Batch Inference of LLM Agents
- Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
- LiLiCorr - Lightweight Likelihood Correlation ofParallel Drafts for Speculative Decoding
- Magpie - ALIGNMENT DATA SYNTHESIS FROM SCRATCH BY PROMPTING ALIGNED LLMS WITH NOTHING
- Decoding Speculative Decoding
- TRAINING DOMAIN DRAFT MODELS FOR SPECULATIVE DECODING - BEST PRACTICES AND INSIGHTS
- CoSine - Collaborative Speculative Inference for Efficient LLM Inference Serving
- Not-a-Bandit Provably No-Regret Drafter Selection in Speculative Decoding for LLMs
- Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
- Speculative Speculative Decoding
- Accelerating Speculative Decoding with Block Diffusion Draft Trees
- Gradient Surgery for Multi-Task Learning
- ProRL
- RLVR - DeepSeek R1
- CAASL: Amortized Active Causal Induction with Deep Reinforcement Learning
- One-2-3-45++ - Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion
- DreamGaussian - Generative Gaussian Splatting for Efficient 3D Content Creation
- DreamFusion - Text-to-3D using 2D Diffusion
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering
- pixelNeRF - Neural Radiance Fields from One or Few Images
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
- SIREN - Implicit Neural Representations with Periodic Activation Functions
- Neural Fields as Learnable Kernels for 3D Reconstruction
- OpenScene - 3D Scene Understanding with Open Vocabularies
- PointNet and PointNet++
- Randomization Inference When N Equals One
- Targeted Cause Discovery with Data-Driven Learning
- LION - Latent Point Diffusion Models for 3D Shape Generation
- Meta Statistical Learning
- Symmetry Paper
- TabPFN - Understanding and Advancing Tabular Foundation Models
- The Geometry of Categorical and Hierarchical Concepts in Large Language Modeels
- Evaluating Language-Model Agents on RealisticAutonomous Tasks
- A Primer on the Inner Workings of Transformer-Based Language Models
- Retrieval Heads explain long context
- DPO vs PPO for alignment
- Mixture of Depths
- ShortGPT
- Effects of Scale on Model Finetuning
- UL2 Objective
- AI Safety - Fundamentals course — Week 3 - RLHF
Lecture notes
Notes taken in courses, grouped by course.
Statistical Information Theory
- Intro
- Entropy
- Symbol Codes
- Joint Entropy
- Channel Coding
- Probabilistic Inference
- Estimating Information
- Information Geometry
Intro to ML
- Lecture 1
- Decision Trees and KNNs
- Evaluation
- Neural Networks Part 2
Introduction to Symbolic AI
- Lecture 1
- Semantics
- SAT Problem
- First Order Logic
- Knowledge Representation
- Herbrand Models
- Problem Solving by Search
- Search with full observability, BFS, DFS, UCS, MiniMax, alpha-beta, Heuristic Search - A* and BestFS
- Search with Partial Observability
Mathematics for ML
- Lecture 1
- Lecture 2
SIT Course Summary
- SIT Course Summary
Reinforcement Learning
- Lecture 2
- Model Free Control
- Value function approximation
- function approximation
Value function Iteration methods
AI Safety - Fundamentals course — Week 2 - What is alignment
- AI Safety - Fundamentals course — Week 2 - What is alignment
Policy Gradient Methods
- Policy Gradient Methods
ML Course Summary
- ML Course Summary
Deep Learning
- CNNs
Quantifying Fairness in ML
- Quantifying Fairness in ML
Computational Optimization
- First Order Optimization
Reflections From First Session
- Reflections From First Session
Reflections From Second Session
- Reflections From Second Session
General Concepts to internalize
Graphical Models
Mixture of Models
NEML
- Lecture 3
Flow Matching and Diffusion Models
- Lecture 1
Flow Matchings and Diffusion Models
- Training the Model to Predict the Vector Field
Concepts
A glossary: one idea per entry, explained briefly.
- LoRa (Low Rank Adaptation)
Excerpts
Mostly other people’s words: article summaries, quoted passages, course summaries. Kept as reading, not as my writing.
- Article summaryContinual Learning for Agents — Jiaxin Zhang
- Article summaryWhat I Learned from RL and Agentic RL Interview Questions — Jiaxin Zhang
- Article summaryWhere Training Data Comes From - Synthesis, Verification and Distillation Across the Training Stack — Jiaxin Zhang
- Course summaryThe Ultimate Guide to RL environments
- Article summaryTowards Looped Model Done Right
- Article summaryThe 4 Bitter Lesson
- Article summaryThe Ultimate Guide to Multi-Harness RL
- Article summaryEnvironment Scaling for Agentic RL
- Article summaryBoW -> JEV — Sebastian Raschka
- Article summaryHow Frontier Labs Train LLMs
- Quoted passagesSuperBPE - Space Travel for Language Models
- Quoted passagesSMELT - Scaling Laws for Compute-Matched MoE Looped Transformers
- Article summaryThe Bitter Lesson Behind Building Agentic RL in Terminal Environments - Evergreen Takeaways
- Article summaryNeural Scaling and the Quanta Hypothesis — Eric J. Michaud
- Course summarySummary of Report
- Article summaryAgency from a Causal Perspective
- Article summaryIncentives
- Article summaryReward Hacking
- Article summaryPrompt-Ideals for spaced repetition
- Article summaryTransformative tools for thought