Paper summary · Sept 2026
Task-Centric Acceleration of SLMs
Provenance
- Link — https://arxiv.org/pdf/2602.24174
- Authors / affiliation —
- Venue, date —
- How it reached me —
- Why I clipped it — Part of the SuperTokenizers for Draft Models Research
Claim
By considering the N-gram statistics of output distributions on a representative dataset, we can add a finetuning step that enhances the base vocabulary with popular N-Grams in a way that reduces latency when serving it.
Techniques and evidence
Tokenization
They generate outputs over a train set that is pre-collected
- Since they only want to improve decode-time latency rather than prefill they only consider outputs, not inputs. under the assumption that the stats under each regime may differ. Then, they define a merge reward by its exprected contribution to seq length reduction by
. Doing this naively could bias greedy decoding towards collapsing multiple plausible continuations into a single dominant path - surpressing alternatives whose prefix collides with the new token’s suffix
- For instance collapsing the bigram [“in”,“the”] can make tokens like “therapy” or “theorem” less likely to appear. To mitigate, they propose the Prefix Collision Score (PCS) defined as
and only collapse into a super-token if the merge reward is high enough AND the PCS doesn’t pass a certain, predefined threshold.
Fine Tuning
They initialize each new “mega-token” as the mean of its constituent token embeddings
Generalizable vs specific
- The technique of adding N-grams “straight” into the model seems general, there’s nothing architecture/data specific about it.
Cherry-picking
- Choosing datasets with extremely structured output data, classification style tasks where the benefit is obvious due to low entropy and abundance of recurring results
Limitations and flaws
The threshold for PCS is flaky, not to mention that it doesn’t take into account the magnitude of the merge reward.
Concrete output
Anki
- We find additional motivation in the uniform information distribution (UID) hypothesis which posits that efficient linguistic representations tend to distribute information evenly across tokens, leading to approximately uniform per-token surprisal along sequences