Harel Lidar

Paper summary · Sept 2026

Task-Centric Acceleration of SLMs

Summary of Task-Centric Acceleration of SLMs · paper ↗

Provenance

Claim

By considering the N-gram statistics of output distributions on a representative dataset, we can add a finetuning step that enhances the base vocabulary with popular N-Grams in a way that reduces latency when serving it.

Techniques and evidence

Tokenization

They generate outputs over a train set that is pre-collected

fmerge(tn)≖frq(tn)⋅(n−1)f_{merge}(t^n) \eqcirc frq(t^n)\cdot (n-1)

. Doing this naively could bias greedy decoding towards collapsing multiple plausible continuations into a single dominant path - surpressing alternatives whose prefix collides with the new token’s suffix

PCS(tn)=P(⋃t′∈V′str⁡(tn) is a prefix of (str⁡(t′)))\mathrm{PCS}(t^n) = P\left( \bigcup_{t' \in \mathcal{V}'} \operatorname{str}(t_n) \text{ is a prefix of } \bigl(\operatorname{str}(t')\bigr) \right)

and only collapse into a super-token if the merge reward is high enough AND the PCS doesn’t pass a certain, predefined threshold.

Fine Tuning

They initialize each new “mega-token” as the mean of its constituent token embeddings enew=1n∑j=1netje_{new}= \frac{1}{n}\sum_{j=1}^{n}e_{t_j}

Generalizable vs specific

Cherry-picking

Limitations and flaws

The threshold for PCS is flaky, not to mention that it doesn’t take into account the magnitude of the merge reward.

Concrete output

Anki