Paper summary · Sept 2026
AdaptiVocab - Enhancing LLM efficiency in focused domains through lightweight vocabulary adaptation
Provenance
- Link — https://arxiv.org/pdf/2503.19693
- Authors / affiliation — Technion, Itai Nakash, Roi Reichart
- Venue, date — Aug 2025
- How it reached me —
- Why I clipped it — SuperTokenizers for Draft Models
Claim
When focusing on narrow domains, one can achieve significant speedups by removing rare tokens and replacing them with common N-Grams
Techniques and evidence
- Identify highest scoring N-grams
- Identify lowest scoring tokens
- Replace set from 1 with set from 2.
- Apply tokenizer to new text -
- Tokenize text with original tokenizer
- Decompose tokens that were removed from the original vocab into smaller chunks
- Merge existing tokens into the new n-tokens where possible
- This is done greedily, otherwise it is too computationally expensive to be worthwhile
- Exponential embedding initialization 1.
Generalizable vs specific
Cherry-picking
Limitations and flaws
Why is this a one<>one replacement of rare and new