Expanding LLM Tokenizers Without Discarding Weights

Summary & Methodology Analysis

Language-specific tokenizer fragmentation in pre-trained on-device LLMs causes increased latency and energy consumption for languages underrepresented at the start of pre-training. To solve this without discarding existing model weights, the paper proposes an in-place expansion recipe. Tokenisation, the process of breaking text into discrete sub-word units, is improved by continuing the source tokenizer's byte-level BPE, a sub-word tokenisation algorithm, merge training on a multilingual corpus to initialize a new 128K merge table. Embedding initialization is performed by copying embedding rows unchanged for the majority of source tokens that map one-to-one, while new rows are initialized by averaging the source sub-token embeddings.

The adaptation process is split into two distinct stages. In Stage 1 adaptation, only the new embedding rows are trained while keeping all other model parameters frozen over 600B tokens. In Stage 2 adaptation, full-model continued pre-training is performed for 400B tokens on a balanced multilingual corpus to integrate the new vocabulary into the model body. The models and datasets used include LFM2-8B-A1B, LFM2.5-8B-A1B, FineWeb 2, MMLU-Pro, GSM8K, MATH500, GSM-Plus, HumanEval+, LiveCodeBench v5, MMMLU, MGSM, IFEval, and Multi-IF. This approach yields significant token-count reduction on under-tokenized languages, reaching up to 4.0x fewer tokens for Thai compared to the source tokenizer.

Despite these improvements, the method has several limitations. The recipe is restricted to continued-BPE constructions and cannot be applied to off-the-shelf third-party tokenizers. Additionally, the expanded vocabulary introduces a per-character decode regression of up to 9% on languages already encoded efficiently by the source tokenizer. Furthermore, training the full embedding matrix in Stage 1 degrades generative performance. The synthesis of compression gains and vocabulary-size costs yields an estimated 2.2x to 3.7x per-character decode speedup for under-represented languages on reference devices, and aggregate benchmark performance reaches an 8-benchmark mean accuracy of 48.3%.