Liquid AI Expands LFM2.5-8B-A1B Tokenizer to Support Underrepresented Languages More Efficiently
Key Takeaways
- ▸Tokenizer vocabulary expanded from 65K to 128K tokens without retraining the entire model from scratch
- ▸Hindi, Vietnamese, and Thai now require 2.4x, 2.6x, and 4.0x fewer tokens respectively, improving on-device decoding speed by 2.2-3.7x for these languages
- ▸Novel in-place upgrade method extends existing BPE merge rules and initializes new embeddings from decompositions of original tokens, preserving learned representations
Summary
Liquid AI has unveiled a novel approach to upgrading model tokenizers in place, expanding LFM2.5-8B-A1B's vocabulary from 65K to 128K tokens without requiring full model retraining. The expansion dramatically improves language representation for Hindi, Vietnamese, and Thai—reducing token counts by 2.4x, 2.6x, and 4.0x respectively—translating to 2.2x to 3.7x faster per-character decoding speeds on-device. The method works by extending the original Byte Pair Encoding (BPE) merge rules, initializing new token embeddings from existing ones, and adapting the model in two stages. This approach addresses a fundamental challenge in on-device LLMs: compact vocabularies necessary for memory efficiency often leave non-English languages severely under-tokenized, causing latency and energy penalties for users. The expanded LFM2.5-8B-A1B model and 128K tokenizer are now available on Hugging Face alongside the full technical methodology.
- Addresses a critical limitation of on-device models: the tradeoff between vocabulary size and deployment feasibility, which has historically disadvantaged underrepresented languages
Editorial Opinion
This work exemplifies thoughtful engineering for the real world. Rather than pursuing the brute-force solution of retraining from scratch, Liquid AI's approach preserves existing learned representations while systematically extending coverage—a pattern more AI teams should emulate. For multilingual on-device inference, this is a meaningful step toward equity; languages like Hindi and Thai have historically suffered from tokenizer bias in mainstream models. The released code and model are welcome contributions to the open ecosystem.
