vLLM Announces Production-Ready Kimi K3 Support With 370 Tokens/Second Throughput
Key Takeaways
- ▸vLLM serves Kimi K3 at up to 370 tokens/sec with speculative decoding—3.14× faster than baseline
- ▸Hybrid prefix caching for recurrent KDA state enables efficient context management at scale
- ▸Open-source DSpark speculative decoding (trained and released by Inferact) integrated for enhanced throughput
Summary
vLLM has launched production-scale support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model featuring a 1M-token context window and hybrid Kimi Delta Attention (KDA) architecture. The integration achieves up to 370 tokens per second with speculative decoding—a 3.14x improvement over the baseline 118 tokens/second—making advanced open-weight models viable for real-world deployment.
Kimi K3's novel architecture required significant optimization work from the vLLM team. The model combines recurrent-state KDA (reducing KV cache overhead) with periodic full-attention layers, native MXFP4 (4-bit) quantization, and 896 experts with only 16 active per token. vLLM implemented hybrid prefix caching for recurrent state, integrated Inferact's open-source DSpark speculative decoding algorithm, and added native support for Kimi K3's tool calling and structured reasoning capabilities.
The solution is production-ready across NVIDIA (Hopper/Blackwell) and AMD (MI355X) platforms, with Docker deployment recipes available for immediate adoption. vLLM's day-0 support demonstrates the project's ability to rapidly adapt serving engines to emerging model architectures.
- Full production support for tool calling, reasoning output, structured output, and prefill/decode disaggregation on day-0
- Cross-platform GPU support (NVIDIA and AMD latest-gen) with Docker deployment recipes
