Trillion-Parameter Model Serving Reaches 511.6 Tokens Per Second Record
Key Takeaways
- ▸Single-stream trillion-parameter model serving can exceed 500 tokens/second on commodity GPU clusters with optimized inference engines
- ▸Mixture-of-experts architecture imposes measurable 'verify tax' where expert routing complexity increases inference latency by 1.8× for speculative decoding strategies
- ▸Reproducible benchmarking with open code and pinned prompts is critical for credible serving infrastructure comparisons; blind re-runs validated the 511.6 tok/s record
Summary
Researchers have achieved a record throughput of 511.6 tokens per second when serving Moonshot AI's Kimi-K2.6 trillion-parameter open model on four NVIDIA B200 GPUs in a single-stream, lossless configuration. The achievement surpasses the leading GPU-based serving providers on the public leaderboard by 13.4% and marks the first single-stream GPU reading exceeding 500 tokens per second for this model size. The technical breakdown reveals that the forward pass costs 6.6ms at 10k context with token verification at 0.397ms, while strategic engine optimization removed 2.1ms per step. The result is reproducible from the public release with a single command, costing approximately $15 and 40 minutes of compute time on rented hardware.
The research provides detailed analysis of serving infrastructure complexity, particularly the 'verify tax' imposed by Kimi-K2.6's mixture-of-experts (MoE) architecture. Comparative testing shows that different draft verification strategies incur vastly different costs: EAGLE's four-token verify costs 8.0–9.3ms per step while DFlash's nine-token verify costs 11.5–16.6ms, creating a throughput-accuracy tradeoff. The measurements are positioned against competitor serving providers including Crusoe (438.1 tok/s), Fireworks (381.2), and CoreWeave (261.8), establishing a clear performance hierarchy on production benchmarking criteria.
- Engine-level optimizations can improve baseline MoE serving performance by 15.5%, suggesting significant headroom in existing production serving stacks
Editorial Opinion
This technical report sets an important standard for infrastructure benchmarking in the increasingly competitive open-model serving market. The emphasis on reproducibility—releasing code, exact prompts, and per-token artifacts—stands in stark contrast to marketing claims from other providers. The detailed analysis of mixture-of-experts complexity reveals that raw token throughput masks the real engineering tradeoffs; future serving comparisons should measure the verification costs specific to each draft family, not just final throughput numbers.
