New Record: 1 Trillion-Parameter Model Serves at 511.6 Tokens/Second on NVIDIA B200s
Key Takeaways
- ▸NVIDIA B200 GPUs set a new leaderboard record for trillion-parameter model serving: 511.6 tokens/second, 13.4% above the prior leader and achieving ~128 tokens/second per accelerator—approximately 3x NVIDIA's own disclosed configuration
- ▸Results are fully reproducible using open-source code with pinned SHA-256 hashes and transparent measurement methodology, democratizing access to infrastructure benchmarking and enabling independent verification by the research community
- ▸Detailed performance analysis identified mixture-of-experts token routing as the dominant bottleneck; verification cost grows linearly with draft-token count, explaining why speculative decoding adds overhead that raw draft quality alone cannot justify
Summary
A new benchmark record has been established for serving trillion-parameter open models at scale. Kimi-K2.6, a 1 trillion-parameter open model, achieved 511.6 tokens per second throughput when deployed across four NVIDIA B200 GPUs—the highest single-stream reading recorded on public serving leaderboards, outperforming Crusoe (438.1 tok/s), Fireworks (381.2), CoreWeave (261.8), and all other competing infrastructure providers. The achievement was measured using the same transparent methodology employed by public serving leaderboards, with reproducible open-source code (modal run stage600_r.py) that any researcher can verify. Technical decomposition of the serving pipeline revealed that mixture-of-experts routing and draft token verification represent the primary performance bottlenecks; switching to an optimized engine removed 2.1 milliseconds per step, yielding a 15.5% improvement. Results were validated through three blind re-runs of the unmodified public release, each landing within the stated performance envelope and lifting the record to 511.6 tokens per second.
- Per-request latency decomposes cleanly: bare forward pass 6.6ms at 10k context, draft verification 0.397ms per token, with optimization reducing step overhead by 2.1ms—insights applicable to any MoE-based large model deployment
Editorial Opinion
This record matters not just for the number, but for the methodology. By publishing open code, artifacts, and transparent measurement protocols, this research moves the serving benchmark space toward reproducibility and away from proprietary black boxes. The technical decomposition of mixture-of-experts routing costs provides a framework that architects can apply across different hardware and models. NVIDIA's B200 demonstrates clear performance advantages for large model inference at scale, and the work validates the importance of optimized engine implementation for unlocking that hardware potential.