Anthropic Introduces 'Warp Decode' for Faster MoE Model Inference on Blackwell GPUs
Key Takeaways
- ▸Warp decode reorganizes MoE inference parallelism around outputs rather than experts, eliminating five non-computational data-management stages during token decode
- ▸Achieves 1.84x throughput improvement on Blackwell GPUs while improving numerical accuracy by 1.4x relative to FP32 reference implementations
- ▸Compresses MoE compute layers into two fused kernels that operate without staging buffers or cross-warp synchronization, reducing memory latency and overhead
Summary
Anthropic has unveiled a novel inference optimization technique called 'warp decode' that significantly improves the performance of Mixture of Experts (MoE) models on NVIDIA's Blackwell GPUs. Rather than organizing computation around expert networks as is conventional, warp decode reorganizes parallelism around output values, assigning each GPU warp to compute a single output neuron. This architectural shift eliminates five data-management stages that perform no actual computation during autoregressive decode, where tokens are generated one at a time.
The technique delivers impressive performance gains: a 1.84x throughput improvement on Blackwell hardware while simultaneously improving numerical accuracy, with outputs 1.4x closer to full FP32 reference values. The approach compresses the entire MoE compute layer into just two highly optimized kernels (moe_gate_up_3d_batched and moe_down_3d_batched) that operate without staging buffers, cross-warp synchronization points, or intermediate data transfers. The innovation is particularly effective for small-batch decode scenarios, where traditional expert-centric approaches waste computational cycles on data organization overhead.
Anthropics credits warp decode with accelerating research and training pipelines for Composer, enabling faster model iterations and more frequent releases. The technique represents a rare case where kernel-level optimization improves both performance and accuracy simultaneously, suggesting deeper efficiency gains beyond simple speed-ups.
- Particularly effective for small-batch autoregressive decode where traditional expert-centric approaches have insufficient computational amortization
Editorial Opinion
Warp decode exemplifies how rethinking fundamental algorithmic assumptions at the GPU instruction level can yield substantial practical improvements. By recognizing that single-token generation has fundamentally different characteristics than batch prefill, Anthropic's team identified and exploited an optimization opportunity that the industry's standard MoE implementations had overlooked. The simultaneous gains in both throughput and numerical accuracy are particularly noteworthy—most low-precision optimizations sacrifice some accuracy for speed, making this result a genuine technical achievement that could influence how MoE inference is implemented across the industry.
