Anyscale Achieves 67% Cost Savings in LLM Serving Through Prefill-Decode Disaggregation
Key Takeaways
- ▸Prefill-decode disaggregation achieved 1.3-2.7x better QPS and up to 67% compute cost reduction on AMD MI325X hardware compared to aggregated serving under identical GPU budgets and SLA constraints
- ▸PD disaggregation does NOT improve time-to-first-token (TTFT) and can actually increase it due to KV cache transfer overhead, making it unsuitable for latency-sensitive interactive applications
- ▸The technique's effectiveness varies significantly by workload, requiring careful tuning of prefill-to-decode ratios and analysis of input/output lengths and KV cache hit rates
Summary
Anyscale has demonstrated significant cost efficiency gains in LLM serving by implementing prefill-decode (PD) disaggregation on AMD MI325X GPUs using Ray Serve LLM and vLLM. The approach separates prompt processing (prefill) and token generation (decode) onto dedicated hardware, eliminating computational contention and achieving 1.3x to 2.7x better queries per second (QPS) compared to traditional aggregated serving—translating to up to 67% cost reduction in compute infrastructure.
PD disaggregation works by having prefill GPUs handle prompt processing while decode GPUs handle token generation. This separation prevents mutual interference in compute, memory bandwidth, and scheduling, allowing each phase to run closer to its theoretical throughput. However, the technique adds operational complexity through KV cache transfers across nodes and requires per-workload tuning of the prefill-to-decode ratio.
Crucially, Anyscale's research reveals important limitations: PD disaggregation does not improve time-to-first-token (TTFT) metrics and can actually degrade them due to the KV cache transfer overhead. Testing on large mixture-of-experts models showed the technique delivers significant savings for certain workloads—particularly those with longer output sequences and favorable KV cache hit rates—but provides minimal or no benefit for others. The findings provide practitioners with clear guidance on when to adopt PD disaggregation versus maintaining simpler aggregated serving architectures.
- Ray Serve LLM orchestration and AMD's RIXL technology enable efficient KV cache transfer between nodes, making disaggregated serving operationally feasible
- Anyscale offers managed solutions bringing similar cost savings to production workloads, with open-source Ray+vLLM stack available for self-hosted deployments
Editorial Opinion
Anyscale's prefill-decode disaggregation research is a valuable contribution to LLM serving efficiency, demonstrating clear cost-saving potential for production deployments. However, the nuanced findings—that PD can harm TTFT and works best on specific workload patterns—underscore that this is an advanced technique requiring deep technical expertise and careful workload profiling rather than a universal optimization. Teams considering PD disaggregation should use Anyscale's detailed workload guidance to avoid costly misdirections.
