DeepSeek V4 Flash Achieves 32 Tokens/Second on Consumer AMD Hardware
Key Takeaways
- ▸284B-parameter DeepSeek V4 Flash model fits entirely in 128GB unified memory and runs locally at 32 tok/s without discrete GPUs
- ▸New ROCmFPX quantization family combines multiple bit-depth formats (2.50–4.25 bits/weight) optimized for AMD ROCm kernels
- ▸68.5% performance improvement over previous best results (HipFire 18.99 tok/s); 2.05× faster than prior AMD Ryzen AI Max baseline
Summary
DeepSeek's V4 Flash model, a 284-billion parameter mixture-of-experts model, has been successfully optimized to run entirely on consumer-grade AMD Ryzen AI MAX+ 395 hardware, achieving up to 32 tokens per second decode speed with no discrete GPU or remote inference required. The implementation uses a newly developed ROCmFPX quantization format family specifically designed for AMD ROCm/HIP execution, reducing model weights from full precision to approximately 2.88 bits per parameter while maintaining quality. This result represents a 68.5% performance improvement over the previous best-known implementation on the same hardware platform and demonstrates that substantial performance gains are still achievable through careful hardware-software co-optimization. All code is publicly available and reproducible on the Lucebox main branch.
- DeepSeek-specific HIP kernel optimizations, DSpark speculative decoding, and collaborative GPU kernel tuning (via Geometric) drove performance gains
- Public implementation removes barriers to running frontier-scale LLMs on consumer hardware; reproducible without proprietary patches
Editorial Opinion
Running a 284-billion parameter frontier model locally on consumer hardware is a watershed moment for AI accessibility. Rather than relying solely on raw compute scaling, this work proves that co-designed quantization formats and kernel-level optimizations can still unlock massive gains—and make large models practical for individuals and small teams without cloud dependencies. As AMD's hardware matures, this demonstrates the real potential of the consumer AI inference market.
