AirLLM Achieves Major Milestone: Running 2.8T Kimi K3 on Single 4GB GPU
Key Takeaways
- ▸Kimi K3 (2.8T parameters) now runs on a single 4GB GPU in 3.72GB VRAM—the largest open-source model to achieve this efficiency milestone
- ▸AirLLM v3.0 added FP8 quantization support, enabling DeepSeek-V3 (671B) to run on ~12GB and Qwen3-235B on ~3GB
- ▸Sparse MoE architecture enables per-expert streaming, loading only the experts that each token routes to rather than entire layers
Summary
AirLLM has announced support for inference of Kimi K3, a 2.8-trillion parameter sparse Mixture of Experts (MoE) model, on a single 4GB GPU without quantization, distillation, or pruning. This represents the largest open-source model ever successfully run on minimal hardware, demonstrating how sparse MoE architecture enables efficient streaming of individual experts rather than entire model layers.
The latest release (v3.0, June 2026) introduced FP8 model support and expanded compatibility with cutting-edge models including DeepSeek-V3 (671B running on ~12GB), Qwen3-235B (running on ~3GB), Llama 3.1 (405B), and others through a unified AutoModel interface. Kimi K3 support (July 2026) pushed the boundaries further, achieving end-to-end inference in just 3.72GB of VRAM on an RTX 6000 Ada GPU.
AirLLM's optimization approach works by streaming only the sparse experts that a given token routes to, rather than loading entire model layers into memory. This fundamentally changes the economics of running state-of-the-art models, making them accessible on consumer and edge hardware. The library continues to mature with improvements including CPU inference support, dynamic model detection, and model compression features offering up to 3x inference speedup.
- Unified AutoModel interface automatically detects model types, simplifying inference across diverse architectures from multiple organizations
- No quantization, distillation, or pruning required—AirLLM achieves efficiency through architecture-aware memory optimization
