WASTE Inference Engine Enables Running Kimi K3 2.78T on MacBook Pro with 29GB Memory
Key Takeaways
- ▸WASTE enables 2.78 trillion parameter models to run on consumer hardware with ~29GB effective memory usage, fundamentally challenging assumptions about local deployment limits
- ▸Separates model trunk (resident in memory) from expert weights (stored on NVMe), with intelligent caching to match the access patterns of mixture-of-experts models
- ▸Open-source project includes multiple client options and platform support, designed for reproducibility and community contribution
Summary
WASTE, an open-source inference engine, has been released to enable running Kimi K3, a 2.78 trillion parameter mixture-of-experts model, on consumer hardware with dramatically reduced memory requirements. The engine successfully demonstrated running the complete 1.42 TB Kimi K3 model on a MacBook Pro with 64 GB of unified memory—a capability previously thought impossible without extensive model pruning or distillation. By separating the model into a memory-resident trunk and storing expert weights on NVMe storage, then dynamically accessing them through an intelligent cache system, WASTE overcomes the traditional constraint of model size exceeding available RAM.
The engine leverages Kimi K3's mixture-of-experts architecture, where only 16 of 896 experts are active per token, to minimize storage bandwidth requirements. Currently achieving approximately 0.32 tokens per second on the MacBook Pro—with a smaller 48B variant reaching 8.9 tokens/sec—WASTE proves that local inference of large models is feasible, though not yet practical for interactive applications. The open-source release includes a CLI client, OpenAI-compatible HTTP server, and multi-platform support (macOS, Linux, Windows) across ARM and x86 architectures, with text and image input support. This initial release represents a significant step toward the broader goal of decentralizing AI infrastructure and enabling local control over model execution and data privacy.
- Current performance of 0.32 tokens/sec is impractical for interactive use but proves the viability of the approach
- Part of broader effort to move AI infrastructure control from cloud providers to end users, improving data privacy and cost economics
Editorial Opinion
WASTE represents a meaningful breakthrough in the ongoing effort to bring capable AI models under user control. While the 0.32 tokens/second performance is far too slow for practical deployment today, the fact that a 2.78 trillion parameter model runs at all on consumer hardware without distillation or pruning is remarkable. As inference engines optimize further and storage bandwidth improves, this could reshape how organizations think about AI infrastructure—trading cloud costs and vendor lock-in for local control and privacy. The open-source approach also matters; shared breakthroughs on memory-efficient inference could compound across the industry.
