Squish: Open-Source Local LLM Inference Server Achieves 14.7x Speedup Over Ollama on Apple Silicon
Key Takeaways
- ▸Squish is an open-source local LLM inference server optimized for Apple Silicon, delivering 1.15-14.7x faster inference than Ollama
- ▸Eliminates cold-start latency (20-30s with Ollama) by maintaining a persistent daemon that keeps models active
- ▸Uses five key architectural innovations including unified memory pooling, INT3 quantization, and KV cache optimization
Summary
Squish is a new open-source local LLM inference server optimized for Apple Silicon that delivers significantly faster performance than Ollama. Built on Apple's MLX framework, Squish achieves 1.15x to 14.7x faster inference depending on prompt repetition patterns. Developer wscholl created the tool to solve the problem of rate-limited cloud APIs (like Gemini) and slow local inference (like Ollama), targeting sub-three-second response times for common tasks like git commit message generation.
The project addresses a gap in the open-source AI infrastructure landscape by providing a lean, optimized inference server specifically designed for Apple Silicon's unified memory architecture. Squish uses five key architectural innovations: a persistent daemon to avoid cold-start latency, unified memory pooling for weights and cache, INT3 quantization by default, aggressive memory governance, and token streaming. These design decisions eliminate the 20-30 second cold-start overhead that plagues Ollama while maintaining a smaller memory footprint.
Scholl's journey from using Gemini (cloud-based but rate-limited) to Ollama (local but slow) illustrates a real pain point for developers wanting to integrate local AI into workflows. With weeks of configuration and tuning unable to achieve the required sub-five-second response times with Ollama, building a purpose-built alternative became necessary. Squish succeeds in delivering the performance and reliability needed for seamless local AI integration.
- Targets developers seeking rate-limit-free, low-latency local AI for real-time tasks like automated commit generation
- Addresses the infrastructure gap between cloud APIs (fast but rate-limited) and existing local solutions (unlimited but slow)
Editorial Opinion
Squish represents the pragmatic side of open-source AI development: a focused, purpose-built tool solving a specific problem that existing solutions couldn't handle. Rather than trying to be a general-purpose inference engine like Ollama, Squish optimizes ruthlessly for Apple Silicon's unique hardware architecture. This kind of specialized, single-platform tool may not have broad adoption, but it demonstrates how developers with specific needs can build superior solutions.
