Speculative Speculative Decoding (SSD) Promises 2x Faster LLM Inference Through Parallel Processing
Key Takeaways
- ▸SSD achieves up to 2x faster LLM inference by running draft and verification models in parallel on separate hardware, rather than sequentially
- ▸The technique pre-generates speculations for multiple anticipated verification outcomes simultaneously, eliminating drafting overhead when predictions are correct
- ▸The open-source engine supports Qwen3 and Llama3 models with production optimizations including tensor parallelism, PagedAttention, and CUDA graphs
Summary
A new open-source inference optimization technique called Speculative Speculative Decoding (SSD) has been released on GitHub by researcher Tanish Kumar, claiming to achieve up to 2x faster LLM inference speeds compared to existing baselines. Unlike traditional speculative decoding where a small model drafts tokens and a large model verifies them sequentially, SSD performs these operations in parallel on separate hardware. The small model anticipates multiple verification outcomes simultaneously and pre-generates speculations for all possibilities, eliminating drafting overhead when predictions are correct.
The lightweight inference engine supports the Qwen3 and Llama3 model families and includes optimized implementations of both standard speculative decoding and autoregressive baselines for comparison. Technical features include tensor parallelism, PagedAttention, CUDA graphs, torch compilation, and prefix caching. The system requires Python 3.11+ and CUDA 12.8 or higher, and was developed and tested on H100 GPUs.
SSD represents a novel approach to the speculative decoding paradigm by distributing computational work across distinct hardware resources rather than processing sequentially. The technique maintains exactness while dramatically reducing latency, particularly beneficial for scenarios where multiple GPUs or accelerators are available. The project is released under an MIT license and includes reference implementations, benchmarking tools, and support for production-grade optimizations.
- Released under MIT license with reference implementations and optimized baselines for benchmarking
Editorial Opinion
SSD represents an elegant architectural innovation that transforms speculative decoding from a sequential to a parallel process, addressing one of the fundamental bottlenecks in current implementations. The 2x speedup claim is significant if it holds across diverse workloads, though real-world performance will depend heavily on hardware configuration and the accuracy of speculation. The open-source release with production-ready optimizations suggests this could quickly influence commercial inference systems, particularly for deployments with multi-GPU resources where the parallel architecture can be fully exploited.
