Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding

Summary

A new research paper introduces Cassandra, an algorithm-hardware co-designed framework that accelerates reasoning Large Language Models (LLMs) on edge devices through self-speculative decoding. The approach addresses a critical bottleneck in deploying reasoning models on consumer hardware: decode-stage overhead that existing methods struggle to mitigate without sacrificing accuracy or requiring additional training.

Cassandra works by constructing a high-performance draft model through fine-grained data selection, using optimized pruning and mantissa truncation to identify the most salient values in model weights and Key-Value (KV) cache. This enables rapid candidate token generation before full-precision parallel verification. Unlike prior self-speculative methods based on layer skipping or structured KV compression, Cassandra achieves significantly higher efficiency gains. The framework also introduces a lightweight encoder-decoder hardware module for seamless integration with commercial GPUs and NPUs, reducing format conversion overhead.

Experimental results demonstrate substantial practical improvements: Cassandra achieves up to 2.41x speedup over the BF16 baseline without any additional training, and on Llama 3 8B running on an NVIDIA GeForce RTX 4090, it generates 1.81x more tokens under the same memory budget compared to Eagle-3, a state-of-the-art competing method. These results suggest that efficient reasoning model deployment on consumer devices is now more practical.

• Training-free self-speculative decoding through fine-grained data selection overcomes previous accuracy-efficiency trade-offs

Editorial Opinion

Cassandra addresses a genuinely important problem: most reasoning LLMs today remain impractical for edge deployment due to their decode-time overhead. By achieving meaningful speedups without retraining, this work lowers barriers to consumer hardware adoption. The hardware co-design aspect is particularly valuable—software-only optimizations have plateaued, so the push toward algorithm-hardware collaboration reflects the maturation of the efficiency optimization space.