OpenLake Launches ExANS: Lossless KV Cache Compression at 622 GB/s
Key Takeaways
- ▸ExANS achieves 1.51× lossless compression of KV cache with 622 GB/s decode throughput on H100 GPUs, exceeding typical 400 Gb/s NIC speeds
- ▸Novel algorithm exploits low entropy in BF16 exponent fields through block-local alignment, bypassing the high entropy of mantissa data
- ▸On-GPU decompression moves the bandwidth bottleneck in LLM inference, reducing TTFT without requiring changes to inference engines
Summary
OpenLake has announced ExANS (Exponent Aligned Asymmetric Numeral Systems), a lossless GPU compression codec for BF16 KV cache that achieves 1.51× compression with 622 GB/s median decode throughput on NVIDIA H100 GPUs. The technology addresses a critical bottleneck in LLM inference—PCIe and network bandwidth limitations—by compressing KV cache blocks on GPU at speeds that exceed typical network wire rates, directly reducing key-to-first-token (TTFT) latency.
Unlike traditional compression approaches that struggle with BF16's high entropy, ExANS exploits the insight that real-world KV blocks exhibit low-entropy exponent fields. The codec isolates and aligns the 8-bit exponent stream using modified Asymmetric Numeral Systems (ANS) encoding while leaving sign and mantissa data uncompressed, achieving lossless compression without any quality loss or modifications to existing inference engines.
ExANS will be released as open source in OpenLake v0.8 with native connectors for vLLM and SGLang. By enabling on-GPU decompression speeds that exceed physical network delivery rates, the technology effectively moves the bandwidth bottleneck in LLM serving, allowing cached KV data to arrive faster than the wire delivers it.
- Open-source release integrates with vLLM and SGLang for immediate adoption in production LLM serving systems
Editorial Opinion
ExANS is an elegant systems solution that succeeds by working with data properties rather than against them. Rather than applying heavy general-purpose compression, the technique identifies and exploits a specific structural weakness in KV cache—the underutilized exponent field—to achieve both speed and ratio simultaneously. If these throughput numbers hold across diverse architectures and scale, this could become a standard optimization in production LLM deployments, meaningfully reducing both latency and hardware costs.
