Ekka: Automated Diagnosis of Silent Errors in LLM Inference
Key Takeaways
- ▸Silent errors in LLM serving frameworks degrade output quality without raising alerts—a critical reliability gap in production systems
- ▸Ekka achieves 80% diagnosis accuracy on real-world bugs at ~$30/case by automatically comparing execution states between buggy and reference implementations
- ▸Tool has already discovered 4 new confirmed bugs in vLLM and SGLang, validating its effectiveness for improving open-source LLM infrastructure
Summary
Researchers have introduced Ekka, a tool for automatically diagnosing silent errors in LLM serving frameworks. Unlike traditional bugs that raise errors, silent errors degrade output quality without alerting the system. Ekka uses differential debugging to compare intermediate execution states between a buggy framework and a trusted reference, pinpointing exactly where divergence occurs.
The tool demonstrates strong practical effectiveness: when tested against 17 real-world bugs from vLLM and SGLang, Ekka achieved 80% pass@1 diagnosis accuracy at approximately $30 per case. Beyond validating against known bugs, Ekka has successfully identified 4 previously unknown bugs that have since been confirmed by developers, demonstrating its utility for improving LLM serving infrastructure reliability.
The research will be presented at ICML 2026, with the authors available for discussion at Poster Session 5 during the conference's poster sessions.
Editorial Opinion
Ekka addresses a critical blind spot in LLM serving reliability: silent failures that go undetected while degrading service quality. The 80% diagnosis accuracy on real bugs and discovery of previously unknown issues validate the differential debugging approach as a practical tool for infrastructure maintainers. This work signals growing maturity in the LLM serving ecosystem, where robustness and debugging capabilities are becoming as important as performance optimization.
