Frame: Neuro-Symbolic SAST Uses LLMs to Close the Recall-Precision Gap in Vulnerability Detection
Key Takeaways
- ▸Neuro-symbolic design uses symbolic verification as a grounding mechanism for LLM proposals, preventing hallucinations and maintaining determinism
- ▸Achieves 0.67 recall / 0.51 precision (F1 0.58) on real-world applications, outperforming Semgrep OSS across recall-precision metrics
- ▸LLM layer recovers ~65 confirmed cross-file vulnerabilities missed by both traditional symbolic engines and pattern-based scanners
Summary
Endor Labs introduced Frame, a research project on neuro-symbolic static application security testing that bridges the traditional divide between symbolic analyzers and pure-LLM scanners. Frame keeps a sound symbolic engine (using taint analysis and Z3-backed separation logic) as its backbone for precision, then adds an optional LLM layer for breadth—including detection of cross-file vulnerabilities through an agentic loop. Crucially, the symbolic engine grounds and verifies every LLM proposal, assigning confidence tiers so symbolic proofs are never conflated with heuristic detections.
Evaluated on Endor Labs' real-world corpus of five production applications and 193 pooled vulnerabilities, Frame's full mode achieved 0.67 recall at 0.51 precision (F1 0.58), compared to Semgrep OSS's 0.52/0.40/0.45. The LLM layer alone recovered approximately 65 confirmed vulnerabilities that both the symbolic engine and Semgrep missed, spanning Java, JavaScript/TypeScript, and C#. A triaging pass further improved precision by removing confident false positives. Notably, the symbolic core prevents LLM hallucination through sink verification and operates using local models only.
- Operates entirely on local models—no external API calls required—addressing data privacy and cost concerns in enterprise security scanning
- Honest evaluation acknowledges ground-truth limitations, particularly that the pooled benchmark includes vulnerabilities identified by Frame's own LLM layer
Editorial Opinion
Frame's neuro-symbolic architecture addresses a real pain point: most security tools force you to choose between false positives (pattern scanners) and false negatives (sound symbolic analysis). By using symbolic verification to ground LLM detections and tier findings by confidence, Frame could become a template for other domains facing similar recall-precision tradeoffs. The research is refreshingly candid about its limitations—especially the risk of ground-truth contamination—which strengthens rather than weakens the contribution. However, the evaluation would benefit from independent human validation and broader benchmarking beyond Endor Labs' corpus.
