Truth Benchmark: Open-Source Tool Systematically Detects Code-Documentation Mismatches
Key Takeaways
- ▸Systematic benchmark created to measure code-documentation verification accuracy with 52 labeled examples across six programming languages
- ▸LLM baselines achieve ~73% accuracy on verification, with 11.5-point gap between rule-based and LLM approaches and 25-point gap between LLM and perfect accuracy
- ▸Extensible evaluation framework allows any model to be plugged in with minimal code, lowering the barrier for researchers to benchmark their approaches
Summary
A new open-source benchmark project has been released to automatically identify when software code fails to implement what its documentation claims. Created by researcher 02zerocool, Truth Benchmark addresses a pervasive problem in software quality: documentation and README files describe features, security properties, or behavior that don't actually exist in the code. This drift causes real harm—security teams audit documentation instead of code, and AI models trained on documentation inherit the falsehoods.
The benchmark includes three progressively sophisticated verification baselines: heuristic rule-based matching, semantic similarity via embeddings (all-MiniLM-L6-v2), and LLM-based verification using local models (Llama 3.1) or OpenAI-compatible APIs. Evaluated on a 52-example dataset across Python, JavaScript, Go, Rust, Java, C#, and SQL, the LLM baseline achieves 73% accuracy. The hardest cases to detect are subtle code errors: wrong constants, off-by-one mistakes, missing authentication checks, and wrong sort direction.
The project is designed to be extensible. Any researcher or practitioner can plug in their own verification model with just two lines of code and compare performance against the baselines. The project actively solicits contributions of new examples, especially cases involving security vulnerabilities (missing encryption, missing validation, wrong protocol) and plausible-but-wrong code that real developers might accidentally write.
- Addresses critical problem where security claims in documentation aren't implemented in code—a vulnerability that AI models trained on documentation inherit
