OpenAI's GPT-5.6 Sol Dominates Security Vulnerability Detection in Pull Requests
Key Takeaways
- ▸GPT-5.6 Sol achieves 100% recall on planted security vulnerabilities in PRs while costing only $0.70 per analysis, establishing a clear cost-quality frontier
- ▸Anthropic's Fable 5 significantly underperforms at $3.61 per PR with lower quality metrics, suggesting enterprises may be overspending on PR security reviews with Claude models
- ▸The benchmark challenges industry assumptions that Anthropic models universally dominate security work—workload distinction is crucial between PR reviews and full-codebase scanning
Summary
A comprehensive benchmark comparing leading AI models for security vulnerability detection in pull requests reveals that OpenAI's GPT-5.6 Sol has achieved clear dominance on the cost-quality frontier. The model reaches 100% recall—finding all planted access-control vulnerabilities—while costing just $0.70 per PR analyzed, making it roughly 45% cheaper than its predecessor GPT-5.5 while maintaining performance. The benchmark tested 10 models across identical corpora of pull requests containing intentionally planted vulnerabilities (IDOR, missing auth, broken authorization), with results validated across multiple testing harnesses including Claude Code. Notably, no Anthropic model reached the cost-quality frontier, with Fable 5 performing significantly worse at $3.61 per PR—more than five times GPT-5.6's cost while delivering lower recall and precision. The research emphasizes that this benchmark specifically measures PR review workloads, where additional context is limited, rather than full-codebase scans where Anthropic models may perform better.
- xAI's Grok 4.6 and Google's Gemini 3.5 Flash also outperform Anthropic, but GPT-5.6 Sol offers the best combination of recall, precision, and cost efficiency for the PR review workload
Editorial Opinion
This benchmark disrupts the narrative that Anthropic models reign supreme for security analysis, revealing that Fable is significantly overpriced for PR security work compared to GPT-5.6. The findings emphasize that organizations must evaluate models empirically rather than relying on market assumptions, and that capability-to-cost ratios matter enormously in infrastructure decisions. However, the authors' distinction between PR reviews and full-codebase scanning suggests rankings may shift for different workloads—a critical reminder that no single model dominates all use cases.
