Researcher Uses LLMs to Discover Dozens of Vulnerabilities in Major Open-Source Projects

Summary

A security researcher has published findings on using large language models (LLMs) like Claude Opus to identify vulnerabilities in well-known open-source projects, discovering over a dozen CVEs entirely through AI-powered analysis without manual code review. The research demonstrates that agentic LLMs can effectively uncover obscure security issues in major projects including Parse Server, HonoJS, ElysiaJS, and Harden Runner. However, the researcher challenges conventional wisdom about AI-assisted security auditing, finding that excessive context and over-scaffolding actually degrades vulnerability detection performance due to "context rot"—where model reliability deteriorates as token count increases. Instead, the most effective approach involves minimal persistent scaffolding combined with maximal targeted exploration, keeping the model's focus anchored to critical details.

• The research reveals primacy/recency biases in LLMs where models perform better when relevant information appears near context boundaries rather than in the middle

Editorial Opinion

This research challenges assumptions about how to effectively deploy LLMs for security work and suggests that simpler, more focused prompting strategies may outperform elaborate orchestration frameworks. The finding that context rot degrades vulnerability detection is particularly important for the security community, as it implies that traditional approaches to comprehensive code analysis may need rethinking when using AI tools. If these results hold at scale, they could reshape how organizations approach AI-assisted security auditing.