Study: AI-Assisted Code Breaks Production Half as Often as Human-Written Code
Key Takeaways
- ▸AI-assisted PRs break production 50% less often than non-AI code (1.9% vs 4.4%), even after controlling for PR size and repository
- ▸Broken-main rates scale 16x with team size: 0.8% at 2–5 engineers versus 12.5% at 40+ engineers—the inflection point where merge queues become essential infrastructure
- ▸Private codebases experience 4.5x higher merge failures than open source, highlighting the complexity of large interdependent codebases
Summary
A comprehensive analysis of over 200,000 pull requests across 477 engineering teams reveals that code written with AI assistance breaks the main branch significantly less frequently than code written without it—1.9% versus 4.4%, according to data from Mergify's merge queue platform. This finding challenges widespread concerns about AI-generated code quality and suggests that AI tools may actually improve reliability by adhering to proven patterns and best practices more consistently.
The research also surfaced a critical secondary finding: the risk of production breaks scales dramatically with team size. Small teams of 2–5 engineers experience breakage on roughly 1 in 130 merges, but teams exceeding 40 engineers see failures on 1 in 8 merges—a 16x increase. Private codebases face 4.5x higher breakage rates than open-source projects (5.1% vs 1.1%), likely due to increased code interdependencies. The analysis found AI assistance already present in 1 of every 7 private merges, though this figure likely understates adoption since most tools leave no trace in pull request metadata.
- AI-assisted pull requests are undercounted in the data; at minimum 1 in 7 private merges involve AI assistance, but attribution is often absent
Editorial Opinion
The data fundamentally reframes how organizations should think about AI code generation tools—not as a shortcut to be scrutinized, but as a quality multiplier. While these tools aren't flawless, the finding that they reduce production breaks suggests they may be better at encoding institutional knowledge and best practices than individual developers. The deeper lesson for engineering leaders: once a team exceeds 15 engineers, a merge queue shifts from convenience to necessity. This data should inform hiring curves and infrastructure investment decisions as companies scale.
