Harvey Launches LAB-AA: First Real-World Legal AI Agent Benchmark
Key Takeaways
- ▸Claude Fable 5 achieves 14.2% all-pass rate on Harvey LAB-AA, significantly outperforming other models and demonstrating the performance advantage of frontier reasoning models in legal AI tasks
- ▸Benchmark tests 120 real-world legal tasks across 24 practice areas with scoring based on professional legal standards, establishing a credible evaluation framework for agentic legal AI
- ▸Frontier legal AI remains nascent: even the best-performing model completes only 14.2% of tasks fully, indicating substantial work ahead before AI agents can reliably handle complex professional legal deliverables
Summary
Harvey has announced LAB-AA (Legal Agent Benchmark), a comprehensive evaluation framework testing AI language models on 120 real-world legal tasks spanning 24 practice areas including M&A, capital markets, tax, litigation, and bankruptcy. The benchmark grades models against binary criteria reflecting professional legal standards, using an all-pass rate metric that measures whether models fully satisfy task requirements.
Claude Fable 5 leads the benchmark with a 14.2% all-pass rate when using Opus 4.8 as a fallback, nearly double the next-best performers (Claude Opus 4.8 and GLM-5.2 at 7.5% each). The benchmark reveals the substantial gap remaining in legal AI: even the top-performing model leaves approximately 86% of professional deliverables incomplete, with 13 of 28 models evaluated failing to fully pass any tasks at all. Only four models exceed 90% on individual criterion pass rates.
Beyond raw performance, Harvey provides detailed metrics on cost per task, output token usage, execution time, and the number of agent turns required per task. These insights reveal the practical tradeoffs between model capability and resource efficiency in legal AI applications, offering practitioners concrete data for selecting models for real-world deployment.
- Comprehensive metrics on cost, execution time, and agent efficiency reveal significant practical tradeoffs—enabling practitioners to balance capability against resource consumption for specific use cases
Editorial Opinion
Harvey LAB-AA represents a meaningful step toward rigorous evaluation of legal AI agents at the frontier. By grounding assessment in actual professional deliverables rather than proxy metrics, the benchmark provides genuine insight into whether AI models can handle real legal work. However, the stark finding—that even leading models complete fewer than 15% of tasks—underscores how far the field remains from deploying autonomous agents on complex legal matters. This benchmark will likely become a key reference point as the industry matures.
