Fireworks AI Benchmark: Agent Failures Stem From Execution Reliability, Not Intelligence
Key Takeaways
- ▸Agent failures are caused by execution reliability (structured output malformations), not raw intelligence—the 'Agent Execution Tax' wastes up to 22.9% of inference on failed retries in multi-step loops
- ▸Cost-per-token is misleading for agentic AI; true deployment cost depends on reliability-adjusted success rates—a cheaper model can cost $40k+ annually more at production scale when retries and failures compound
- ▸Best-reasoning models don't always win in agent deployments; those with consistent structured output and stable latency under repeated loops outperform higher-intelligence alternatives
Summary
A comprehensive benchmark from Fireworks AI and Notte testing 720 browser automation tasks across four LLMs reveals that agent failures stem not from model intelligence but from execution reliability — specifically, structured output consistency. The study introduces the 'Agent Execution Tax,' quantifying waste from malformed JSON outputs requiring retries. The worst-performing model exhibited a 22.9% execution tax compared to zero for the best, costing over $40,000 annually in wasted inference at typical production volumes (10,000 tasks/day).
The research tested three leading models with distinct profiles: GLM-5 (best accuracy but highest cost for compliance workflows), MiniMax M2.5 (best value for scaled production), and Kimi K2.5 (fastest inference with zero execution overhead for customer-facing agents). The findings challenge conventional wisdom that superior reasoning capabilities guarantee better agent performance, demonstrating instead that reliability and structured output consistency matter more in production systems.
The research highlights that inference infrastructure—not the model alone—shapes execution reliability through structured output consistency, latency predictability, and stable performance under repeated loops. Cost-per-token metrics mask true deployment costs, as models with higher per-task success rates can be significantly cheaper at scale despite higher nominal token prices.
Editorial Opinion
This research reframes how the industry evaluates LLMs for agentic AI. The focus on execution reliability over raw intelligence reflects the field's maturation—as agent frameworks standardize, success depends less on breakthrough capabilities and more on engineering discipline around structured outputs and infrastructure reliability. For enterprises building production agents, this benchmark should shift procurement priorities away from leaderboard rankings toward real-world deployment readiness metrics. The $40k annual waste figure at modest scale suggests that reliability is not a nice-to-have; it's the primary cost driver in agent economics.
