How AI Agents Spend Your Money: Study Reveals 1000x Token Consumption Differences Between Models

Summary

A new arXiv research paper presents the first systematic analysis of token consumption patterns in agentic coding tasks, examining how eight frontier LLMs spend tokens when performing complex coding work. The study analyzed trajectories on the SWE-bench Verified benchmark and found that agentic tasks are uniquely expensive, consuming roughly 1000x more tokens than simple code reasoning or chat tasks, with input tokens rather than output tokens driving the overall cost.

The research reveals striking inefficiency and unpredictability in model behavior: identical tasks can require vastly different token budgets (varying by up to 30x across runs), and surprisingly, higher token consumption does not translate to higher accuracy—accuracy often peaks at intermediate costs and plateaus thereafter. Token efficiency varies dramatically between models: Anthropic's Claude-Sonnet-4.5 and Kimi's K2 consistently outperform alternatives, consuming over 1.5 million fewer tokens than OpenAI's GPT-5 on the same tasks.

Perhaps most concerning, the study found that frontier LLMs fundamentally fail at predicting their own token costs, with correlations as weak as 0.39, and systematically underestimate real expenses. The research also reveals a critical gap between human-perceived task difficulty and actual computational effort: expert human ratings only weakly correlate with observed token consumption, suggesting agents tackle problems in ways humans don't anticipate.

• Higher token consumption does not improve accuracy—performance often peaks at intermediate costs and saturates at higher token budgets
• Frontier models systematically fail to predict their own token usage (correlations up to 0.39) and underestimate real costs, revealing a fundamental gap in their self-awareness

Editorial Opinion

This research exposes a critical blind spot in the AI industry: we're deploying agents at scale without understanding—or even being able to predict—their true economic costs. The finding that advanced models systematically underestimate their own token consumption is particularly troubling, suggesting that cost projections for agent-based systems may be fundamentally unreliable. For companies like Anthropic, the positive positioning of Claude-Sonnet-4.5 as a token-efficient option has real market implications, but the broader insight is sobering: the industry lacks basic self-knowledge about its own resource consumption. Until models can accurately forecast their costs and teams can reliably predict agent behavior, deploying AI agents in production remains a high-risk financial proposition.