JetBrains Tests Caveman AI Agent Skill: Actual Token Savings Fall Short of 65% Marketing Claims
Key Takeaways
- ▸Caveman skill delivers ~8.5% token savings on real agentic tasks, not the advertised 65%, because code and tool outputs remain untouched
- ▸Cost savings align with token savings at approximately 10%, with no statistical degradation in task success rates across 82 paired tasks
- ▸Marketing claims of 65% apply to chat-style Q&A prose, not to the code-heavy output of AI coding agents
Summary
JetBrains conducted rigorous testing of the popular "Caveman" skill, which promises to reduce AI agent token usage by compressing verbose narration into terse, caveman-like speech. The skill's creators claim 65% token savings, but JetBrains' real-world testing on 86 paired agentic tasks found actual savings of only 8.5%, or about 10% in cost reduction.
The discrepancy stems from how AI agents function differently than chat interfaces. While Caveman correctly compresses explanatory text between tool calls, the majority of tokens in agentic workflows are consumed by code snippets, file diffs, tool invocations, and exact error strings—none of which the skill modifies. As a result, the skill's compression primarily affects narration rather than the core output that dominates agent sessions.
Despite the modest real-world savings, JetBrains found no detectable quality degradation across their testing. Task success rates were statistically indistinguishable between the standard and Caveman-enabled runs (sign test p = 0.82), meaning the terse style transfer did not harm the agent's ability to complete complex, multi-step coding tasks. The researchers concluded the skill is safe and genuinely saves cost, just not at the advertised scale.
- Skill remains safe to use if adopted for style preference, but actual savings are modest compared to marketing claims
Editorial Opinion
This research exemplifies a key friction in AI tooling: marketing claims based on one use case (chat Q&A) don't transfer to another (agentic coding). JetBrains' willingness to publish unflattering findings—debunking a popular tool's marketing rather than simply adopting its claims—sets a high bar for vendor testing rigor. The 8.5% savings isn't trivial at scale, but honest measurement matters more than inflated promises.
