Third-Party Testing Reveals rtk Skill Fails to Deliver on Advertised Token Savings
Key Takeaways
- ▸rtk advertised 60–90% token savings but measured results showed +7.6% higher costs at low reasoning effort and ±0% at high effort
- ▸File reads via Claude Code's built-in Read/Grep tools bypass rtk's hook entirely, severely limiting the addressable token pool
- ▸Only 33% of bash commands are eligible for rtk's compression, and they account for under 20% of tool-result characters in real sessions
Summary
A comprehensive technical benchmark of the "rtk" (Rust Token Killer) skill—a CLI proxy designed to reduce Claude Code token usage by compressing command output—has revealed significant discrepancies between advertised and actual performance. The rtk tool claimed 60–90% token savings by intercepting bash commands and returning compressed output (e.g., condensing git status from eleven lines to a single-line summary), but independent testing using SkillsBench and claude-sonnet-5 showed the opposite: a +7.6% token cost increase at low reasoning effort (p=0.004) and no measurable change at high effort.
The analysis identified two structural limitations explaining the gap. First, Claude Code's built-in tools (Read, Grep) bypass the rtk hook entirely, meaning file operations—a major component of agent work—are never compressed. Second, only 33% of bash calls are eligible for rtk's rewriting, and those carry just under 20% of tool-result characters in typical sessions, capping rtk's theoretical savings at approximately 3% of input tokens. The compression itself was technically sound, but the addressable problem proved much smaller than the marketing claims suggested. Task quality remained unchanged in both arms, ruling out correctness regressions as a confounding factor.
- Structural limits cap rtk's theoretical maximum impact at ~3% of total input tokens, far below advertised savings claims
- Task quality and correctness were unaffected in both test arms, suggesting no hidden trade-off between compression and reliability
Editorial Opinion
The rtk test case is a reminder that even technically sound compression strategies have hard structural limits when applied to real agent workloads. The tool's developers were correct about the compression ratio on isolated command output—the math checks out—but incorrect about its relevance to end-to-end token bills. Most agent bytes flow through channels (built-in tools, file reads, context re-use) that no bash-level hook can touch. Marketing token-saving tools without first measuring what fraction of a real session's tokens are actually compressible sets a false expectation.
