Developer Teaches AIs to Use SDKs: Testing Shows AI and Human Developer Experience Are Fundamentally Different
Key Takeaways
- ▸AI agents and humans require completely different SDK design approaches—agents scan for patterns and fabricate plausible values where humans would read documentation and follow setup instructions
- ▸When multiple AI agents independently make the same mistake, it's a signal of poor API design rather than a user error; consistent failures point to systemic problems that should be fixed in the tool itself
- ▸AI agents have no intuition about computational cost and will call any available function repeatedly in hot loops; all interfaces should be designed as idempotent and cheap or risk being called thousands of times per second
Summary
A software developer tested an SDK with 30 different AI agents and discovered they all failed in remarkably consistent ways—revealing critical insights about building tools for AI users versus humans. Rather than treating these failures as bugs, the developer identified systemic design problems: AI agents fabricate plausible-looking placeholder values instead of reading instructions, converge on identical wrong implementation patterns independently, and have no intuition about computational expense, calling functions in hot loops without hesitation.
The breakthrough came when the developer realized the agents that broke the SDK were best positioned to fix it. By pointing failing agents at the production codebase, they successfully debugged and implemented fixes to the SDK itself, including redesigning placeholder formats, making functions idempotent and cheap to call repeatedly, and adding CLI command aliases that matched agent expectations. This flip—from SDK users to SDK maintainers—proved so effective that some agents accumulated context across dozens of fixes.
The experience reveals that AI developer experience (DX) and human DX are fundamentally different problems requiring distinct design approaches. The findings have immediate practical implications for anyone building APIs, SDKs, CLIs, or other tools intended to be used by AI agents: abandon fillable-blank placeholders in favor of file references, treat consistent agent mistakes as design flaws rather than user errors, and architect all interfaces assuming they will be called in unexpected and expensive ways.
- Failing AI agents can serve as effective debuggers and contributors when given access to source code—they've experienced problems firsthand and understand root causes better than developers working in isolation
Editorial Opinion
This report is a fascinating inversion of how we typically think about developer tools. Rather than dismissing AI agents as incompetent users, the developer treated consistent failure patterns as design feedback—a philosophy that could fundamentally improve how tools are built not just for AI, but for all users. The insight that 'if every agent guesses the same wrong command, the command isn't wrong' is a useful antidote to developer arrogance. Most crucially, using AI agents as both QA and development partners suggests a future where human-AI collaboration on infrastructure and tooling becomes standard practice.
