Documentation as a Planning Tool: Kapa's Research on AI Agent Behavior Reveals Surprising Knowledge Base Usage Patterns
Key Takeaways
- ▸Knowledge base search was the most-used tool in the AI agent, rivaling the combined usage of 30 native platform tools
- ▸Documentation served three distinct roles: as a fallback for questions native tools couldn't handle (32.1%), as context provider for native tool results (~7%), and as an internal planning mechanism for the agent
- ▸Agents use documentation not just to answer users, but to decide which native tools to call—a planning and reasoning function that was completely unexpected
Summary
Kapa, a company that builds customer-facing AI assistants on top of technical documentation, recently analyzed 1,192 conversations with an internal AI agent to understand how different tools were being used. The agent was equipped with 30 native tools for platform interactions (search_conversations, display_chart, etc.) and a single knowledge base search tool for reading documentation, FAQs, and API references. Surprisingly, the knowledge base search tool proved to be the most-used tool, called almost as frequently as all native tools combined, defying initial expectations that native tools would dominate.
Kapa identified three distinct patterns for knowledge base usage. First, documentation served as a fallback mechanism in 32.1% of conversations, answering product questions that native tools couldn't address—like setup instructions and integration comparisons. Second, in approximately 7% of conversations, the knowledge base tool complemented native tools by providing context: native tools answered "what is true" about a user's account, while documentation answered "what it means." Most intriguingly, Kapa discovered that the knowledge base tool was frequently used not to answer users directly, but to enable the agent to understand how to use its own native tools—acting as an internal briefing mechanism for agent planning and decision-making.
These findings challenge conventional assumptions about AI agent architecture and suggest that knowledge base search should be reconsidered as a core capability rather than a secondary feature. The research demonstrates that access to comprehensive documentation enables agents to be more intelligent about tool selection and provides a safety net for handling edge cases outside their native functionality.
- Product teams may be underestimating the value of comprehensive, accessible documentation in agent design
Editorial Opinion
This research offers a crucial insight for AI agent developers: knowledge base search should not be treated as an afterthought but as a first-class tool in agent design. The finding that agents use documentation for internal planning—to reason about which tools to invoke—suggests that well-structured knowledge bases can significantly improve agent decision-making beyond just providing user-facing answers. For companies building production AI agents, prioritizing documentation quality and searchability may yield better results than shipping more native tools.
