Kapa Cuts RAG Query Costs by One-Third with Smart Context Pruning
Key Takeaways
- ▸Kapa engineered a small LLM pruning layer that removes 68% of unnecessary RAG context while retaining 96% recall
- ▸Retrieved chunks represent the largest cost driver in their pipeline (two-thirds of query cost), making pruning high-ROI
- ▸The solution outperforms fixed-score cutoffs by evaluating chunk relevance holistically—recognizing that some chunks are only valuable in combination with others
Summary
Kapa, an AI assistant platform for answering questions over complex product knowledge bases, has developed a novel approach to optimize the expensive retrieval-augmented generation (RAG) pipeline. The company inserted a pruning layer between its retriever and generator steps—a small, cost-effective LLM that analyzes retrieved context chunks and removes those unnecessary for answering the user's question. The approach drops approximately 68% of context while maintaining 96% recall, cutting overall query costs by roughly one-third.
The problem Kapa addressed is fundamental to RAG systems: retrievers deliberately prioritize recall over precision, returning many candidate chunks to ensure the answer exists somewhere in the context. However, every chunk passed to the expensive generator incurs a cost, even if unused. In Kapa's pipelines, retrieved chunks represent about two-thirds of query costs—more than the answer generation, conversation history, and system prompt combined. Traditional solutions like fixed reranker score cutoffs fail because they cannot account for chunk interdependencies; some chunks are only relevant when combined with others.
Kapa's solution avoids the pitfalls of pointwise scoring by evaluating chunks collectively. The small pruning LLM reads both the question and all retrieved chunks together, identifying which ones contribute to the final answer. This approach recognizes that relevance is not a property of individual chunks in isolation, but of chunk sets that collectively answer the question. The breakthrough has broad implications for cost-sensitive RAG deployments, particularly as multi-agent systems grow their context windows.
- Query costs drop by approximately one-third, enabling better scaling of AI assistants over complex product knowledge bases
Editorial Opinion
This is an elegant engineering solution to a real problem in production RAG systems. Rather than throwing more expensive models at the retrieval problem, Kapa leveraged a cheap resource—a smaller LLM doing simple analysis—to dramatically improve cost efficiency. The framing of relevance as a property of chunk sets rather than individual chunks is insightful and suggests that many RAG optimizations will come not from better retrievers, but from smarter filtering between retrieval and generation.
