Zalando Deploys AI-as-Judge Framework to Automate Search Quality Assurance Across New Markets
Key Takeaways
- ▸Zalando's LLM-as-a-judge framework achieves high correlation with human judgment and enables automated, scalable search quality evaluation across multiple languages and markets
- ▸The framework shifts search quality assurance from reactive (post-launch) to proactive (pre-launch) by allowing comprehensive testing of new market search systems before user traffic arrives
- ▸Automated test generation, semantic query clustering, and NER-based attribute extraction reduce reliance on manual human expertise while maintaining broad test coverage across diverse search scenarios
Summary
Zalando has implemented an LLM-as-a-judge framework for search quality assurance, enabling the company to proactively evaluate search result relevance at scale with multi-language support. The approach addresses a critical challenge faced during the company's 2025 expansion into three new countries—Luxembourg, Portugal, and Greece—where traditional manual quality assurance processes would be inefficient and reactive. Previously, search quality validation relied heavily on human experts manually testing translated queries and annotating errors, a method that could not identify issues before launch when user signals were unavailable.
The new framework automates the evaluation process while adhering to key principles including high test coverage across product categories and search scenarios, automated test generation to avoid handcrafted cases, multi-language support, and reproducibility for verification after fixes. By leveraging real search queries from existing markets, clustering semantically similar queries, and employing Named Entity Recognition to extract attributes like product names, brands, colors, and sizes, Zalando has created a scalable, data-driven approach that shifts quality assurance from reactive (post-launch issue identification) to proactive (pre-launch validation). This capability significantly reduces the dependency on manual human expertise while ensuring new market launches maintain high search quality standards.
Editorial Opinion
Zalando's application of LLM-as-a-judge for pre-launch search quality validation represents a pragmatic use of AI to solve a genuine operational challenge—ensuring search experience quality in new markets before they go live. This approach demonstrates how large language models can effectively replicate expert human judgment at scale, particularly valuable in scenarios where ground truth from real users is unavailable. The framework's emphasis on reproducibility and multi-language support makes it a replicable model for other e-commerce platforms expanding internationally.
