Accept-Language Redirects May Be Blocking AI Crawlers and Search Engines, New Research Warns
Key Takeaways
- ▸Most search engine bots don't send Accept-Language headers, but many AI crawlers send browser-like defaults (typically en-US,en;q=0.9) that don't reflect user intent
- ▸Redirecting HTML based on Accept-Language can skew indexing toward the wrong locale and reduce coverage of non-English content for both search engines and AI systems
- ▸AI crawlers don't adapt Accept-Language based on prompt language, user locale, or conversation context—the header is simply an implementation default
Summary
New research from Merj reveals that locale-adaptive redirects based on the Accept-Language HTTP header could be inadvertently blocking AI crawlers and reducing search engine indexing quality. While traditional search engine bots like Googlebot don't send Accept-Language headers in crawl requests, many emerging AI crawlers send browser-like defaults (typically "en-US,en;q=0.9") that don't reflect actual user intent. This disconnect means websites using Accept-Language-based redirects may funnel AI bots into incorrect locales, limiting coverage of non-English content.
The research tested major search engines, AI platforms, and web-access LLMs to examine how they fetch HTML documents and handle redirect chains. Results showed that AI crawlers consistently failed to adapt Accept-Language headers based on prompt language, user locale, or conversation context—the header values were simply implementation defaults inherited from underlying browser engines. This creates a problematic scenario where automated systems are treated as English-speaking users regardless of actual use case.
The findings represent a significant shift from previous SEO best practices, which primarily warned against IP-based geolocation redirects. According to Google's documentation on locale-adaptive pages, the search giant has long recommended separate locale URLs with hreflang annotations instead of dynamic redirects. However, Accept-Language wasn't historically a concern because major search engines simply didn't send the header. The proliferation of AI crawlers has changed this calculus, introducing new edge cases that can impact both traditional search indexing and LLM-based retrieval systems.
The research particularly highlighted that even Googlebot can end up presenting default Accept-Language headers when following redirects during rendering, potentially creating indexing issues for multi-language sites. Notable exceptions included Applebot and PetalBot, which demonstrated different behavior based on custom internal logic. The practical recommendation is clear: websites should avoid redirecting HTML content based on Accept-Language headers to ensure proper discoverability across both traditional search engines and emerging AI retrieval systems.
- Even Googlebot can present default Accept-Language headers when following redirects during rendering, creating potential indexing complications
- Best practice remains using separate locale URLs with hreflang annotations rather than dynamic redirects based on language detection
Editorial Opinion
This research highlights an important blind spot in how websites handle the emerging AI crawler ecosystem. While the traditional SEO playbook correctly avoided IP-based redirects, Accept-Language seemed harmless because major search engines didn't use it. The explosion of AI crawlers—many built on browser automation frameworks—has inadvertently reintroduced this old edge case with new consequences. The finding that these crawlers don't contextualize language headers is particularly revealing about the maturity gap between traditional search infrastructure and newer AI retrieval systems.
