AI Crawler Activity Reaches 68.9M Monthly Visits: OpenAI Dominates Real-Time Content Retrieval
Key Takeaways
- ▸AI crawler activity has achieved mainstream scale with nearly 69 million monthly visits across over 850,000 websites, with 59% of analyzed sites receiving crawler visits
- ▸OpenAI's dominance is overwhelming, controlling 81% of all AI crawler requests, fundamentally shaping how content is discovered and accessed through AI systems
- ▸Real-time answer generation has become the primary use case (56.9% of crawler activity), displacing traditional index-building as AI systems retrieve current content to answer user queries in real time
Summary
A comprehensive analysis of 858,457 websites on the Duda platform reveals that AI crawler activity has reached significant scale, with 68.9 million visits in February 2026 and 59% of analyzed sites receiving at least one AI crawler visit. OpenAI dominates the landscape, accounting for 81% of all AI crawler requests (55.8 million visits), followed distantly by Anthropic's Claude at 16.6%. The data shows a dramatic shift in how AI systems interact with the web: 56.9% of crawling activity is now tied to real-time answer generation for user queries rather than training data collection (28.8%) or content indexing (14.3%).
Year-over-year growth in AI referral traffic has been substantial, with total LLM referrals increasing 72.7% and Claude experiencing explosive 23x growth. ChatGPT continues to be the primary driver, responsible for nearly all real-time content retrieval activity. The analysis also reveals a strong correlation between allowing AI crawlers and receiving human traffic, with crawled sites averaging 527.7 sessions compared to 164.9 for non-crawled sites, suggesting that AI-generated visibility is becoming an increasingly important traffic driver for web publishers.
- Significant correlation exists between allowing AI crawlers and increased human traffic, with crawled sites receiving 3.2x more sessions on average, positioning AI visibility as a legitimate SEO consideration
Editorial Opinion
This analysis underscores OpenAI's near-monopolistic position in AI-powered search and discovery, raising important questions about market concentration and web publisher dependency on a single platform for visibility. While the correlation between AI crawler access and human traffic is compelling, the causation remains unclear—it's unclear whether sites benefit from AI crawling itself or whether already-successful sites are simply more likely to be crawled. As Google enters the market with its new Google-Agent crawler, the current landscape could shift dramatically, potentially reshaping which companies control access to web content.
