AI Bot Traffic Overwhelms Independent Mapping Sites, Forces Developer to Implement Crawler Blocks

Summary

Independent developer Gary Gale, creator of the Vaguely Rude Places Map and WOEplanet, has been forced to implement aggressive bot-blocking measures after AI crawlers from major tech companies consumed his entire monthly map tile allowance in under 24 hours. On February 1st, traffic to his sites surged from a daily average of 8,300 hits to 335,000 hits, with crawlers from Meta, OpenAI, ByteDance, Amazon, Apple, Anthropic, and Huawei repeatedly requesting the same URLs or systematically walking through identifier sequences.

The sudden influx exhausted Gale's generous 200,000 monthly map tile allowance from Stadia Maps, rendering the sites' maps largely useless until the monthly reset. The developer has since switched from Stamen's Toner map tiles to Carto's Positron tiles and placed all sites behind CloudFlare with AI Crawl Control enabled and LLM-related robots.txt files implemented. These measures have reduced traffic by 75%, though some bots—particularly Microsoft's BingBot—continue to ignore robots.txt directives.

The incident highlights growing tensions between independent web developers and AI companies whose data-hungry crawlers can inadvertently overwhelm smaller sites. While the sites remain operational, Gale expressed that he is "mourning the death of my little section of the open internet," noting that the measures necessary to keep his projects sustainable have fundamentally changed how they operate. The case underscores broader concerns about the infrastructure costs AI training imposes on the open web ecosystem.

• The incident illustrates the infrastructure burden AI training places on independent web developers and the broader open internet ecosystem