May 1, 2024Confirmed
AI crawler pulls 73 TB in a month and sticks Read the Docs with a $5,000 bandwidth bill
AI loose in the worldAvailabilityDERIVED
Approved by person, Oct 11, 2026. How entries are approved
Read the Docs, which hosts documentation for open-source projects on limited sponsored AWS and Cloudflare plans, disclosed in July 2024 that AI crawlers were repeatedly downloading large zipped documentation files hundreds of times a day. One crawler pulled 73 TB of zipped HTML in May 2024, almost 10 TB in a single day, costing the nonprofit over $5,000 in bandwidth charges. Blocking AI crawlers cut downloaded-file bandwidth by about 75 percent, from roughly 800 GB to 200 GB per day, which the team estimated at about $50/day or $1,500/month in origin bandwidth.
What this means for youIf you run a website or documentation for a project, AI crawlers can quietly drive up your hosting bill. Check your bandwidth usage and block or rate-limit known AI crawlers in your firewall or CDN settings.
Sources
- Read the Docs bloghttps://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/
Incident records from the AI Agent Incident Database by sipi.bot (https://sipi.bot/incidents/), licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
Source data license: CC BY 4.0. Attribution to sipi.bot is required.
Covered by snapshot 45
da93f03b406f409940d0a6ef47b526607f82ebf457d98f69299b88aceede576a
This entry is record number 2385. It has not been edited since it was approved; the hash above would not match if it had. How to verify