<p>AI crawlers have downloaded over 100 TB in the last couple months, partially from bugs and abusive behavior. We ask AI crawlers to include rate and bandwidth limits, and ask them to partner with us to mitigate impact and support our work.</p><p><a href="https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/" target="_blank" rel="nofollow noopener noreferrer" translate="no"><span class="invisible">https://</span><span class="ellipsis">about.readthedocs.com/blog/202</span><span class="invisible">4/07/ai-crawlers-abuse/</span></a></p>