09/04/2026
For 30 years, the web ran on a quiet agreement. Search engines could crawl websites for free. In return, they sent traffic back. Publishers got readers, search engines got content, and the arrangement held. AI has broken it.
Over half of all web traffic is now AI bots, according to Cloudflare, which manages more than 30 percent of the world's top sites. Most of that traffic comes with no links back, no readers sent, and no revenue generated for the sites being scraped. So publishers are starting to block AI crawlers. And this is where the dynamic turns self-defeating.
Sites spreading misinformation are far less likely to block AI crawlers than reputable ones. As high-quality publishers opt out, AI summaries draw from a shallower and less reliable pool. One recent study found that roughly one in six sources already used by AI search tools is itself an AI-generated website. There's a term for what happens when AI trains increasingly on AI-generated text: model collapse.
On September 15, Cloudflare begins blocking AI crawlers by default on advertising-supported pages. Up to 30 percent of the world's top sites will stop appearing in AI search summaries almost overnight.
The quality of AI-generated answers is likely to get worse before any new economic arrangement stabilizes. What that means for anyone who relies on the web for accurate information is worth thinking through.
Read the full story, link in comments.