Cloudflare Gives AI Crawlers Until September 15 to Pay Up or Get Blocked

One page crawled, one visit sent back: that used to be the deal. Not anymore, and the numbers prove it: Anthropic's bot returns one referral for every 73,000 pages it scrapes, compared to roughly one in 14 for Google. Cloudflare just decided enough is enough.

Cloudflare's new AI crawler rules, explained

Starting September 15, 2026, Cloudflare's default settings will block "mixed-use" crawlers from any pages that host ads. These are bots that scrape a site for search indexing, AI training, and AI agent use all in one pass, without letting the site owner pick and choose.

The new rule hits new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Search crawlers still get in by default. Training and agent bots get shown the door on any ad-supported page, unless the owner flips the switch back on.

Why Google is the real target

Cloudflare never says the word Google out loud, but nobody's fooled. Googlebot handles search indexing and feeds Gemini, AI Overviews, and AI Mode at the same time, and Google lets websites opt into a separate crawler that only crawls for traditional search results, but if a publisher wants to appear in AI Mode without training Google's models, there's no such option.

That's the loophole Cloudflare is closing. If Google wants search visibility without losing training access, it will now need to prove its bots actually separate the two jobs.

Getting paid, not just crawled

Cloudflare is also rebranding its old Pay Per Crawl tool into Pay Per Use, and the shift matters: instead of billing every single fetch, site owners get paid when their content actually shows up inside an AI chatbot's answer. Launch partners Ceramic.ai and You.com are first in line, and Cloudflare says more AI companies can join the program.

Why this Cloudflare AI crawler policy actually matters

Cloudflare sits in front of roughly 20% of all internet traffic, which turns this into an internet-wide default that individual publishers could never enforce alone. For small blogs and newsrooms scraped thousands of times a day for nothing in return, that's a genuine win.

For OpenAI, Anthropic, and every startup building agentic browsers, it's a two-month countdown to negotiate, pay, or lose access to a fifth of the web. Google, ironically, might have the trickiest homework of all.