Have It Both Ways: Stay Discoverable in Search While Disallowing AI Training
Cloudflare, Tuesday, September 15th, 2026
Cloudflare lets sites stay indexed for search while blocking the same crawlers from using content for AI training.
Cloudflare addresses the bind facing publishers whose content is crawled by bots that serve both search indexing and AI training.
The post describes controls that let site owners remain discoverable in search results while denying permission for their content to be used as model training data.
It covers how Cloudflare identifies mixed-use crawlers and holds operators accountable to declared purposes.
The company frames this as restoring meaningful choice to content owners who previously faced an all-or-nothing decision.