If you have a domain behind Cloudflare, crawlers from OpenAI, Anthropic, Google and half a dozen more are reading your site every day - and until recently, deciding what to do about them was a pain: either you hit the “Block AI bots” button and blocked everything (breaking stuff you actually wanted), or you let everything through. Since July 1, 2026, Cloudflare has shipped a new taxonomy and new AI traffic controls for every plan, including Free, and you can finally decide with precision: I want to show up in search engines, but I don’t want models trained on my content.
Here’s where each option lives, what it actually does, and how to test it without breaking your real traffic.
The three categories you can now control
The underlying change is that Cloudflare stopped treating “AI bot” as a single thing. According to the Block AI Bots docs, bots are now classified by behavior:
- Search: crawlers that index your content to answer questions about it later. This is the traffic that sends visitors back to you.
- Agent: automation acting in real time on a person’s behalf: ChatGPT’s fetch bots (
ChatGPT-User), browser-driving agents, and so on. - Training: crawlers taking your content to train or fine-tune a model. This includes mixed-purpose crawlers that do search and training at the same time.
Each category is configured separately, with three options:
- Block (on all pages): blocks across the entire zone.
- Block on pages with ads: blocks only on pages where Cloudflare detects ads (the logic: where there are ads, human attention is what monetizes).
- Allow (do not block): blocks nothing.
Blocking applies both to verified bots classified with that behavior and to unverified bots that behave the same way - so a crawler doesn’t get a free pass just by not registering.
Where to configure it
Everything lives in one place: open your zone, go to Security > Settings, filter by Bot traffic and open Configure AI bot policies. The official docs confirm it. If you want to see it before diving in, here’s a screenshot of the panel as it looks right now, with the Search, Agent and Training options.
My recommended setup for a blog or content site: Training set to Block (all pages), Search set to Allow, and Agent set to Block if you don’t care about assistants reading your site in real time - or Allow if you want to show up in ChatGPT answers and the like. That’s an editorial call only you can make.
If you want to see another side of Cloudflare for working with content, I also explained how to scrape entire websites with Cloudflare in one call.
The big warning: defaults change on September 15, 2026
This is what most people will miss. According to Cloudflare, on September 15, 2026, new defaults kick in for newly onboarded domains: Training and Agent will be blocked on pages with ads, while Search stays allowed.
But there’s a second, trickier change: mixed-purpose crawlers that combine search and training (Cloudflare explicitly names Googlebot, Applebot and BingBot) will be blocked by any configuration that blocks Training, including the legacy “Block AI bots” toggle. In other words: if you turn on training blocking, Googlebot goes in the bag too.
Does that mean you disappear from Google? Not necessarily: Cloudflare’s managed robots.txt blocks Google-Extended (Google’s training token), not Googlebot, and Google keeps those two uses separate. But with the September change, if your policy is “block Training,” Cloudflare’s reasoning is that a bot doing both gets treated by its most restrictive use. If you don’t want that, you can opt out in Security > Settings before September 15 and confirm you want no changes to Training crawlers that also crawl for Search.
And heads up: the old “Block AI bots” toggle is deprecating on September 15, 2026, and it never covered mixed-purpose bots anyway. If you had it on, migrate to the new policies.
The Cloudflare managed robots.txt: the diplomatic layer
Besides the hard blocking, Cloudflare maintains a managed robots.txt. You enable it in Security > Settings, filtering by Bot traffic, under Set your preference to block training in robots.txt. Per the managed robots.txt docs - available on all plans - this is what gets prepended to your robots.txt (merging with yours if you already have one):
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# ... plus: Bytespider, CCBot, Applebot-Extended, meta-externalagentThe use=reference signal is new: it’s part of the Content Signals extension and means “you may index and link to me, but not reproduce me in full.” And let’s be clear about this: robots.txt is voluntary. It states your preference and works as a rights reservation (the text itself invokes Article 4 of the EU copyright directive), but a scraper with no scruples can ignore it. That’s why the right combo is robots.txt for the well-behaved + real blocking for the rest.
One thing that will pop up: Google Search Console may report Syntax not understood for the Content Signals lines. Cloudflare says it has observed no impact on crawling or SEO from those warnings. Ugly, but harmless.
How to test it without breaking real traffic
Before flipping hard blocks on, do this:
- Measure first. In Security > Events you can see challenged or blocked requests, and with Security Analytics you can filter blocked AI bot traffic. Turn on blocking and watch it there for a few days.
- Simulate a crawler with curl by swapping the user-agent. It’s the fastest way to confirm the rule bites:
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.2" https://yourdomain.com/If the Training policy is set to Block, you should get a 403. Repeat with a normal browser -A "Mozilla/5.0 ..." to confirm human traffic still gets a 200.
- Use “Block on pages with ads” as a middle step if your site monetizes with ads and a full block scares you: it only affects pages with ads.
- Don’t enable Bot Fight Mode blindly. It’s free and useful, but per its docs it issues computationally expensive challenges to known bot patterns, it may challenge API or mobile app traffic, and it cannot be bypassed with WAF rules: it runs in a separate pipeline where Skip, Bypass and Allow actions do nothing. If you need exceptions (your uptime monitor, your API clients), you want Super Bot Fight Mode (paid) or custom rules instead.
Reinforcing with Cloudflare custom WAF rules
If the panel falls short (you want to block one specific crawler, or challenge instead of block), WAF custom rules are available on the Free plan: 5 rules, all actions except Log, no regex. And if you want a self-hosted free WAF alternative, check out SafeLine WAF. A typical rule to challenge training crawlers by user-agent:
(http.user_agent contains "GPTBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "CCBot") or (http.user_agent contains "Bytespider")Action: Managed Challenge if you want to give legitimate clients a chance, Block if you want certainty. User-agent filtering is spoofable, obviously, but as a second layer on top of Cloudflare’s classification it does its job: it catches the ones that announce themselves and leaves the fine detection work to Cloudflare’s engine.
My recommendation, short version
Turn on the new policies today: Training blocked across the zone, Search allowed, Agent up to you, and enable the managed robots.txt. Then decide before September 15 what to do about mixed-purpose crawlers - because if you don’t decide, Cloudflare decides for you with the new defaults. Your content is yours; the free panel finally lets you act like it.
FAQ
Does blocking AI bots in Cloudflare hurt my Google rankings?
Not if you do it right: keep the Search category on Allow, and as far as Google is concerned the managed robots.txt only blocks Google-Extended, the training token - not the search Googlebot. The nuance arrives on September 15, 2026, when blocking Training will also block mixed-purpose crawlers like Googlebot unless you use the opt-out in Security Settings.
Is blocking AI bots in Cloudflare free?
Yes. The new AI Traffic policies (Search, Agent, Training), the managed robots.txt and Bot Fight Mode are all available on the Free plan. You also get 5 free WAF custom rules if you need tailored logic.
Is robots.txt enough to stop AI from training on my content?
No. Robots.txt compliance is voluntary: it states your preference and strengthens your legal position, but it doesn’t technically prevent access. That’s what Cloudflare’s blocking policies are for - they actually deny the request.
What do you think?
Leave your opinion, question or suggestion. Comments are synced with GitHub Discussions .