Cloudflare's Disallow AI Training switch: what changed on September 15, what got migrated, and the one setting that now removes you from Google
Cloudflare's post is written for a site owner choosing a setting. This page reads it from both sides. It translates the migration tables into plain language, names the exact case where Block now drops you out of Google, Apple and Bing search, flags the Bing gap that lasts until 2027, and checks the press claim that free-plan customers were switched to blocking today.
Read the original first: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
In one minute
- Today, September 15, Cloudflare turned on a Training setting called Disallow AI Training, on every plan. It tells Googlebot, Applebot and Bingbot to keep crawling for search but stop training on your pages.
- It works two ways at once: Cloudflare writes the no-training lines into your robots.txt through Bot Preference Sync, and it blocks every training crawler that is not on its Accountable list at the network edge.
- Accountable operators so far: Apple, Google and Microsoft for their mixed-use crawlers, plus Amazon, Anthropic, Meta and OpenAI, who already run separate search and training bots.
- The catch: Block and Block on pages with ads now apply to Googlebot, Applebot and Bingbot too. Pick Block for Training and you leave search.
- Existing zones were migrated to keep the effect you had. The old Block AI Bots switch became Search Allow, Training Disallow AI Training, Agent Block on pages with ads. Nobody was moved out of search.
- New ad-supported domains now get training disallowed and agents blocked on ad pages by default. New domains without ads get Allow on all three.
- Bing will not read the no-training preference from robots.txt until early 2027. For Bing, use the NOARCHIVE meta tag today.
The problem, in one paragraph
Googlebot, Applebot and Bingbot are mixed-use crawlers. One bot, one user agent, two jobs: it builds the search index, and the same pages feed AI training and AI summaries. A robots.txt file can allow or disallow a user agent. It cannot tell one user agent to do half its job. So for two years the choice for a site owner was: let the crawler train, or block it and vanish from search.
Cloudflare's numbers show how sites feel about that. Less than 1 percent of its sites block search bots. 17 percent have turned on something to block training. The demand was for a split that robots.txt alone could not express.
What Cloudflare built: a preference plus an enforcer
Cloudflare now sorts bot behavior into three buckets. Search is crawling to build an index. Training is crawling to train or fine-tune a model. Agent is a user-directed bot fetching a page for a person, such as a chat assistant's fetch bot or a browser-use agent. One bot can sit in more than one bucket.
The new setting lives in the Training row and is called Disallow AI Training. It is named for the Disallow: line it writes into your robots.txt. When you select it, two things happen.
- Bot Preference Sync publishes the no-training preference in robots.txt. For Google that is a Disallow for Google-Extended. For Apple it is a Disallow for Applebot-Extended. These are the operators' own opt-out tokens, and both companies have stated that opting out of training does not affect search ranking.
- Cloudflare blocks every other training crawler at the edge. That includes the training-only bots run by Amazon, Anthropic, Meta and OpenAI. Blocking those never touched search, because those companies already run separate search and training crawlers.
Cloudflare's own summary of the mechanism: 'we publish the preference, identify who is crawling, classify why they are crawling, and block the ones that ignore it, then report what each operator actually does on Radar.' The robots.txt line is the polite ask. The edge block is what happens to crawlers that do not honor it.
Who counts as Accountable, and what they had to promise
Disallow AI Training only lets a mixed-use crawler keep crawling for search if its operator is on Cloudflare's Accountable list. To get on the list, an operator must meet or commit to four things:
- A way for site owners to opt out of AI training, through robots.txt or a similar standard.
- A way to opt out of AI summaries, set with the operator directly now and through Cloudflare next year.
- URL-level visibility into which pages were made available for training, plus metrics on how content appeared in search.
- Assurance that opting out of training will not affect traditional search results.
Apple, Google and Microsoft are Accountable for their mixed-use crawlers. Amazon, Anthropic, Meta and OpenAI are Accountable because their training crawlers are already separate. Note the word 'commit'. Apple does not yet offer URL-level inspection; Cloudflare says Apple showed it an in-progress tool for next year. Google says its URL-level tools for Google-Extended are due 'in the weeks to come'. Cloudflare Radar is where it says it will track whether each operator delivers.
The part that bites: Block now means search too
Before today, the Block and Block on pages with ads options in the Training row did not apply to Googlebot, Applebot or Bingbot, because blocking them would have hurt search. Now that Disallow AI Training exists for the keep-search case, Cloudflare made Block mean what it says. From Cloudflare's post: 'If you want mixed-use crawlers gone entirely, you now have to say so. Select Block. It will stop Applebot, Bingbot, and Googlebot from reaching your site, search included.'
So the rule for the Training row is now simple. Disallow AI Training: no training, search stays. Block on pages with ads: all training crawlers, mixed-use included, blocked on pages Cloudflare detects as serving an ad, so Google may lose those pages from its index. Block: all training crawlers blocked everywhere, and Googlebot, Applebot and Bingbot go with them.
Cloudflare did not move any existing zone into Block. But if someone on your team opens the setting this week and picks the strongest-sounding option, the site leaves Google. That is the operational risk of today's change, and it is a human one.
What happened to your existing zones today
Cloudflare's answer to 'what do I need to do' is 'Nothing, in almost every case. Your current settings carry over on their own.' Here is what carrying over meant, from its two migration tables, in words.
If you never touched the newer Search, Training, Agent controls and only had the legacy Block AI Bots switch:
- Legacy switch off: Search Allow, Training Allow, Agent Allow. No change in effect.
- Legacy switch set to Block: Search Allow, Training Disallow AI Training, Agent Block on pages with ads.
- Legacy switch set to Block on pages with ads: the same result, Search Allow, Training Disallow AI Training, Agent Block on pages with ads.
If you had already configured the granular controls, each row keeps its practical effect, with one translation: a Training row set to Block or Block on pages with ads was moved to Disallow AI Training. Search and Agent rows kept their old values.
Read that second rule carefully. Under the old definitions, Training Block never reached Googlebot, so moving it to Disallow AI Training preserves what you actually had: search on, training off. Cloudflare chose to keep effect over label. That is the right call, and it is why the change is safe by default. It is only unsafe if someone re-selects Block after today.
Two legacy features are going away in the same move. Block AI Bots is deprecated in favor of the three rows. Managed Robots.txt is deprecated in favor of Bot Preference Sync, and customers who used it are being migrated.
New domains: the default flips, depending on ads
From today, onboarding a new domain offers one of two presets, chosen by whether the site makes money from ads.
- No ads: Preference Sync on, Search Allow, Training Allow, Agent Allow.
- Ads: Preference Sync on, Search Allow, Training Disallow AI Training, Agent Block on pages with ads.
Cloudflare's reasoning is that an ad only pays when a person sees the page. Training replaces the visit with an answer, and an agent fetches the page with nobody there to see the ad. You can change either preset during onboarding or later.
This is the piece the July coverage turned into 'Cloudflare will block AI crawlers on ad pages by default on September 15.' It is true for new ad-supported domains. It is not a change to existing zones, which brings us to the claim worth checking.
The free-plan claim, checked
TechCrunch's July 1 story, and most of the coverage that copied it, said the new defaults would apply to 'new Cloudflare customers, new sites set up by existing customers, and all existing free customers.' That last group is a lot of small business sites, including many an MSP hosts for clients.
Cloudflare's July 1 post says: 'For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads.' The only free-tier mention in that post is that the new controls are available to Free customers too. Today's post says existing customers' settings carry over and lists the migration tables above, with no free-plan exception.
We could not find the 'all existing free customers' line in either Cloudflare post. It may have come from a briefing, or it may be a misreading of 'available to Free customers.' Either way, the safe move is not to assume. Open the free zones you manage and look at the three rows.
The Bing gap
Microsoft is on the Accountable list on the strength of a commitment. Bing does not yet read a no-training preference from robots.txt. Cloudflare says Microsoft is building that for early 2027. Until then, from the post: 'selecting Disallow AI Training will not automatically convey a no-training preference to Bing through robots.txt.'
What works for Bing today is the NOARCHIVE meta tag on the page, which Microsoft says does not affect ranking, plus Bing Webmaster Tools' Block URLs or Content Removal tool for specific pages. So if keeping content out of Microsoft's training matters to a client, the Cloudflare switch alone does not do it yet.
If you build agents or fetchers
The Agent row has Allow, Block on pages with ads, and Block. There is no Disallow option for agents, because, in Cloudflare's words, 'the Internet does not yet have a well-established directive for expressing Disallow preferences to agents.' It points at the ai-prefs work as the standard to watch.
Practical meaning: a chat assistant's fetch bot or a browser-use agent hitting a new ad-supported Cloudflare domain is blocked on the ad pages by default from today. On existing zones it depends on what the owner set. The bots that keep working are the ones Cloudflare can identify and classify: honest user agents, published operator identity, robots.txt respected. Cloudflare's BotBase lets operators register a bot and declare its behavior from the dashboard. An unidentified fetcher is the one that gets treated as the worst thing it might be.
How to check a zone in two minutes
- Open the zone in the Cloudflare dashboard. Go to Security Settings, then Configure AI bot policies.
- Read the three rows. For most business sites that want search and no training: Search Allow, Training Disallow AI Training, Agent your call.
- If Training says Block, confirm the owner wants to leave Google, Apple and Bing search. If not, change it to Disallow AI Training.
- Confirm Bot Preference Sync is on, then load /robots.txt on the site and look for the Google-Extended and Applebot-Extended lines.
- If Bing training matters, add NOARCHIVE to the relevant pages.
- Note what each Accountable operator has actually shipped on Cloudflare Radar before you promise a client the opt-out is fully in force.
Who is affected
| Case | Status |
|---|---|
| Existing zones with the legacy Block AI Bots switch on | Migrated today to Search Allow, Training Disallow AI Training, Agent Block on pages with ads. Search unchanged. |
| Existing zones that set Training to Block under the old controls | Moved to Disallow AI Training. Same effect as before: training off, search on. Re-selecting Block now removes search crawlers. |
| New ad-supported domains | Default preset disallows training and blocks agents on ad pages. Changeable at onboarding. |
| New domains without ads | Default preset allows Search, Training and Agent. |
| Sites that want out of Bing training | Not covered by the switch until early 2027. Use NOARCHIVE and Bing Webmaster Tools. |
| Agent and fetcher operators | Blocked on ad pages of new ad-supported domains by default. No Disallow directive for agents yet. |
| Free-plan customers | Cloudflare's posts describe no free-plan-specific change. Press said otherwise in July. Check the zone. |
What to do
- Open every Cloudflare zone you manage this week and read the Search, Training and Agent rows. Fix anything that says Block on Training unless leaving search is the intent.
- Turn on Bot Preference Sync and confirm the robots.txt now carries the Google-Extended and Applebot-Extended lines.
- Tell whoever touches Cloudflare on your team, in one sentence: Block on Training now removes the site from Google.
- For clients who care about Bing, add NOARCHIVE to the pages in question and note the 2027 date.
- If you run a fetch bot or agent, make sure it identifies itself honestly and respects robots.txt, and consider registering it in BotBase.
- Do not sell a client 'your content is out of AI training' on the strength of this switch alone. It covers Accountable operators and blocks the rest at Cloudflare's edge. Crawlers that never touch Cloudflare are outside it.
What is still unknown
- How Cloudflare detects that a page serves an ad. Its docs say 'automated detection' and give no mechanism, so Block on pages with ads has an edge we cannot see.
- Whether the 'all existing free customers' claim in July coverage came from a Cloudflare briefing or a misreading. Neither Cloudflare post contains it.
- What 'certain new domains' means in the line 'Disallow AI Training will become part of the recommended configuration for certain new domains.' The presets suggest it means ad-supported ones.
- Whether Apple and Google deliver the URL-level transparency tools on the timelines Cloudflare relays. Both are commitments, not shipped features.
- How Cloudflare's block presents to a blocked agent. The posts say blocked; they do not say whether that is a 403, a challenge, or something else.
- The 1 percent and 17 percent figures are Cloudflare's own measurements of its own customers.
Sources
- Cloudflare, 'Have it both ways: stay discoverable in search while disallowing AI training' — the original report
- Cloudflare, July 1, 'Your site, your rules: new AI traffic options for all customers'
- Cloudflare docs, Block AI Bots and the Search, Training, Agent controls
- Cloudflare, August 21, 'Say it once: introducing Bot Preference Sync'
- Cloudflare, August 28, 'BotBase for Operators'
- TechCrunch, July 1, the report that said all existing free customers get the new defaults