robots.txt AI training opt-outs just got a shared syntax. On October 30, 2026, Google, Microsoft, OpenAI and Anthropic each said they would honor one policy token, __usage_ai-training, so a site can state a single training preference instead of writing a rule for every lab’s crawler.
The catch is timing. Only Microsoft has a firm start date, and that is February 1, 2027.
The token comes from an IETF Internet-Draft written by Anthropic’s Kevin Kelley, and the datatracker is blunt that it “is not endorsed by the IETF and has no formal standing.” It still matters, because the four companies whose crawlers dominate most server logs have now agreed on what it means.
I read the draft, all four vendor statements and each lab’s crawler documentation. Below is the syntax, the support timeline, the parts it deliberately does not cover, and the robots.txt I would actually ship today.
What Is the __usage_ai-training Token?
The __usage_ai-training token is a fake user agent that carries a usage preference instead of naming a crawler. You write it in a normal robots.txt group, and Disallow means “do not use this content to train generative AI models.”
Here is the draft’s own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That says: do not train on anything, except the press kit, which you are happy to have models learn from.
The draft’s selling point is that it “introduces no new directive names and no new grammar.” Path matching, longest-match precedence and group structure all work exactly as RFC 9309 already defines them. Any tool that parses robots.txt today can parse this.
Three rules in the draft are easy to miss:
User-Agent: *does not count. A wildcard group must not be read as a preference for the token. If no group names__usage_ai-training, you have expressed no preference at all.- It is not a crawl directive. The draft says the token “is not a directive for crawling behavior.” Crawlers may still fetch your pages. If you want to stop the fetch, you block the crawler’s own product token.
- Specific beats general. Vendors may treat their own tokens, such as Google-Extended, as more specific than the shared one, and private licensing agreements can override it.
The definition of “AI training” is borrowed from the AIPREF vocabulary draft: using an asset to modify the learned parameters of a generative AI model. That covers training, not answering questions with your content.
The token answers one question, “may you train on this?”, and nothing else.
AI Training Opt-Out Support: Who Honors It and When
All four statements went out on October 30, but they are not equally concrete.
| Provider | Shared token support | Existing training control |
|---|---|---|
| Anthropic | Implementation started (no date) | ClaudeBot |
| Microsoft | Feb 1, 2027 | NoArchive / NoCache (retiring) |
| 2027 (aim) | Google-Extended | |
| OpenAI | ”Coming months” | GPTBot |
(Sources: Microsoft Bing Blog, Google Search Central Blog, PPC Land)
Anthropic has no blog post. Its commitment is Kelley’s message to the IETF AIPREF mailing list, which PPC Land quotes saying Anthropic “has already started implementing support for this draft” and that “it’ll take us some time to add support across our services.” Anthropic’s own crawler help page does not mention the token yet.
Microsoft gave the most detail. It will honor the token from February 1, 2027, and retire NoArchive and NoCache as AI controls by March 1, 2027.
Google says in John Mueller’s post that it aims to support the control “in 2027.” If you opt out with the shared token and do not mention Google-Extended, Google will treat that as a Google-Extended disallow.
OpenAI says it will implement the signal in the coming months, as a voluntary effort, and that it “will not replace” its existing robots.txt controls. PPC Land points out that OpenAI describes the scope as “foundational generative AI training,” which is narrower wording than the draft’s definition. Whether that matters in practice is an open question.
Meta, Amazon and Apple are not among the four, so the shared token does nothing for their crawlers yet.
Right now, a robots.txt that relies only on the new token is honored by nobody at full scale. It is a forward-looking line, not a replacement.
Training vs Search vs Grounding: What the Token Does Not Block
This is where most of the confusion will come from. Every lab now separates at least two jobs, and the training token touches only one.
OpenAI’s crawler overview splits them cleanly. GPTBot crawls content that “may be used in training our generative AI foundation models.” OAI-SearchBot surfaces sites in ChatGPT search, and opted-out sites “will not be shown in ChatGPT search answers.” ChatGPT-User fetches pages when a user asks, and OpenAI notes that “robots.txt rules may not apply” to those user-initiated visits.
Anthropic’s crawler help page has the same three-way split: ClaudeBot for training, Claude-User for user requests, and Claude-SearchBot for search quality. It also warns against blocking Anthropic’s IP ranges, because then the bots cannot read your robots.txt and the opt-out may not stick.
Microsoft goes further and adds a separate grounding preference, which controls whether your content can inform Copilot and Bing answers. It is page-level only:
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">
disallowextended keeps your pages in Microsoft’s own experiences, such as Copilot, while excluding third-party grounding services like Web IQ. The Bing post warns that restricting grounding “will significantly reduce brand visibility in AI experiences.”
Google’s AI Overviews and AI Mode are not affected by the training token at all. PPC Land notes they are governed by Googlebot rules and a Search Console setting.
If your goal is “don’t train on my work but still cite me”, the training token is the right tool. If your goal is “keep me out of AI answers”, it does nothing.
AI Crawler User Agents by Job
Before editing robots.txt, it helps to see every current bot on one page. These are the product tokens the vendors document today, grouped by what blocking them actually does.
| Provider | Training | Search / answers | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Google-Extended (control token) | Googlebot | n/a | |
| Microsoft | Shared token or page meta | Bingbot + grounding meta | n/a |
(Sources: OpenAI crawler overview, Anthropic crawler help, Google, Microsoft Bing Blog)
OpenAI also runs OAI-AdsBot, which only visits landing pages submitted as ads and, per its docs, is not used for training.
Spoofed user agents are common, so do not trust the string alone. OpenAI publishes IP ranges for each bot as JSON files, such as openai.com/gptbot.json and openai.com/searchbot.json, and Anthropic publishes its list at claude.com/crawling/bots.json. Checking a request’s source IP against those lists is the reliable way to know whether “ClaudeBot” in your logs is really Anthropic.
Microsoft is the outlier: it has no separate training crawler, so the shared token and page-level meta tags are the only training controls it will offer.
How to Block AI Training in robots.txt Today
Because support is staggered, the practical answer for the next year is belt and braces: the shared token plus the per-crawler rules that already work. Here is the version I would ship for a site that wants out of training but wants to stay in AI search:
# Shared AI training preference (IETF draft, honored progressively from 2026–2027)
User-Agent: __usage_ai-training
Disallow: /
# Per-provider training crawlers, still needed until each vendor ships support
User-Agent: GPTBot
Disallow: /
User-Agent: ClaudeBot
Disallow: /
User-Agent: Google-Extended
Disallow: /
# Search and user-fetch bots stay allowed so you still get cited
User-Agent: OAI-SearchBot
Allow: /
User-Agent: Claude-SearchBot
Allow: /
User-Agent: *
Allow: /
A few notes from reading the docs closely:
- Do this per subdomain. Anthropic’s help page says the robots.txt has to sit at the top level of each subdomain you want to opt out.
- Google-Extended has no crawler of its own. It is a control token read by Google’s existing crawlers, which is why disallowing it does not touch Search.
- Bing training opt-outs also work at page level with the
ai-training:disallowmeta tag or anX-Robots-Tagheader, and the page-level setting overrides robots.txt. - Nothing here is retroactive. The draft ties preferences to the robots.txt in effect when content was retrieved, and Anthropic’s wording is that blocking ClaudeBot excludes your site’s future material.
If you want to understand how to make a site more useful to agents rather than less, the WebMCP guide to AI-ready websites covers the opposite direction.
Ship both layers now, and remove the per-crawler training rules only after each vendor confirms it reads the shared token.
Why This Blog Still Allows Every AI Crawler
The Vibelog does the opposite of the template above. Our robots.txt explicitly allows GPTBot, ClaudeBot, Google-Extended and the rest, because the whole point of a site like this is to be the source an answer engine cites.
That is a trade, and I would not recommend it to everyone. A paid newsletter, a documentation site that sells support, or a publisher whose traffic is collapsing has good reasons to opt out of training. HubSpot and other publishers have been open about how much search traffic shifted to AI answers in 2026, and the training signal is one of the few levers site owners actually control.
Writing it up also showed me our own file has drifted. Anthropic’s help page lists three current bots, and our robots.txt still names older Anthropic user agents next to them. It costs nothing, but it is a reminder that these files rot quietly.
Website builders are starting to sell AI visibility as a feature for the same reason. Webflow’s answer engine tracking is covered in the Webflow AI review, and Wix’s AI agent approach is in the Wix Harmony breakdown.
If you also run Codex or ChatGPT for the code behind your site, December brings a separate change worth planning for, covered in the ChatGPT and Codex shared usage limits post.
Is the Shared Signal Enforceable?
No. The draft uses “MAY” language for whether consumers skip fetching content they cannot use, and none of the four announcements describe penalties or an audit mechanism.
That is normal for robots.txt. The original protocol was never enforceable either, and it worked for decades because the largest crawlers chose to follow it. The new token gets the same deal: it is exactly as strong as the companies honoring it.
What changed on October 30 is that the four largest model trainers in the West agreed in public on one syntax and one definition. That makes a preference easier to express and easier to point to if a provider ignores it.
Treat __usage_ai-training as the line you will eventually rely on, and keep the per-crawler rules as the line that works today.