One robots.txt Rule to Opt Out of AI Training: What Google, OpenAI & Microsoft Announced

What was announced

On October 30, 2026, Google, OpenAI, and Microsoft each said they plan to honor one shared rule in robots.txt that tells AI companies not to use your content to train their models. Until now, opting out meant writing a separate rule for each company — GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on — and keeping that list current as new names appeared. The shared rule is meant to let you say it once.

It's based on work in the Internet Engineering Task Force's AI Preferences (AIPREF) working group. It isn't a finished standard, and no one honors it yet:

• Microsoft says its new preferences take effect on February 1, 2027.
• Google says it intends to support the signal in early 2027.
• OpenAI says it plans to implement it in the coming months and will share details later.

Some coverage also lists Anthropic among the backers. When we wrote this, we couldn't find an announcement from Anthropic itself, so check its crawler documentation before relying on it.

What the rule looks like

The shared rule uses ordinary robots.txt syntax with a special name on the User-agent line. That name, __usage_ai-training, isn't a real crawler — it's a "policy token" that stands for using your content to train AI. To opt out your whole site:

User-agent: __usage_ai-training
Disallow: /

To opt out one section, such as your blog, use a path instead:

User-agent: __usage_ai-training
Disallow: /blog/

Three things about this group are easy to get wrong:

• It states a preference about use, not access. It doesn't stop anything from fetching your pages. Microsoft's announcement is explicit that these settings govern how content is used, not whether it's crawled. To keep a crawler out, you still need a rule that names it.
• It doesn't change crawling for anyone else. Under the robots.txt standard (RFC 9309), a crawler follows the group that names it, or the User-agent: * group if none does. No crawler is called __usage_ai-training, so adding the group leaves Googlebot, Bingbot, and everyone else exactly where they were.
• Saying nothing isn't saying no. Microsoft says both its training and grounding preferences default to allow when you haven't published one.

How Google will read it — and the Gemini catch

Google already has an AI control: Google-Extended, a robots.txt token it introduced in 2023. According to Google's crawler documentation, it manages whether content Google crawls can be used to train future Gemini models and for grounding — Gemini using your pages as sources when it answers.

Google says that when a site uses the shared token to opt out of training and doesn't mention Google-Extended, it will treat that as if the site had disallowed Google-Extended. Because that token covers grounding as well as training, the shared opt-out on its own would also take you out of grounding in Gemini Apps.

Google's post says sites that specifically allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks through grounding. The trade-off is that the same token also covers training — so with Google, as currently documented, training and Gemini grounding stay tied together. You can allow both or disallow both, but these controls don't let you split them. Google's plans could change before launch, so check its documentation in early 2027.

None of this touches regular Google Search. Google says Google-Extended doesn't affect a site's inclusion in Search and isn't a ranking signal. AI Overviews and AI Mode are governed by a separate control in Search Console, covered below.

How Microsoft will read it

Microsoft's announcement goes further than the robots.txt rule:

• Training can be set site-wide with the shared __usage_ai-training group, or per page with a meta tag or HTTP header. A page-level setting overrides robots.txt.
• Grounding — using your content in Copilot and other AI answers — is a separate preference set per page, not in robots.txt. A new DisallowExtended value limits grounding to Microsoft's own experiences, excluding grounding services Microsoft provides to other companies.
• Visibility trade-off. Microsoft warns that restricting grounding will significantly reduce your content's visibility in AI-generated answers. It says these preferences don't change eligibility for regular Bing search results.
• Old tags are being retired. The new preferences replace noarchive and nocache as Microsoft's AI controls. Sites using them have until March 1, 2027 to switch; after that, those tags won't be read as training or grounding preferences.

Microsoft's post includes the exact page-level syntax. Use it rather than guessing, since a typo in a meta tag fails silently.

What about OpenAI and Anthropic today?

OpenAI says the shared signal won't replace its existing robots.txt controls. According to OpenAI's crawler documentation, disallowing GPTBot tells OpenAI not to use your content for training foundation models, and it's independent of OAI-SearchBot, which is what makes your pages eligible for ChatGPT search. You can block one and allow the other.

Anthropic's help center says blocking ClaudeBot in robots.txt signals that your site's future content should be excluded from its training data.

Until the shared rule is live — and for any AI company that never adopts it — company-specific rules are the only way to state your preference.

Training, AI answers, and search are separate switches

The most common mistake with AI controls is treating them as one switch. Opting out of training doesn't remove you from AI Overviews, and opting out of AI Overviews doesn't stop training. Decide each one separately:

For the AI Overviews side, see our guide to Search Console's AI performance report and opt-out.

Should you opt out of AI training?

It's a business decision more than an SEO one. None of these training controls is a ranking factor in Google or Bing search, so opting out won't hurt — or help — your regular search traffic.

Sites that sell content, publish original research, or produce creative work often opt out because they don't want it used to train models without permission or payment. Many small businesses leave training alone: their pages describe services, prices, and locations, and what they care about is whether AI answers cite them — which the search and grounding controls govern, not the training ones.

If Gemini sends you traffic, remember the Google catch above before adding the shared opt-out.

What to do now

1. Read your robots.txt. Open yoursite.com/robots.txt and look for groups naming GPTBot, ClaudeBot, Google-Extended, or other AI bots, plus any broad User-agent: * / Disallow: / rule that catches everything. Make sure every block is one you meant. Our robots.txt guide explains how the rules are matched.
2. If you want out of training now, keep using company rules. The shared rule doesn't work anywhere yet. For the three companies above:

   User-agent: GPTBot
   Disallow: /
   
   User-agent: ClaudeBot
   Disallow: /
   
   User-agent: Google-Extended
   Disallow: /

   Don't block OAI-SearchBot, Googlebot, or Bingbot unless you mean to leave those search results.
3. Adding the shared group early does nothing yet. Today no crawler reads it. Once Google supports it, though, it will also cover Gemini grounding unless you name Google-Extended, so decide on that first.
4. Using noarchive or nocache for Bing? Plan to switch to Microsoft's new preferences before March 1, 2027.
5. Check back in early 2027. These are announced plans, and the syntax and dates can still change while the IETF work continues.

Where Peak Visibility fits

Every Peak Visibility scan includes a robots.txt file. It's deliberately minimal: it allows all crawlers and points them to your new sitemap. It doesn't add any AI-training rules, because whether to opt out is your decision, not ours. If your current robots.txt already has AI rules — or you add the shared training group — copy our Sitemap: line into your existing file instead of replacing it. Uploading ours over yours would silently undo your opt-out.

FAQ

Is the shared AI training opt-out live yet?

No. As of October 31, 2026, it's an announced plan. Microsoft says its new preferences take effect February 1, 2027, Google says early 2027, and OpenAI says the coming months. Until then, use each company's own robots.txt rules.

Will opting out of AI training hurt my Google or Bing rankings?

No. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal, and Microsoft says its training and grounding preferences don't change eligibility for regular Bing search results.

Does __usage_ai-training block AI crawlers?

No. It states a preference about how your content is used, not whether it can be fetched. To keep a crawler off your site, disallow that crawler by name.

Will it remove my site from AI Overviews or ChatGPT search?

Not by itself. Google's AI Overviews and AI Mode have their own control in Search Console, and ChatGPT search uses OAI-SearchBot, which is separate from OpenAI's training controls. With Google, though, the shared opt-out also covers grounding in Gemini Apps unless you explicitly allow Google-Extended.

Do I still need GPTBot and Google-Extended rules?

Yes, for now. The shared rule isn't honored anywhere yet, and OpenAI and Google both say their existing company-specific controls will keep working alongside it.