🔍 Read the full analysis: Which Data Does The Shared AI Training Opt-Out Cover? on ThorstenMeyerAI.com
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, takes effect on different schedules, and does not cover AI answers based on live web content. Meta expressed support but did not commit to honoring it.
OpenAI, Google, Microsoft and Anthropic said they will honor a shared robots.txt signal asking websites not to use their content to train generative AI models. The voluntary measure gives publishers one common instruction to publish instead of separate training preferences for each participating company, but it does not block AI systems from crawling a site or using current web content to answer questions.
The proposed instruction is a robots.txt user-agent group for User-Agent: __usage_ai-training, followed by Disallow: /. It is set out in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published on October 30 by Kevin Kelley, an Anthropic standards lead. The companies’ announcements point to the same draft.
The companies have different implementation schedules. Anthropic said it has already started implementing support, while noting that support across its services will take time. OpenAI aims to implement the signal “in the coming months.” Google aims to support it in 2027; its crawler documentation says “early 2027.” Microsoft says it will honor the training preference from February 1, 2027, as it crawls websites.
Meta’s communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and described a common signal as an important step. That statement did not say whether Meta will honor the signal. The source material also reports no commitment from Apple, Amazon, xAI or Perplexity.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, the change could reduce the work of maintaining separate training instructions for each AI company. A single line can express a clear preference to the four companies that have announced support, rather than relying only on individual crawler names or company-specific controls. That matters to websites deciding whether their material may contribute to future generative AI models.
But the signal is a request, not a technical lock. Robots.txt communicates a site owner’s preference; it does not itself prevent a crawler from accessing pages. Its practical effect depends on each company implementing and honoring it. The announcements also do not establish that all AI developers will follow the instruction, or that existing training data will be changed.
The distinction between training and other uses is important. The shared preference concerns foundational generative AI training, not every use of a page by an AI service. Microsoft is introducing a separate page-level “AI Grounding” preference for content used to inform answers in Bing, Copilot and Microsoft 365. The two controls therefore address different activities, and a publisher seeking limits on both needs to account for both.
Products Worth Considering
Amazon Kindle Paperwhite (12th Generation, 2024), 16 GB, 7-inch Display, Jade
Apple MacBook Air 13-inch with M5 chip, 16GB memory, 1TB SSD, and 12MP Center Stage camera (Midnight)
Anker SOLIX C300 Portable Power Station, 288Wh
As an affiliate, we earn on qualifying purchases.
How Existing Controls Fit
Before the shared proposal, site owners commonly used company-specific crawler rules. These included GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has also used page-level NoArchive and NoCache tags. The newly announced signal does not immediately replace those existing controls across all participating services.
Google says that, where a site uses the new token but has no Google-Extended rule, it plans to interpret the shared preference as though the publisher had disallowed Google-Extended. Google says the new control will not affect Google Search. Microsoft says its preferences do not change eligibility for Bing search results. OpenAI says its existing crawler controls will continue to apply.
Microsoft’s schedule differs from the others: its shared training preference and separate AI Grounding preference take effect on February 1, 2027. Microsoft says NoArchive and NoCache will stop counting as training or grounding preferences after March 1, 2027. Publishers using those tags will need to review their instructions before that change. The proposal is an individual Internet-Draft, not an adopted internet standard; a working-group chair said on October 21 that the group had not declared consensus on anything.
““has already started implementing support for this draft””
— Anthropic, in its announcement of the draft
Adoption and Scope Still Vary
It is not yet clear how consistently the four companies will apply the preference across their services, or whether other AI companies will adopt it. The announcements also set out different start dates, so a publisher cannot assume that adding the line will have the same effect immediately across every service. Anthropic’s statement that implementation has begun does not specify when all its services will support the signal.
The draft does not establish an enforcement mechanism or guarantee that a company will follow a site’s instruction. It also does not say that content already used to train a model will be removed, and it does not prevent the use of live web pages to provide AI answers. Meta’s position remains a stated general support rather than a clear implementation commitment. The proposal’s eventual status in the IETF process is also undecided.
Publishers Must Track Rollout Dates
Website operators can add the shared user-agent group if they want to request exclusion from generative AI training by participating companies. They should keep existing company-specific instructions in place while checking each provider’s rollout, since the shared control is not active everywhere at once. Publishers using Microsoft’s NoArchive or NoCache tags should also review their rules ahead of the company’s March 1, 2027 change, and consider whether they need a separate AI Grounding preference.
The next developments are company implementation updates and any further commitments from Meta or other AI providers. The draft may also progress through the IETF’s working-group process, but no formal adoption has been announced in the source material. Until then, the shared signal is a voluntary preference with a defined training focus, not a universal restriction on access or AI use.
Key Questions
What does the shared AI training opt-out cover?
It signals that a website does not want participating companies to use its content for generative AI model training. It does not cover every use of the site by AI services.
Does the signal stop AI companies from crawling a website?
No. The robots.txt entry expresses a preference but does not technically block access. Its effect depends on a company choosing to implement and honor it.
Does it stop websites from appearing in search or AI answers?
The shared training signal is not a search exclusion and does not cover AI answers based on live web content. Microsoft announced a separate AI Grounding preference for certain services; Google and Microsoft say their relevant controls do not change search eligibility.
When will the companies honor the signal?
Anthropic says implementation has begun; OpenAI plans to implement it in the coming months; Google aims for early 2027; and Microsoft’s stated date is February 1, 2027. The schedules are not identical.
Has Meta agreed to honor the opt-out?
Not based on the statement described here. Meta expressed support for publishers’ ability to state preferences, but its communications chief did not say whether the company will honor this specific signal.
Source: ThorstenMeyerAI.com
