A single switch instead of many
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will adopt a new standard for publishers who do not want their pages used to train AI, The Verge reported. Until now, a site had to signal its wishes to each company separately. The new control lets a publisher apply one preference across multiple AI companies.
The proposal comes from Kevin Kelley, a technical lead for standards at Anthropic, and moves through the Internet Engineering Task Force, the body behind many core internet standards.
How it works
The signal lives in robots.txt, the long-standing file that tells web crawlers what they may and may not fetch, whether for search indexing, archiving or AI training. The new piece is a shared policy token that covers AI training preferences specifically, so a publisher can express a single rule rather than listing every company’s crawler.
Per The Verge’s reporting, the point is to separate the training question from search indexing and archiving. Microsoft said it will offer granular controls that let publishers opt out of AI training or AI grounding, the practice of pointing a model to a verified source to reduce hallucinations. Microsoft also noted that sites will still appear in Bing search results regardless of the choice.
Who is on board, and who is not
Meta’s communications head, Andy Stone, welcomed the idea of a common signal that conveys preferences to multiple providers at once and said the company looks forward to working with others on implementation, but the statement does not say whether Meta will honor it.
The announcement arrives as publishers struggle with a steady flow of AI crawlers. Other efforts, such as services that let publishers charge crawlers for access, have tried to push back on unauthorized scraping; a shared opt-out is a lighter-touch alternative.
What to watch
A standard is only as good as its adoption. The proposal is still moving through the IETF process, and each company has its own rollout plans. Publishers will want to see which crawlers actually respect the token, how disputes are handled, and whether the signal reaches beyond the four named companies.
Practical takeaways
- Publishers: watch for the finalized token name and syntax before editing robots.txt.
- Keep search-indexing and AI-training preferences in mind separately; the new signal is about training.
- Check each AI company’s documentation for when it starts honoring the signal.
Bottom line
A shared robots.txt token would turn a patchwork of crawler-by-crawler rules into one opt-out. It is a meaningful step for publishers, though the real test is whether it is honored.