AI firms back one robots.txt opt-out for AI training

Website owners may soon be able to opt out of AI training at several companies with a single line in robots.txt. On October 30, 2026, OpenAI and Google each said they will honour a shared “policy token” for AI-training preferences, based on work in the IETF’s AI Preferences working group. The Verge reports that Microsoft and Anthropic are backing it too.

What changes for site owners

Today, a site that wants to keep its pages out of AI training has to name each company’s crawler separately in robots.txt. The shared token is meant to replace that list with one broad signal that every participating provider reads. Google says that when a site uses the token to opt out of training, and does not mention Google-Extended separately, it plans to treat that as if the site had blocked Google-Extended, its own AI-training control since 2023.

Both companies say their existing controls stay in place. OpenAI says the new signal “will not replace” its current crawler rules, and Google says company-specific instructions will keep working alongside the shared one, so a site can opt out broadly and still make exceptions for individual providers.

When it takes effect

The timing differs by company. OpenAI says it aims to implement the signal “in the coming months”. Google says it aims to support it, along with other providers, in 2027. According to The Verge, Anthropic has already adopted the signal, Microsoft plans to support it next year, and the proposal came from an Anthropic standards lead within the IETF process. The Verge also reports that Meta voiced support without saying whether it will honour the token.

Google notes that the proposal is a draft and that the IETF group is still working on the rest of its vocabulary. It also says the approach is consistent with the EU’s code of practice for general-purpose AI providers, which treats robots.txt as a main way for publishers to state machine-readable preferences.

Why it matters (analysis)

For small publishers, one shared switch is simpler than tracking a growing list of crawler names. It is still voluntary: the token only binds providers that choose to honour it, and none of the announcements describe enforcement. Search visibility is a separate question: The Verge reports that Microsoft says sites will still appear in Bing results whatever their training preference. More on policy and regulation.

This report is not legal, financial, or health advice. See the disclaimer.