Google, Microsoft, OpenAI and Anthropic back a single robots.txt signal for websites to opt out of AI training

Google, Microsoft, OpenAI and Anthropic have said they will honour a shared signal that lets websites opt out of having their content used to train AI models by all participating companies at once, instead of setting rules for each company’s web crawler separately.

The signal is a “policy token”, __usage_ai-training, placed in a website’s robots.txt file, the text file sites have long used to tell crawlers what they may access. It was proposed in an Internet Draft by Kevin Kelley, a standards technical lead at Anthropic, submitted to the Internet Engineering Task Force’s AI Preferences working group. The IETF lists it as an individual draft, which does not represent official standardisation, Search Engine Journal reported.

The companies announced their plans on Friday 30 October. According to The Verge and Search Engine Journal, Anthropic has started implementing the signal, Google and Microsoft plan to support it in 2027, and OpenAI says it will do so “in the coming months”. Google said it would treat an opt-out through the shared token as if the site had blocked Google-Extended, its existing AI-training control. Microsoft said it would begin honouring new training and grounding preferences on 1 February 2027 and that sites would still appear in Bing search results whatever their settings.

Meta’s communications head Andy Stone said a common signal was “an important step”, The Verge reported, but did not say whether Meta would honour it.

Under the draft, existing robots.txt rules are not treated as a training preference, and the token does not control crawling itself.