OpenAI, Google, Microsoft, and Anthropic Unite on New Standard to Opt Out of AI Scraping

Image: The biggest names in AI are backing a new way to opt out of scraping · Source

Major AI companies are adopting a shared standard allowing publishers to opt out of AI data scraping via a unified 'policy token' in robots.txt files, aiming to simplify and respect site owners' AI training preferences.

Leading artificial intelligence firms OpenAI, Google, Microsoft, and Anthropic have announced their support for a new standard designed to give publishers a unified way to opt out of having their content scraped for AI training purposes. Published reporting by The Verge reveals that this standard introduces a "policy token" signal embedded within a website’s existing robots.txt file—a well-established method used to communicate crawling permissions to web robots.

Traditionally, website owners needed to specify their AI content usage preferences separately with each company deploying AI crawlers, complicating enforcement and compliance. The new approach, proposed by Kevin Kelley, a technical lead for standards at Anthropic and developed within the Internet Engineering Task Force (IETF), aims to streamline this process by providing a shared signal. When AI crawlers from participating companies detect this token, they are programmed to respect the publisher's choice and refrain from scraping content.

This initiative comes amid growing concerns from publishers about unauthorized scraping by numerous AI models ingesting vast amounts of web content. Some existing measures included the Really Simple Licensing (RSL) standard, which enabled charging web crawlers for access, and protective efforts such as Cloudflare’s default AI crawler blocking.

Microsoft has committed to offering granular control options that enable publishers to opt out of allowing their content to be used either for AI training or as verified information sources to prevent AI hallucinations, a process known as grounding. The company clarified that even opted-out sites will remain discoverable via Bing search results. Google has implemented similar features that separate usage of site content for AI training and AI-enhanced search features without affecting general search visibility.

Anthropic has already incorporated the policy token system, while Google and Microsoft plan adoption in 2027. OpenAI expects to implement it within the upcoming months. Meta’s communications head Andy Stone expressed support for the common signal concept but has not detailed if Meta will enforce it immediately.

This collective effort marks a significant step toward standardizing how websites can communicate their preferences regarding AI training data usage, reflecting the industry’s movement toward more transparent and publisher-respecting AI development practices.