Tech Giants Back New Standard to Let Publishers Block AI Training Crawlers

Four major technology companies—Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC—have jointly announced their backing for a new website crawling standard designed to give publishers control over how AI developers access their content for model training.

The proposed standard builds on the existing robots.txt file format, which websites have long used to communicate with search engine crawlers and other automated visitors. Publishers can currently use robots.txt to block specific crawlers, limit crawling frequency, or restrict access to certain pages.

Today, managing AI-related scraping requires publishers to add separate entries to their robots.txt file for each AI provider and sometimes for multiple crawlers operated by the same provider. The new standard would consolidate this process, allowing a single robots.txt entry to block all scrapers collecting training data for generative AI models. The restriction would not apply to models that "perform exclusively non-generative tasks."

The four companies have already submitted a draft proposal to the Internet Engineering Task Force, the nonprofit organization responsible for maintaining the Robots Exclusion Protocol that governs robots.txt formatting standards. Their proposal aims to integrate the new crawling standard directly into this established protocol.

Implementation Timeline and Additional Controls

OpenAI and Microsoft plan to add support for the new standard within the coming months, while Google has indicated it will implement the feature sometime next year. Microsoft is also introducing a separate crawling control called AI Grounding, which addresses scenarios where an AI model accesses a webpage to answer a specific user question rather than to gather training material.

Through AI Grounding, website operators can prevent Microsoft's services from making such requests, and the feature will extend to third-party AI applications built on Microsoft's technology infrastructure. OpenAI is developing comparable controls for similar use cases.

Meta Platforms Inc. has expressed general support for the concept underlying the new standard but has not yet committed to implementing it.