Quick Summary
- A proposed policy token could let publishers express a preference against having their website content used to train AI models.
- The idea is to place a shared signal in website settings, including robots.txt, so multiple AI companies can interpret the same instruction.
- The proposal is still a standards effort, and its practical effect will depend on adoption, technical definitions and how companies respond to the signal.
A common signal for content-use preferences
Publishers have long used robots.txt to communicate crawling preferences to search engines and other automated services. A new proposal known as a policy token aims to extend that idea by giving websites a shared way to express whether their content may be used for AI training. The proposal was reported on November 1 and involves work on a standard that could be recognised by several major AI companies rather than requiring publishers to configure separate rules for every crawler.
The reported proposal is associated with Kevin Kelley, a technical lead for standards at Anthropic, and is being discussed through the Internet Engineering Task Force process. The concept is to allow a publisher to add a policy signal to its website configuration. AI companies that agree to support the signal could then read that instruction and apply it when collecting or processing content for training.
The appeal is simplicity. A publisher that wants to restrict training use may otherwise need to identify each crawler, understand the company behind it and keep separate rules updated as services change. A common token could reduce that administrative work and create a more consistent signal across the web. It could also help smaller publishers that lack the resources to negotiate individual arrangements with large technology companies.
A technical preference is not the same as enforcement
The proposal should not be confused with a technical mechanism that can physically prevent a company from accessing a webpage. Robots.txt is generally a request to compliant crawlers, not an access-control system. A crawler can ignore the file unless it is blocked through other technical measures or subject to a legal agreement. A policy token would face the same basic limitation: it can communicate a preference, but its effectiveness depends on whether the recipient honours it.
That makes adoption central to the proposal. If major AI developers support the proposal and publish clear policies for respecting it, publishers may gain a more practical way to express their position. If support is inconsistent, websites could still face a fragmented landscape in which some crawlers follow the signal and others do not. Standards bodies can define a common format, but they cannot by themselves guarantee that every operator will comply.
Publishers will also need clarity about what the token covers. AI training, search indexing, retrieval-augmented generation and real-time page access are different activities. A publisher may want a page to appear in search results while objecting to its use in model training. A useful specification should make those distinctions explicit rather than treating all automated access as one category.
The proposal comes amid continuing disputes over how AI systems collect web content and how publishers are compensated. A shared signal could become one part of a broader framework involving licensing, crawler identification, access controls and commercial agreements. It is unlikely to settle questions about copyright or fair use on its own.
Website owners should follow the standards process and avoid assuming that adding an experimental token will automatically change how every AI company handles their content. They should keep existing crawler policies in place and review the terms of services that access their sites. The value of a common policy signal will become clearer once the specification is finalised and major services explain how they intend to implement it.
A robots.txt signal is not a technical lock
The proposed policy token aims to reduce the need for publishers to maintain separate training instructions for every provider. Google says it plans to interpret a shared opt-out as a Google-Extended disallow when a site has not specified Google-Extended separately. Anthropic says it has adopted the signal, while OpenAI expects implementation in the coming months and Google and Microsoft have described plans for 2027. Meta has expressed interest but has not made the same implementation commitment.
The proposal is still an Internet-Draft, not a completed IETF standard, and robots.txt remains a voluntary instruction. It does not prevent a non-compliant crawler from requesting a public page, erase data already collected or replace authentication and contractual controls. Publishers should keep their existing provider-specific rules until each company confirms that its crawler honours the new token. The important development is the move toward a shared vocabulary, not an immediate technical guarantee that every site can block every unwanted use.