[ai-control] New draft: draft-kelley-aipref-training-preference-signal-00

[Kevin Kelley <[e-mail]>]

Howdy everyone,

I've just submitted a new I-D,
draft-kelley-aipref-training-preference-signal:


https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/

It describes a way to express AI-training preferences in robots.txt without
adding any new grammar. A site operator writes an ordinary rule group whose
User-Agent line names a "policy token":

    User-Agent: __usage_ai-training
    Disallow: /
    Allow: /press-kit/

This reads as "a preference against AI training for everything except
/press-kit/, and a preference in favor of it there." Lookup uses the usual
RFC 9309 path matching and longest-match precedence.

This is a preference about use, not a traffic-management signal; the two
are intentionally separate, and ordinary rules for a crawler’s own product
token still govern fetching (though the draft does ask crawlers not to
fetch what they can’t use).

This differs from the attachment draft that we put aside in order to make
progress on the vocabulary.  Like I've mentioned, I've spent a lot of time
investigating how deployed parsers would interact with a wide variety of
ways of expressing preferences in robots.txt files.

The mechanism in my draft uses only existing syntax, so parsers and
crawlers that don't implement it but comply with RFC 9309 should gracefully
ignore preference expressions and still receive other signals. It also
avoids relying on the areas where deployed parsers behave differently (and
let me tell you, there are plenty!).

And it builds on prior art: some crawler operators already publish tokens
that control how content is used rather than which crawler can fetch it.
The draft introduces the term “policy token” as a vendor-neutral name for
that pattern, and calls the existing vendor-specific ones “proprietary
policy tokens”.

Wildcard groups never match the new, vendor-neutral policy token, so
existing User-Agent: * rules aren’t read as expressing a preference, which
was a deliberate design goal. As a result, a robots.txt file that doesn’t
use this mechanism, as is true of every one published today, expresses no
preference at all.  Rules for crawler product tokens and proprietary policy
tokens keep whatever meaning they have today.

I’m proposing this as input to the WG’s attachment work, and I'm planning
to open a PR against the attachment draft to get thoughts from all of you.
I'll try to explain my thinking a little bit more there.  (Thank you to
everyone who's already weighed in off-list!)

On the vocabulary front, the policy token takes its meaning directly from
the "AI Training" definition in draft-ietf-aipref-vocab-08.

I'm also excited to share that Anthropic has already started implementing
support for this draft.  This complements our existing crawler controls
rather than replacing them.

It'll take us some time to add support across our services and work through
the inevitable complications and edge cases that I'm sure we'll discover.
Also, as standards in this area evolve, how we interpret and apply these
signals may change.  I'm looking forward to what we'll learn from having
"running code" in the field and how that will inform the work we still have
to do together.

I'm very pleased that, over the coming months, all of our hard work will
start to become something that website publishers can use.  Putting a
working signal in people's hands feels like tangible evidence that all of
the effort and heart we've put in, and all of the long hours and long
flights, are paying off.

We hope that others will join us and make this a single signal that is
respected across the Internet.

I'm curious to hear everyone's reactions and happy to answer any questions.

On a personal note:  I'm incredibly grateful to everyone who has invested
time and effort in the WG's work and engaged in good faith, from every
point of view.  I hope this is just the beginning of what we can ship
together, and I look forward to seeing all of you in San Francisco.

Cheers,

-K