[ai-control] Re: New draft: draft-kelley-aipref-training-preference-signal-00

Alissa Cooper <alissa@cooperw.in> Wed, 04 November 2026 15:20 UTC

Received: from fhigh-b8-smtp.messagingengine.com (fhigh-b8-smtp.messagingengine.com [202.12.124.159]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by mx.ietf.org (Postfix) with ESMTPS id 97F9646 for <ai-control@ietf.org>; Wed, 04 Nov 2026 15:20:49 +0000 (UTC)
Authentication-Results: mx.ietf.org; dkim=pass header.d=cooperw.in header.s=fm2 header.b=sAxH61TB; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="QU/+Q2Rv"; spf=pass (mx.ietf.org: domain of alissa@cooperw.in designates 202.12.124.159 as permitted sender) smtp.mailfrom=alissa@cooperw.in; dmarc=pass (policy=none) header.from=cooperw.in
Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.stl.internal (Postfix) with ESMTP id 46A1A7A0121 for <ai-control@ietf.org>; Wed, 4 Nov 2026 10:20:43 -0500 (EST)
Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Wed, 04 Nov 2026 10:20:43 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cooperw.in; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1793805643; x=1793892043; bh=HDcsl8P2oG R0/tqLTplz4tAH9nDRyuaWEAU3JFpLwc0=; b=sAxH61TBevaajju3r+WbTVUvOp 9iMJj0sZmvdwGjioKjuqbjTHVoUenI8YEg/u8z2R6e922jMb31kvmvyc7LBDM/dA E5kTi8huBJ/vz7lfr/YCts7d88NkQ8z7krU/5whnv2Dpdy8G7rM0h/tJXfiU2M4d feRgUZj2718KhQV08Nj/1VJo2AsJfv36T06MX1EtFFxKgZRBUyZt/qdtoWCqSx6H diKYxmoFnREPSQdNQqcvtK3bxfh8V6nLJNdlkeQtGg8f6HGIK450qUHBE6CJ+hIy KgH0qGbSueNocSl7zGarUa7oUTSF4QfoS+TzaNDYRkQVgMbZFqk5tUstz36A==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1793805643; x=1793892043; bh=HDcsl8P2oGR0/tqLTplz4tAH9nDRyuaWEAU 3JFpLwc0=; b=QU/+Q2RvdunkIRicb4XL2jrQKoqFAHcEkYbvCX4PMF/YNk3C5/F oOmg7pK2bz7X97So1Cu54Ml5nhdGzXK3yKMumrk6/Kfd0L4PLMgdcuWOwsSqDw0n 6b2iS0V2QVwI+9Wi5yHUpCCAfiq3TYU3DXak3PricDvY8vk13pa1SDxRHDnot+qW 1rLC7/yAyU/zb+Q7dGNAcuIZ8VLjK+ZdtODFW5ADcSHwGoma66vtcT7ew0usaGde YS+OxIf+dtfBlFUIk5P6ItrZBbIIBpZlYTknjlYQf32wueYs72Y6/pt3/GT7YN1f O4I+m6N4gyTJmm+r3bGsBZnR+uqBiLaWcSw==
X-DKIM2-Info: draft=ietf-dkim-dkim2-spec-08; repo=github.com/dkim2wg/interop; date=2026-10-28; sw=lmtpprox; action=sign d=cooperw.in a=rsa-sha256;
DKIM2-Signature: i=1; m=1; t=1793805643; d=cooperw.in; mf=PGFsaXNzYUBjb29wZXJ3LmluPg==; rt=PGFpLWNvbnRyb2xAaWV0Zi5vcmc+; s=fm2:rsa-sha256:NCxDXzHt46RO+vaFOm1Tca0NAYDPuivCsACwyW3cyligYw9 acoP2JFFmvJeVWrC2AjF6ZhKn1bfESMml0Oscl0fzkVUdV8uVYme38URP6ky4x2e /qgxMdlOM90kPHWMDJizdybzzZsbAB1mwwEbh9GoMr4hs6AD3cAzryRNy0Rh4Qe8 gxLKP60WMy6FdG7KmkNkB50dpKHNTqkp6AbUcNTRBXnyhiaTCYEzwTU/2HMftQuU TwOpxee/KxqurTGJZedWtmJ7rjE0PLCz1zm2G4eFsZRc7tx3f8eHJVCUdcFUApMG 3RL8vsuZdVxnVzvWk4HMswCSlouV6T1DBlLPUuQ==;
X-DKIM2-Info: draft=ietf-dkim-dkim2-spec-08; repo=github.com/dkim2wg/interop; date=2026-10-28; sw=lmtpprox; action=mi-m=1; hc=10; hn=content-type,date,feedback-id,from,in-reply-to,message-id, mime-version,references,subject,to;
Message-Instance: m=1; h=sha256:Js0ldkcXe48gydADL375D0uM5pyNsfL200xWnP8Vs7g=:aFs4l5gmwVUy74Z6X/PavfONEf3WkYiakj0cr1Nu3HM=;
X-ME-Sender: <xms:S03rah0XBA24Q9iyk3wDvMhkkJGtub5bNGznTqgSVFyuyy3mxJLFtQ> <xme:S03rat922x6JU4kzF5S-fiEkK-noqjaNIVooZa2-VQ32cpeEo0Zparf0y58bUaJwZ pDNgdvW3_IRhppcszSjxR0wWzmcLZw9WO9mKmjaM5SYJUAW926Etg>
X-ME-Received: <xmr:S03ras_DglDEQ2NnE5E4ukDW4-QvG0Gk59e0sKoEusN9UyEUVvGolv-oVpJT7sD2IbHQvQ>
X-ME-Proxy-Cause: dmFkZTF/liJmEUslqGjjbfQMhVfl+nuixa+wee7EvnCKmN3mMNLu+gM9vCodJQqxwUeH0u me2EX2AI2HcFIEinGLPq7qtFLKBjVzF/JKuNMbQRlCu6D0FEgBsXJ4usEHKQSFTG5Qgr0J tHXIdEO/Axld8ygHqqAofOUxLnJ8HpLJ140FOoadOiytPYe0bgjcQq8Y1mtmJrQndIZ1Nf /GnjiG49omW6TCVOzjP0p2/eS1zxBDhC91PiYgHuJhHjZgRltlavgfyH3ra8AeaVKNTp5m 8xkxlyas0Zc0gH0ce5qeLp/+3sxls2jkdn2uu7SiwUwtILGYtawGuR6Iw242Kiou98EcOG EzBhUWdIOgkDL9Y8GTowgUQBVQ7Kb//xUIdmzg19dJ0pxitO6QJisSXcXuomYN9B6kbLWl 4bYRQShuLFiDAR6UT8UVwP8twLt0A3HCGICOGsQ0MPZDiUsv07kE/FSlQEUyBJHwRUVxqK zDmIRkqbPP/W8DNU9kDd0D5SQ+g8dgHo87D/g2gbT7cOhtBxAwfNKmB09cAGDEtRq4XN9M K7FGM5cNYHN8F4bcii0/kFBCkrBFooVL7wZE1GsahPhcor/d2taSdBsVORhJF2kLzAq967 bbX2x29jZd5FXEbv22yRFCQx8FVvfsk0Xnxaz1kvH7qky8xwsIPLMTlWMH/A
X-ME-Proxy: <xmx:S03rarptDW-zKw-l0cu7Sd1SuNcS-WhSiOP4kVTyQ1hTupRdF4nKQQ> <xmx:S03rano_50i7v4PV6HqUMsxkft5w7BBErEE1dMP3Jumk6RkL3gPJ6g> <xmx:S03rahkka4YPUTL2n5vhjChHtDv1mmL6WjOsI7uNgOHNe5fvrAFUyw> <xmx:S03rahHKPRRTKmdlfXwK_fiehhPCrlxGSrd2_-Y3rDXeJMN4e9Aj8g> <xmx:S03rav3sXrDfWLP65vjsPlPXKHr9J8cUGsEzaRmjYfv5YA61wDqU3w_3>
Feedback-ID: i1214409c:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA for <ai-control@ietf.org>; Wed, 4 Nov 2026 10:20:42 -0500 (EST)
From: Alissa Cooper <alissa@cooperw.in>
Content-Type: multipart/alternative; boundary="Apple-Mail=_1A00CD75-33D7-434A-B5B8-277679685D2B"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.600.51.1.1\))
Date: Wed, 04 Nov 2026 10:20:31 -0500
References: <CAGfqOAY_t6UmzZw+nYzTfRVhyHuGb7RyEwmcwetwAF01f4ONbA@mail.gmail.com>
To: ai-control@ietf.org
In-Reply-To: <CAGfqOAY_t6UmzZw+nYzTfRVhyHuGb7RyEwmcwetwAF01f4ONbA@mail.gmail.com>
Message-Id: <80F59EBC-92B5-4F73-B78C-C8D57D24B66C@cooperw.in>
X-Mailer: Apple Mail (2.3864.600.51.1.1)
X-Spamd-Bar: ------------------------
Message-ID-Hash: GAN2KMWUSMLW64EFXMYACTRFECEBUA2G
X-Message-ID-Hash: GAN2KMWUSMLW64EFXMYACTRFECEBUA2G
X-MailFrom: alissa@cooperw.in
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.10
Precedence: list
Subject: [ai-control] Re: New draft: draft-kelley-aipref-training-preference-signal-00
List-Id: AI Control <ai-control.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ai-control/qhIHBiSQsFOKcKS8Yq56Dl2nhZU>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ai-control>
List-Help: <mailto:ai-control-request@ietf.org?subject=help>
List-Owner: <mailto:ai-control-owner@ietf.org>
List-Post: <mailto:ai-control@ietf.org>
List-Subscribe: <mailto:ai-control-join@ietf.org>
List-Unsubscribe: <mailto:ai-control-leave@ietf.org>

FYI, I wrote up a piece about this draft and the related company announcements, to try to contextualize this for a broader audience: https://www.techpolicy.press/what-the-new-ai-training-optout-really-means/

Given this approach of re-using the training definition from the vocabulary doc and overloading the User-Agent string, I do wonder why a policy token was not defined for the search category, whose definition is roughly as stable as the training definition. More broadly, the document is silent about extensibility beyond the mentions of proprietary policy tokens, insofar as it doesn’t say anything about how other standardized (or “cross-vendor”) policy tokens get defined. No registry or registration policy are mentioned. This could leave the impression that there is no intent for this mechanism to support other non-proprietary vocabulary categories.

Alissa

> On Oct 30, 2026, at 11:39 AM, Kevin Kelley <kelleyk-ietf@kelleyk.net> wrote:
> 
> Howdy everyone,
> 
> I've just submitted a new I-D, draft-kelley-aipref-training-preference-signal:
> 
>     https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
> 
> It describes a way to express AI-training preferences in robots.txt without adding any new grammar. A site operator writes an ordinary rule group whose User-Agent line names a "policy token":
> 
>     User-Agent: __usage_ai-training
>     Disallow: /
>     Allow: /press-kit/
> 
> This reads as "a preference against AI training for everything except /press-kit/, and a preference in favor of it there." Lookup uses the usual RFC 9309 path matching and longest-match precedence.
> 
> This is a preference about use, not a traffic-management signal; the two are intentionally separate, and ordinary rules for a crawler’s own product token still govern fetching (though the draft does ask crawlers not to fetch what they can’t use).
> 
> This differs from the attachment draft that we put aside in order to make progress on the vocabulary.  Like I've mentioned, I've spent a lot of time investigating how deployed parsers would interact with a wide variety of ways of expressing preferences in robots.txt files.
> 
> The mechanism in my draft uses only existing syntax, so parsers and crawlers that don't implement it but comply with RFC 9309 should gracefully ignore preference expressions and still receive other signals. It also avoids relying on the areas where deployed parsers behave differently (and let me tell you, there are plenty!).
> 
> And it builds on prior art: some crawler operators already publish tokens that control how content is used rather than which crawler can fetch it.  The draft introduces the term “policy token” as a vendor-neutral name for that pattern, and calls the existing vendor-specific ones “proprietary policy tokens”.
> 
> Wildcard groups never match the new, vendor-neutral policy token, so existing User-Agent: * rules aren’t read as expressing a preference, which was a deliberate design goal. As a result, a robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all.  Rules for crawler product tokens and proprietary policy tokens keep whatever meaning they have today.
> 
> I’m proposing this as input to the WG’s attachment work, and I'm planning to open a PR against the attachment draft to get thoughts from all of you.  I'll try to explain my thinking a little bit more there.  (Thank you to everyone who's already weighed in off-list!)
> 
> On the vocabulary front, the policy token takes its meaning directly from the "AI Training" definition in draft-ietf-aipref-vocab-08.
> 
> I'm also excited to share that Anthropic has already started implementing support for this draft.  This complements our existing crawler controls rather than replacing them.
> 
> It'll take us some time to add support across our services and work through the inevitable complications and edge cases that I'm sure we'll discover.  Also, as standards in this area evolve, how we interpret and apply these signals may change.  I'm looking forward to what we'll learn from having "running code" in the field and how that will inform the work we still have to do together.
> 
> I'm very pleased that, over the coming months, all of our hard work will start to become something that website publishers can use.  Putting a working signal in people's hands feels like tangible evidence that all of the effort and heart we've put in, and all of the long hours and long flights, are paying off.
> 
> We hope that others will join us and make this a single signal that is respected across the Internet.
> 
> I'm curious to hear everyone's reactions and happy to answer any questions.
> 
> On a personal note:  I'm incredibly grateful to everyone who has invested time and effort in the WG's work and engaged in good faith, from every point of view.  I hope this is just the beginning of what we can ship together, and I look forward to seeing all of you in San Francisco.
> 
> Cheers,
> 
> -K
> _______________________________________________
> ai-control mailing list -- ai-control@ietf.org
> To unsubscribe send an email to ai-control-leave@ietf.org