[ai-control] No preference, default "allowed", preference in favor: clarifying the readings
JF Farny <jff@aday.fr> Thu, 05 November 2026 18:05 UTC
Received: from mx1.aday.fr (mx1.aday.fr [185.65.96.25]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx.ietf.org (Postfix) with ESMTPS id 7C34A46 for <ai-control@ietf.org>; Thu, 05 Nov 2026 18:05:20 +0000 (UTC)
Authentication-Results: mx.ietf.org; dkim=pass header.d=aday.fr header.s=mail header.b=EyXzhser; spf=pass (mx.ietf.org: domain of jff@aday.fr designates 185.65.96.25 as permitted sender) smtp.mailfrom=jff@aday.fr; dmarc=pass (policy=reject) header.from=aday.fr
Received: from lxc-sd-dovecot.sd.edd (lxc-sd-dovecot.sd.edd [192.168.193.56]) by mx1.aday.fr (Postfix) with ESMTP id 038F444281 for <ai-control@ietf.org>; Thu, 5 Nov 2026 19:04:27 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=aday.fr; s=mail; t=1793901867; bh=/eYECXSzdtAn4i6LpoJSGYugJprnYuOFNNDLBKjY7as=; h=From:Subject:Date:References:To:In-Reply-To:From; b=EyXzhserhAPqAnDLpLH2xEiqL2W6Ro/FMyp/pytORk2qvT0Gl43sB1cObqiDeTceb 8XMDRvjauyVOOAgntJh2vbBHkcvkTJ/dSXj+Qz3Vhnqnbi1AA8TPyYbSol77EZkFwF tAHqSvQh50xRCcdT41vcLCQpJAVN80o4jEnvd+cw=
Received: from smtpclient.apple (unknown [10.75.225.30]) by lxc-sd-dovecot.sd.edd (Postfix) with ESMTPSA id 48544326099B for <ai-control@ietf.org>; Thu, 5 Nov 2026 19:05:19 +0100 (CET)
From: JF Farny <jff@aday.fr>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3901.100.1.1.12\))
Date: Thu, 05 Nov 2026 19:05:09 +0100
References: <CAGfqOAY_t6UmzZw+nYzTfRVhyHuGb7RyEwmcwetwAF01f4ONbA@mail.gmail.com>
To: ai-control@ietf.org
In-Reply-To: <CAGfqOAY_t6UmzZw+nYzTfRVhyHuGb7RyEwmcwetwAF01f4ONbA@mail.gmail.com>
Message-Id: <790A44EA-CB58-4901-B7BD-D964AA63249D@aday.fr>
X-Mailer: Apple Mail (2.3901.100.1.1.12)
X-Spamd-Bar: -----------------------
Message-ID-Hash: 2SJYU5PMU72H6XGDDT6VTMM5OEJZQ275
X-Message-ID-Hash: 2SJYU5PMU72H6XGDDT6VTMM5OEJZQ275
X-MailFrom: jff@aday.fr
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.10
Precedence: list
Subject: [ai-control] No preference, default "allowed", preference in favor: clarifying the readings
List-Id: AI Control <ai-control.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ai-control/_k4uoIWY_e2uY7Nv_Tc7P9prulw>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ai-control>
List-Help: <mailto:ai-control-request@ietf.org?subject=help>
List-Owner: <mailto:ai-control-owner@ietf.org>
List-Post: <mailto:ai-control@ietf.org>
List-Subscribe: <mailto:ai-control-join@ietf.org>
List-Unsubscribe: <mailto:ai-control-leave@ietf.org>
Kevin, all, RFC 9309 §1 defines its rules as those "crawlers are requested to honor when accessing URIs": reading robots.txt is a precondition for access, and its scope is that of the access. The draft places in the User-Agent line, where §2.2.1 expects a crawler's name, a token that identifies a use made at another time (§3, "in effect when the asset was retrieved"). Three consequences follow. 1. §4.2 reuses the lookup mechanism of §§2.2.1 and 2.2.2 and interprets "allowed" as a preference in favor. Yet §2.2.2 makes a path matched by no rule "allowed": this is simply the default crawling behavior. Adding the token to an existing group that disallows a few paths, as goal 3 of §1.1 recommends, therefore expresses a preference in favor of AI training for the rest of the site. 2. "No preference has been expressed" (§4.2) also covers a file reduced to "User-Agent: *" / "Disallow: /", and a host whose robots.txt yields no parseable group. The only interpretation consistent with all these cases is unknown. 3. The declaring party is the service operator (§2), defined by what it operates. [VOCAB] §3.2 leaves legal and contractual mechanisms aside; RFC 9309 §1 states that its rules "are not a form of access authorization"; and the vocabulary editor wrote here on 18 June 2025 that "y" is not "a definitive license for that use". A consumer interpreting "allowed" as permission to use the content is therefore relying on a permission established elsewhere. §4.4 says the token "is not a directive for crawling behavior", and §2 says it "need not appear in the User-Agent header field of any outbound request". If it does not affect crawling behavior, which act does it address, and for which reader? Why should this information be carried in robots.txt rather than in an "other record" (§2.2.4) or the Content-Usage rule of [ATTACH]? Should the document say that a path matched by no rule yields "unknown", distinct from "in favor", the latter being a statement about use whose permission is established by other means? Suggested text for §4.2: "If no rule in the policy-token group matches the path, the answer for that path is unknown ([VOCAB] §5). A preference in favor is a statement by the service operator about use; permission to use is established by other means." And since this gives the §2.2.2 default a new meaning and extends the interpretation beyond the access for which robots.txt was the premise, should the draft explicitly declare that it updates RFC 9309? JF Farny Aday
- [ai-control] New draft: draft-kelley-aipref-train… Kevin Kelley
- [ai-control] Re: New draft: draft-kelley-aipref-t… Laurent Le Meur
- [ai-control] Re: New draft: draft-kelley-aipref-t… Nate Hake
- [ai-control] Re: New draft: draft-kelley-aipref-t… Alissa Cooper
- [ai-control] No preference, default "allowed", pr… JF Farny
- [ai-control] Re: New draft: draft-kelley-aipref-t… Mark Nottingham
- [ai-control] Which party does a robots.txt readin… JF Farny
- [ai-control] Re: New draft: draft-kelley-aipref-t… Alex Gibson
- [ai-control] Re: New draft: draft-kelley-aipref-t… Kevin Kelley
- [ai-control] Re: New draft: draft-kelley-aipref-t… Kevin Kelley
- [ai-control] Re: New draft: draft-kelley-aipref-t… Nate Hake
- [ai-control] Re: New draft: draft-kelley-aipref-t… Srecko Jovancevic
- [ai-control] Re: New draft: draft-kelley-aipref-t… angela.mills-wade
- [ai-control] Re: New draft: draft-kelley-aipref-t… Martin Thomson
- [ai-control] Re: New draft: draft-kelley-aipref-t… Alex Gibson
- [ai-control] Re: New draft: draft-kelley-aipref-t… Kevin Kelley
- [ai-control] Re: New draft: draft-kelley-aipref-t… Kevin Kelley