[ai-control] No preference, default "allowed", preference in favor: clarifying the readings

JF Farny <jff@aday.fr> Thu, 05 November 2026 18:05 UTC

Received: from mx1.aday.fr (mx1.aday.fr [185.65.96.25]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx.ietf.org (Postfix) with ESMTPS id 7C34A46 for <ai-control@ietf.org>; Thu, 05 Nov 2026 18:05:20 +0000 (UTC)
Authentication-Results: mx.ietf.org; dkim=pass header.d=aday.fr header.s=mail header.b=EyXzhser; spf=pass (mx.ietf.org: domain of jff@aday.fr designates 185.65.96.25 as permitted sender) smtp.mailfrom=jff@aday.fr; dmarc=pass (policy=reject) header.from=aday.fr
Received: from lxc-sd-dovecot.sd.edd (lxc-sd-dovecot.sd.edd [192.168.193.56]) by mx1.aday.fr (Postfix) with ESMTP id 038F444281 for <ai-control@ietf.org>; Thu, 5 Nov 2026 19:04:27 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=aday.fr; s=mail; t=1793901867; bh=/eYECXSzdtAn4i6LpoJSGYugJprnYuOFNNDLBKjY7as=; h=From:Subject:Date:References:To:In-Reply-To:From; b=EyXzhserhAPqAnDLpLH2xEiqL2W6Ro/FMyp/pytORk2qvT0Gl43sB1cObqiDeTceb 8XMDRvjauyVOOAgntJh2vbBHkcvkTJ/dSXj+Qz3Vhnqnbi1AA8TPyYbSol77EZkFwF tAHqSvQh50xRCcdT41vcLCQpJAVN80o4jEnvd+cw=
Received: from smtpclient.apple (unknown [10.75.225.30]) by lxc-sd-dovecot.sd.edd (Postfix) with ESMTPSA id 48544326099B for <ai-control@ietf.org>; Thu, 5 Nov 2026 19:05:19 +0100 (CET)
From: JF Farny <jff@aday.fr>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3901.100.1.1.12\))
Date: Thu, 05 Nov 2026 19:05:09 +0100
References: <CAGfqOAY_t6UmzZw+nYzTfRVhyHuGb7RyEwmcwetwAF01f4ONbA@mail.gmail.com>
To: ai-control@ietf.org
In-Reply-To: <CAGfqOAY_t6UmzZw+nYzTfRVhyHuGb7RyEwmcwetwAF01f4ONbA@mail.gmail.com>
Message-Id: <790A44EA-CB58-4901-B7BD-D964AA63249D@aday.fr>
X-Mailer: Apple Mail (2.3901.100.1.1.12)
X-Spamd-Bar: -----------------------
Message-ID-Hash: 2SJYU5PMU72H6XGDDT6VTMM5OEJZQ275
X-Message-ID-Hash: 2SJYU5PMU72H6XGDDT6VTMM5OEJZQ275
X-MailFrom: jff@aday.fr
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.10
Precedence: list
Subject: [ai-control] No preference, default "allowed", preference in favor: clarifying the readings
List-Id: AI Control <ai-control.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ai-control/_k4uoIWY_e2uY7Nv_Tc7P9prulw>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ai-control>
List-Help: <mailto:ai-control-request@ietf.org?subject=help>
List-Owner: <mailto:ai-control-owner@ietf.org>
List-Post: <mailto:ai-control@ietf.org>
List-Subscribe: <mailto:ai-control-join@ietf.org>
List-Unsubscribe: <mailto:ai-control-leave@ietf.org>

Kevin, all,

RFC 9309 §1 defines its rules as those "crawlers are requested to honor when accessing
URIs": reading robots.txt is a precondition for access, and its scope is that of the
access. The draft places in the User-Agent line, where §2.2.1 expects a crawler's name, a
token that identifies a use made at another time (§3, "in effect when the asset was
retrieved"). Three consequences follow.

1. §4.2 reuses the lookup mechanism of §§2.2.1 and 2.2.2 and interprets "allowed" as a
 preference in favor. Yet §2.2.2 makes a path matched by no rule "allowed": this is
 simply the default crawling behavior. Adding the token to an existing group that
 disallows a few paths, as goal 3 of §1.1 recommends, therefore expresses a preference
 in favor of AI training for the rest of the site.

2. "No preference has been expressed" (§4.2) also covers a file reduced to
 "User-Agent: *" / "Disallow: /", and a host whose robots.txt yields no parseable
 group. The only interpretation consistent with all these cases is unknown.

3. The declaring party is the service operator (§2), defined by what it operates.
 [VOCAB] §3.2 leaves legal and contractual mechanisms aside; RFC 9309 §1 states that
 its rules "are not a form of access authorization"; and the vocabulary editor wrote
 here on 18 June 2025 that "y" is not "a definitive license for that use". A consumer
 interpreting "allowed" as permission to use the content is therefore relying on a
 permission established elsewhere.

§4.4 says the token "is not a directive for crawling behavior", and §2 says it "need not
appear in the User-Agent header field of any outbound request". If it does not affect
crawling behavior, which act does it address, and for which reader? Why should this
information be carried in robots.txt rather than in an "other record" (§2.2.4) or the
Content-Usage rule of [ATTACH]?

Should the document say that a path matched by no rule yields "unknown", distinct from
"in favor", the latter being a statement about use whose permission is established by
other means? Suggested text for §4.2: "If no rule in the policy-token group matches the
path, the answer for that path is unknown ([VOCAB] §5). A preference in favor is a
statement by the service operator about use; permission to use is established by other
means."

And since this gives the §2.2.2 default a new meaning and extends the interpretation
beyond the access for which robots.txt was the premise, should the draft explicitly
declare that it updates RFC 9309?

JF Farny
Aday