MIME-Version: 1.0
References: 
 <CADTQi=dSGTGJtczLTZ1ua77C8h4pvGhrVkz3MZV7SO280MJrKQ@mail.gmail.com>
 <19B52E65-D016-4606-A329-70D6FF10E955@vaibhavbajpai.com>
 <CAKaUSfUFR82mCjy7qWzgibnuatQ7YdJ_yStur3b2zMR0KbMWkQ@mail.gmail.com>
 <6d65e967-b8a0-46be-9d02-00b66adb306a@Spark>
In-Reply-To: <6d65e967-b8a0-46be-9d02-00b66adb306a@Spark>
From: Shivdeep Singh <shivdeepsachdeva@gmail.com>
Date: Fri, 21 Aug 2026 12:47:15 +0530
Message-ID: 
 <CAKaUSfUH=yZiHgP0=YjCnEpFKyVC3meZV9MYH+JfjYqQcF1sfA@mail.gmail.com>
To: nick@lifelightlabs.com
Content-Type: multipart/alternative; boundary="000000000000de4d8f0659896d8c"
Message-ID-Hash: EJTALP5F7CHHO6JB4E3WHN6UVZP4T34M
CC: Vaibhav Bajpai <contact@vaibhavbajpai.com>, web-bot-auth@ietf.org,
 "Mirja Kuehlewind (IETF)" <ietf@kuehlewind.net>,
 Sauron <sauron=40google.com@dmarc.ietf.org>
Precedence: list
Subject: =?utf-8?q?=5BWeb-bot-auth=5D_Re=3A_Follow-up_from_IETF_126=3A_Comments_on_dr?=
	=?utf-8?q?aft-illyes-webbotauth-cbcp?=
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/web-bot-auth/pXKM0dkbAWsMqcAaXObxB6WUID4>

--000000000000de4d8f0659896d8c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Nick, thank you, and the tag is the sharper version of this. I had the
scope sentence and the consumption path in 5.5.3. That a conformant proof
must carry tag=3Dhttp-message-signatures-directory is what makes it concret=
e,
since that value has no story on a jwks_uri response.

Agreed that the ambiguity is the problem rather than either reading.

This is not hypothetical at our end. We serve a hosted directory carrying
four operators' keys under one identifier. Per-tenant authorities would
resolve that, but they require an origin per operator, so the route for
operators without one is per-tenant jwks_uri, which lands squarely in the
gap.

So if Thibault lands on directory-only, I am happy to draft the sentence
for wherever a jwks_uri or cimd operator would actually find it: that key
material published under those types cannot carry an Appendix B proof, and
that a verifier receiving it redistributed falls back to the thumbprint
identifier in 4.3, which does not survive rotation. Happy to open it as a
PR, or leave it with the editors.

Shivdeep

On Fri, Aug 21, 2026 at 8:49=E2=80=AFAM Nick Mathews <nick@lifelightlabs.co=
m> wrote:

> Shivdeep, Vaibhav, Gary,
>
> Speaking to the Appendix B question as the contributor of the test vector=
s
> that validate it (E.2.3), not for the spec text itself.
>
> I think both of your readings have support in the text as written, which
> is the problem. The framing sentence describes the concern: what a verifi=
er
> checks when it wants the domain a key is published under, rather than the
> URL on its own. Nothing in the B.1 construction consumes a directory-form=
at
> field; it is a per-key signature over @authority and content-digest with
> created and expires, and that applies mechanically to any HTTPS response
> that serves key material, including a hosted jwks_uri. But the normative
> hooks are directory-shaped: the required tag value is literally
> http-message-signatures-directory, and the consumption path runs through
> 5.5.3. So an implementer today cannot claim conformant proofs on a jwks_u=
ri
> response even though the mechanism would work there.
>
> Our verifier implements B.1 on directory responses only, because that is
> what the text licenses and what the vectors cover. If the intent is
> any-type, the appendix should say so and give the tag a story for
> non-directory responses. If the intent is directory-only, then the
> consequence you describe, jwks_uri and cimd operators falling back to
> thumbprint identity with no rotation per 4.3, deserves a sentence where
> those operators will find it. Either sentence is cheap and the ambiguity =
is
> not. I would like to see Thibault confirm the intended scope before the B=
CP
> points operators at the mechanism.
>
> On the point 3 convergence, one merchant-side datapoint: distinguishable
> per-purpose identities are not just registry hygiene. Our policy layer ke=
ys
> rules on the resolved identifier, so a crawler that separates search from
> training from assistant retrieval gets separate policy rows a merchant ca=
n
> treat differently. A single generic identity collapses that to one row, a=
nd
> the measurement in the paper matches what we see from merchants: they wan=
t
> to say yes to one purpose and no to another, and the identifier is the on=
ly
> handle they have.
>
> Nick
> On Aug 20, 2026 at 6:26=E2=80=AFAM -0400, Shivdeep Singh <
> shivdeepsachdeva@gmail.com>, wrote:
>
> Vaibhav, Gary, Nick,
>
> Point 3 and Nick's argument converge on the same mechanism, and I have a
> scope question about it.
>
> Under draft-meunier-webbotauth-httpsig-protocol-02 an identity is the
> resolved Signature-Agent URL, so distinguishable per-purpose crawlers mea=
n
> distinct URLs. An operator without a stable origin cannot use the directo=
ry
> type, since 5.5 requires that member value to be an origin, which leaves
> jwks_uri or cimd.
>
> Appendix B opens with "It applies to the directory type in Section 5.5." =
I
> read that two ways. Either the possession proof in B.1 is available only =
to
> the directory type, in which case jwks_uri and cimd material can never
> satisfy 5.5.3 and falls back to thumbprint identity, which 4.3 says has n=
o
> rotation. Or the scope line is about domain binding specifically, the
> concern of that appendix, and B.1's mechanism is available to any type, i=
n
> which case saying so would help implementers.
>
> I do not think this changes either recommendation. It affects who can act
> on them, so it seemed worth asking before the BCP points operators at the
> mechanism.
>
> Shivdeep Singh
>
> On Thu, Aug 20, 2026 at 2:53=E2=80=AFPM Vaibhav Bajpai <contact@vaibhavba=
jpai.com>
> wrote:
>
>> Hi everyone,
>>
>> This paper just got published, and we believe would be
>> valuable input to the BCP draft:
>>
>> >From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in
>> the Age of AI
>> https://dl.acm.org/doi/pdf/10.1145/3831956.3831960
>>
>> The measurements suggest that being identifiable and respecting
>> robots.txt is
>> necessary, but no longer sufficient. The harder emerging problem is
>> ensuring
>> that authenticated crawlers interpret a fragmented set of publisher
>> signals
>> consistently, predictably, and according to the intended type of AI use.
>>
>> Our measurements show that today=E2=80=99s system combines heavy depende=
nce on
>> robots.txt,
>> fragmented AI-specific policies, inconsistent naming, and only partially
>> coherent configurations.
>>
>>
>> To this end, here is some input for the BCP:
>>
>> 1.) Define how crawlers should handle conflicting signals and precedence=
:
>> Explicitly establish that discovery/descriptor files do not grant access
>> and
>> recommend conservative handling of contradictory permission signals.
>>
>> 2.) Separate crawler identity by purpose. Search, training, and
>> user-triggered/assistant retrieval should be distinguishable where they
>> represent materially different uses. Measurements show that that
>> publishers
>> are expressing use-specific preferences: e.g., search=3Dyes, ai-train=3D=
no.
>>
>> 3.) Strengthen identity requirements and encourage a canonical crawler
>> registry: Documentation alone isn=E2=80=99t solving crawler identity, as=
 is
>> apparent
>> from the very large number of unrecognized User-Agent strings in our
>> measurements. Plus, when an operator runs crawlers for materially
>> different
>> purposes, those crawlers should have distinguishable identities, e.g.:
>>
>>   ExampleSearchBot
>>   ExampleTrainingBot
>>   ExampleAssistantFetcher
>>
>>   rather than one generic ExampleBot.
>>
>> 4.) Explicitly prohibit identity evasion/policy shopping: A crawler
>> shouldn=E2=80=99t
>> spoof identities, rotate identifiers to evade rules, or choose whichever
>> applicable policy gives it the most permissive result.
>>
>> 5.) Encourage policy revalidation rather than indefinitely caching
>> permission
>> decisions: Crawlers should periodically revalidate cached crawler-policy
>> resources and must not assume that a previously observed permission
>> remains
>> valid indefinitely.
>>
>> 6.) Require transparent interpretation and validation. Crawler operators
>> should
>> document supported policy mechanisms, conflict behavior and caching, and
>> ideally provide a URL testing tool showing publishers exactly how their
>> crawler interprets the site=E2=80=99s policies, e.g., =E2=80=9CThis is h=
ow our crawler
>> interprets your site.=E2=80=9D
>>
>> Kind regards, Vaibhav
>>
>> > On 27. Jul 2026, at 14:42, Sauron <sauron=3D40google.com@dmarc.ietf.or=
g>
>> wrote:
>> >
>> > Hi all,
>> > Thanks for the engagement and support during the working group session
>> at IETF126.
>> > As mentioned during the meeting, we are looking for even more feedback
>> on the Crawler Best Practices draft:
>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/ /
>> https://github.com/garyillyes/cbcp Specifically, input on the scope,
>> rate control conventions, and handling of non-malicious crawlers would b=
e
>> really useful to help move the draft forward. Please send any comments t=
o
>> the list or open issues directly on GitHub.
>> > Thanks,
>> > Gary
>> > PS: if you have ideas where to send the following two drafts that CBCP
>> depends on, that would be hugely welcome
>> > 1. JAFAR:
>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
>> > 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
>> > _______________________________________________
>> > Web-bot-auth mailing list -- web-bot-auth@ietf.org
>> > To unsubscribe send an email to web-bot-auth-leave@ietf.org
>>
>> _______________________________________________
>> Web-bot-auth mailing list -- web-bot-auth@ietf.org
>> To unsubscribe send an email to web-bot-auth-leave@ietf.org
>>
> _______________________________________________
> Web-bot-auth mailing list -- web-bot-auth@ietf.org
> To unsubscribe send an email to web-bot-auth-leave@ietf.org
>
>

--000000000000de4d8f0659896d8c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p dir=3D"ltr">Nick, thank you, and the tag is the sharper=
 version of this. I had the scope sentence and the consumption path in 5.5.=
3. That a conformant proof must carry tag=3Dhttp-message-signatures-directo=
ry is what makes it concrete, since that value has no story on a jwks_uri r=
esponse.</p><p dir=3D"ltr">Agreed that the ambiguity is the problem rather =
than either reading.</p><p dir=3D"ltr">This is not hypothetical at our end.=
 We serve a hosted directory carrying four operators&#39; keys under one id=
entifier. Per-tenant authorities would resolve that, but they require an or=
igin per operator, so the route for operators without one is per-tenant jwk=
s_uri, which lands squarely in the gap.</p><p dir=3D"ltr">So if Thibault la=
nds on directory-only, I am happy to draft the sentence for wherever a jwks=
_uri or cimd operator would actually find it: that key material published u=
nder those types cannot carry an Appendix B proof, and that a verifier rece=
iving it redistributed falls back to the thumbprint identifier in 4.3, whic=
h does not survive rotation. Happy to open it as a PR, or leave it with the=
 editors.</p><p dir=3D"ltr">Shivdeep</p></div><br><div class=3D"gmail_quote=
 gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Aug 2=
1, 2026 at 8:49=E2=80=AFAM Nick Mathews &lt;<a href=3D"mailto:nick@lifeligh=
tlabs.com">nick@lifelightlabs.com</a>&gt; wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">


<div>
<div name=3D"messageBodySection">
<div>Shivdeep, Vaibhav, Gary,</div>
<div>=C2=A0</div>
<div>Speaking to the Appendix B question as the contributor of the test vec=
tors that validate it (E.2.3), not for the spec text itself.</div>
<div>=C2=A0</div>
<div>I think both of your readings have support in the text as written, whi=
ch is the problem. The framing sentence describes the concern: what a verif=
ier checks when it wants the domain a key is published under, rather than t=
he URL on its own. Nothing in the B.1 construction consumes a directory-for=
mat field; it is a per-key signature over @authority and content-digest wit=
h created and expires, and that applies mechanically to any HTTPS response =
that serves key material, including a hosted jwks_uri. But the normative ho=
oks are directory-shaped: the required tag value is literally http-message-=
signatures-directory, and the consumption path runs through 5.5.3. So an im=
plementer today cannot claim conformant proofs on a jwks_uri response even =
though the mechanism would work there.</div>
<div>=C2=A0</div>
<div>Our verifier implements B.1 on directory responses only, because that =
is what the text licenses and what the vectors cover. If the intent is any-=
type, the appendix should say so and give the tag a story for non-directory=
 responses. If the intent is directory-only, then the consequence you descr=
ibe, jwks_uri and cimd operators falling back to thumbprint identity with n=
o rotation per 4.3, deserves a sentence where those operators will find it.=
 Either sentence is cheap and the ambiguity is not. I would like to see Thi=
bault confirm the intended scope before the BCP points operators at the mec=
hanism.</div>
<div>=C2=A0</div>
<div>On the point 3 convergence, one merchant-side datapoint: distinguishab=
le per-purpose identities are not just registry hygiene. Our policy layer k=
eys rules on the resolved identifier, so a crawler that separates search fr=
om training from assistant retrieval gets separate policy rows a merchant c=
an treat differently. A single generic identity collapses that to one row, =
and the measurement in the paper matches what we see from merchants: they w=
ant to say yes to one purpose and no to another, and the identifier is the =
only handle they have.</div>
<div>=C2=A0</div>
<div>Nick</div>
</div>
<div name=3D"messageReplySection">On Aug 20, 2026 at 6:26=E2=80=AFAM -0400,=
 Shivdeep Singh &lt;<a href=3D"mailto:shivdeepsachdeva@gmail.com" target=3D=
"_blank">shivdeepsachdeva@gmail.com</a>&gt;, wrote:<br>
<blockquote type=3D"cite">
<div dir=3D"ltr">Vaibhav, Gary, Nick,<br>
<br>
Point 3 and Nick&#39;s argument converge on the same mechanism, and I have =
a scope question about it.<br>
<br>
Under draft-meunier-webbotauth-httpsig-protocol-02 an identity is the resol=
ved Signature-Agent URL, so distinguishable per-purpose crawlers mean disti=
nct URLs. An operator without a stable origin cannot use the directory type=
, since 5.5 requires that member value to be an origin, which leaves jwks_u=
ri or cimd.<br>
<br>
Appendix B opens with &quot;It applies to the directory type in Section 5.5=
.&quot; I read that two ways. Either the possession proof in B.1 is availab=
le only to the directory type, in which case jwks_uri and cimd material can=
 never satisfy 5.5.3 and falls back to thumbprint identity, which 4.3 says =
has no rotation. Or the scope line is about domain binding specifically, th=
e concern of that appendix, and B.1&#39;s mechanism is available to any typ=
e, in which case saying so would help implementers.<br>
<br>
I do not think this changes either recommendation. It affects who can act o=
n them, so it seemed worth asking before the BCP points operators at the me=
chanism.<br>
<br>
Shivdeep Singh</div>
<br>
<div class=3D"gmail_quote">
<div dir=3D"ltr" class=3D"gmail_attr">On Thu, Aug 20, 2026 at 2:53=E2=80=AF=
PM Vaibhav Bajpai &lt;<a href=3D"mailto:contact@vaibhavbajpai.com" target=
=3D"_blank">contact@vaibhavbajpai.com</a>&gt; wrote:<br></div>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex">Hi everyone,<br>
<br>
This paper just got published, and we believe would be<br>
valuable input to the BCP draft:<br>
<br>
&gt;From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in =
the Age of AI<br>
<a href=3D"https://dl.acm.org/doi/pdf/10.1145/3831956.3831960" rel=3D"noref=
errer" target=3D"_blank">https://dl.acm.org/doi/pdf/10.1145/3831956.3831960=
</a><br>
<br>
The measurements suggest that being identifiable and respecting robots.txt =
is<br>
necessary, but no longer sufficient. The harder emerging problem is ensurin=
g<br>
that authenticated crawlers interpret a fragmented set of publisher signals=
<br>
consistently, predictably, and according to the intended type of AI use.<br=
>
<br>
Our measurements show that today=E2=80=99s system combines heavy dependence=
 on robots.txt,<br>
fragmented AI-specific policies, inconsistent naming, and only partially<br=
>
coherent configurations.<br>
<br>
<br>
To this end, here is some input for the BCP:<br>
<br>
1.) Define how crawlers should handle conflicting signals and precedence:<b=
r>
Explicitly establish that discovery/descriptor files do not grant access an=
d<br>
recommend conservative handling of contradictory permission signals.<br>
<br>
2.) Separate crawler identity by purpose. Search, training, and<br>
user-triggered/assistant retrieval should be distinguishable where they<br>
represent materially different uses. Measurements show that that publishers=
<br>
are expressing use-specific preferences: e.g., search=3Dyes, ai-train=3Dno.=
<br>
<br>
3.) Strengthen identity requirements and encourage a canonical crawler<br>
registry: Documentation alone isn=E2=80=99t solving crawler identity, as is=
 apparent<br>
from the very large number of unrecognized User-Agent strings in our<br>
measurements. Plus, when an operator runs crawlers for materially different=
<br>
purposes, those crawlers should have distinguishable identities, e.g.:<br>
<br>
=C2=A0 ExampleSearchBot<br>
=C2=A0 ExampleTrainingBot<br>
=C2=A0 ExampleAssistantFetcher<br>
<br>
=C2=A0 rather than one generic ExampleBot.<br>
<br>
4.) Explicitly prohibit identity evasion/policy shopping: A crawler shouldn=
=E2=80=99t<br>
spoof identities, rotate identifiers to evade rules, or choose whichever<br=
>
applicable policy gives it the most permissive result.<br>
<br>
5.) Encourage policy revalidation rather than indefinitely caching permissi=
on<br>
decisions: Crawlers should periodically revalidate cached crawler-policy<br=
>
resources and must not assume that a previously observed permission remains=
<br>
valid indefinitely.<br>
<br>
6.) Require transparent interpretation and validation. Crawler operators sh=
ould<br>
document supported policy mechanisms, conflict behavior and caching, and<br=
>
ideally provide a URL testing tool showing publishers exactly how their<br>
crawler interprets the site=E2=80=99s policies, e.g., =E2=80=9CThis is how =
our crawler<br>
interprets your site.=E2=80=9D<br>
<br>
Kind regards, Vaibhav<br>
<br>
&gt; On 27. Jul 2026, at 14:42, Sauron &lt;sauron=3D<a href=3D"mailto:40goo=
gle.com@dmarc.ietf.org" target=3D"_blank">40google.com@dmarc.ietf.org</a>&g=
t; wrote:<br>
&gt;<br>
&gt; Hi all,<br>
&gt; Thanks for the engagement and support during the working group session=
 at IETF126.<br>
&gt; As mentioned during the meeting, we are looking for even more feedback=
 on the Crawler Best Practices draft: <a href=3D"https://datatracker.ietf.o=
rg/doc/draft-illyes-webbotauth-cbcp/" rel=3D"noreferrer" target=3D"_blank">=
https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/</a> / <a hre=
f=3D"https://github.com/garyillyes/cbcp" rel=3D"noreferrer" target=3D"_blan=
k">https://github.com/garyillyes/cbcp</a> Specifically, input on the scope,=
 rate control conventions, and handling of non-malicious crawlers would be =
really useful to help move the draft forward. Please send any comments to t=
he list or open issues directly on GitHub.<br>
&gt; Thanks,<br>
&gt; Gary<br>
&gt; PS: if you have ideas where to send the following two drafts that CBCP=
 depends on, that would be hugely welcome<br>
&gt; 1. JAFAR: <a href=3D"https://datatracker.ietf.org/doc/draft-illyes-web=
botauth-jafar/" rel=3D"noreferrer" target=3D"_blank">https://datatracker.ie=
tf.org/doc/draft-illyes-webbotauth-jafar/</a><br>
&gt; 2. REP-ext <a href=3D"https://datatracker.ietf.org/doc/draft-illyes-re=
pext/" rel=3D"noreferrer" target=3D"_blank">https://datatracker.ietf.org/do=
c/draft-illyes-repext/</a><br>
&gt; _______________________________________________<br>
&gt; Web-bot-auth mailing list -- <a href=3D"mailto:web-bot-auth@ietf.org" =
target=3D"_blank">web-bot-auth@ietf.org</a><br>
&gt; To unsubscribe send an email to <a href=3D"mailto:web-bot-auth-leave@i=
etf.org" target=3D"_blank">web-bot-auth-leave@ietf.org</a><br>
<br>
_______________________________________________<br>
Web-bot-auth mailing list -- <a href=3D"mailto:web-bot-auth@ietf.org" targe=
t=3D"_blank">web-bot-auth@ietf.org</a><br>
To unsubscribe send an email to <a href=3D"mailto:web-bot-auth-leave@ietf.o=
rg" target=3D"_blank">web-bot-auth-leave@ietf.org</a><br></blockquote>
</div>
_______________________________________________<br>
Web-bot-auth mailing list -- <a href=3D"mailto:web-bot-auth@ietf.org" targe=
t=3D"_blank">web-bot-auth@ietf.org</a><br>
To unsubscribe send an email to <a href=3D"mailto:web-bot-auth-leave@ietf.o=
rg" target=3D"_blank">web-bot-auth-leave@ietf.org</a><br></blockquote>
</div>
</div>

</blockquote></div>

--000000000000de4d8f0659896d8c--