[Web-bot-auth] Re: Follow-up from IETF 126: Comments on draft-illyes-webbotauth-cbcp

[Shivdeep Singh <[e-mail]>]

Nick, thank you, and the tag is the sharper version of this. I had the
scope sentence and the consumption path in 5.5.3. That a conformant proof
must carry tag=http-message-signatures-directory is what makes it concrete,
since that value has no story on a jwks_uri response.

Agreed that the ambiguity is the problem rather than either reading.

This is not hypothetical at our end. We serve a hosted directory carrying
four operators' keys under one identifier. Per-tenant authorities would
resolve that, but they require an origin per operator, so the route for
operators without one is per-tenant jwks_uri, which lands squarely in the
gap.

So if Thibault lands on directory-only, I am happy to draft the sentence
for wherever a jwks_uri or cimd operator would actually find it: that key
material published under those types cannot carry an Appendix B proof, and
that a verifier receiving it redistributed falls back to the thumbprint
identifier in 4.3, which does not survive rotation. Happy to open it as a
PR, or leave it with the editors.

Shivdeep

On Fri, Aug 21, 2026 at 8:49 AM Nick Mathews <[e-mail]> wrote:

> Shivdeep, Vaibhav, Gary,
>
> Speaking to the Appendix B question as the contributor of the test vectors
> that validate it (E.2.3), not for the spec text itself.
>
> I think both of your readings have support in the text as written, which
> is the problem. The framing sentence describes the concern: what a verifier
> checks when it wants the domain a key is published under, rather than the
> URL on its own. Nothing in the B.1 construction consumes a directory-format
> field; it is a per-key signature over @authority and content-digest with
> created and expires, and that applies mechanically to any HTTPS response
> that serves key material, including a hosted jwks_uri. But the normative
> hooks are directory-shaped: the required tag value is literally
> http-message-signatures-directory, and the consumption path runs through
> 5.5.3. So an implementer today cannot claim conformant proofs on a jwks_uri
> response even though the mechanism would work there.
>
> Our verifier implements B.1 on directory responses only, because that is
> what the text licenses and what the vectors cover. If the intent is
> any-type, the appendix should say so and give the tag a story for
> non-directory responses. If the intent is directory-only, then the
> consequence you describe, jwks_uri and cimd operators falling back to
> thumbprint identity with no rotation per 4.3, deserves a sentence where
> those operators will find it. Either sentence is cheap and the ambiguity is
> not. I would like to see Thibault confirm the intended scope before the BCP
> points operators at the mechanism.
>
> On the point 3 convergence, one merchant-side datapoint: distinguishable
> per-purpose identities are not just registry hygiene. Our policy layer keys
> rules on the resolved identifier, so a crawler that separates search from
> training from assistant retrieval gets separate policy rows a merchant can
> treat differently. A single generic identity collapses that to one row, and
> the measurement in the paper matches what we see from merchants: they want
> to say yes to one purpose and no to another, and the identifier is the only
> handle they have.
>
> Nick
> On Aug 20, 2026 at 6:26 AM -0400, Shivdeep Singh <
> [e-mail]>, wrote:
>
> Vaibhav, Gary, Nick,
>
> Point 3 and Nick's argument converge on the same mechanism, and I have a
> scope question about it.
>
> Under draft-meunier-webbotauth-httpsig-protocol-02 an identity is the
> resolved Signature-Agent URL, so distinguishable per-purpose crawlers mean
> distinct URLs. An operator without a stable origin cannot use the directory
> type, since 5.5 requires that member value to be an origin, which leaves
> jwks_uri or cimd.
>
> Appendix B opens with "It applies to the directory type in Section 5.5." I
> read that two ways. Either the possession proof in B.1 is available only to
> the directory type, in which case jwks_uri and cimd material can never
> satisfy 5.5.3 and falls back to thumbprint identity, which 4.3 says has no
> rotation. Or the scope line is about domain binding specifically, the
> concern of that appendix, and B.1's mechanism is available to any type, in
> which case saying so would help implementers.
>
> I do not think this changes either recommendation. It affects who can act
> on them, so it seemed worth asking before the BCP points operators at the
> mechanism.
>
> Shivdeep Singh
>
> On Thu, Aug 20, 2026 at 2:53 PM Vaibhav Bajpai <[e-mail]>
> wrote:
>
>> Hi everyone,
>>
>> This paper just got published, and we believe would be
>> valuable input to the BCP draft:
>>
>> >From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in
>> the Age of AI
>> https://dl.acm.org/doi/pdf/10.1145/3831956.3831960
>>
>> The measurements suggest that being identifiable and respecting
>> robots.txt is
>> necessary, but no longer sufficient. The harder emerging problem is
>> ensuring
>> that authenticated crawlers interpret a fragmented set of publisher
>> signals
>> consistently, predictably, and according to the intended type of AI use.
>>
>> Our measurements show that today’s system combines heavy dependence on
>> robots.txt,
>> fragmented AI-specific policies, inconsistent naming, and only partially
>> coherent configurations.
>>
>>
>> To this end, here is some input for the BCP:
>>
>> 1.) Define how crawlers should handle conflicting signals and precedence:
>> Explicitly establish that discovery/descriptor files do not grant access
>> and
>> recommend conservative handling of contradictory permission signals.
>>
>> 2.) Separate crawler identity by purpose. Search, training, and
>> user-triggered/assistant retrieval should be distinguishable where they
>> represent materially different uses. Measurements show that that
>> publishers
>> are expressing use-specific preferences: e.g., search=yes, ai-train=no.
>>
>> 3.) Strengthen identity requirements and encourage a canonical crawler
>> registry: Documentation alone isn’t solving crawler identity, as is
>> apparent
>> from the very large number of unrecognized User-Agent strings in our
>> measurements. Plus, when an operator runs crawlers for materially
>> different
>> purposes, those crawlers should have distinguishable identities, e.g.:
>>
>>   ExampleSearchBot
>>   ExampleTrainingBot
>>   ExampleAssistantFetcher
>>
>>   rather than one generic ExampleBot.
>>
>> 4.) Explicitly prohibit identity evasion/policy shopping: A crawler
>> shouldn’t
>> spoof identities, rotate identifiers to evade rules, or choose whichever
>> applicable policy gives it the most permissive result.
>>
>> 5.) Encourage policy revalidation rather than indefinitely caching
>> permission
>> decisions: Crawlers should periodically revalidate cached crawler-policy
>> resources and must not assume that a previously observed permission
>> remains
>> valid indefinitely.
>>
>> 6.) Require transparent interpretation and validation. Crawler operators
>> should
>> document supported policy mechanisms, conflict behavior and caching, and
>> ideally provide a URL testing tool showing publishers exactly how their
>> crawler interprets the site’s policies, e.g., “This is how our crawler
>> interprets your site.”
>>
>> Kind regards, Vaibhav
>>
>> > On 27. Jul 2026, at 14:42, Sauron <sauron=[e-mail]>
>> wrote:
>> >
>> > Hi all,
>> > Thanks for the engagement and support during the working group session
>> at IETF126.
>> > As mentioned during the meeting, we are looking for even more feedback
>> on the Crawler Best Practices draft:
>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/ /
>> https://github.com/garyillyes/cbcp Specifically, input on the scope,
>> rate control conventions, and handling of non-malicious crawlers would be
>> really useful to help move the draft forward. Please send any comments to
>> the list or open issues directly on GitHub.
>> > Thanks,
>> > Gary
>> > PS: if you have ideas where to send the following two drafts that CBCP
>> depends on, that would be hugely welcome
>> > 1. JAFAR:
>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
>> > 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
>> > _______________________________________________
>> > Web-bot-auth mailing list -- [e-mail]
>> > To unsubscribe send an email to [e-mail]
>>
>> _______________________________________________
>> Web-bot-auth mailing list -- [e-mail]
>> To unsubscribe send an email to [e-mail]
>>
> _______________________________________________
> Web-bot-auth mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>
>