[Web-bot-auth] Re: Follow-up from IETF 126: Comments on draft-illyes-webbotauth-cbcp

[Joshua Ashcroft <[e-mail]>]

Shivdeep, Nick, Vaibhav,

Second implementer datapoint on the scope question first: we implement
B.1 on directory-type responses only, same reading as Nick's verifier,
for the same reason - it is what the text licenses and what the vectors
cover. So a production verifier and now two production directories have
independently landed on the narrow reading, which makes the ambiguity
worth a sentence whichever way it resolves.

One consequence I have not seen raised: the scope decision has an
enrollment multiplier. Cloudflare's enrollment requires a signature of
your directory, the B.1 artifact, as a condition of registration. If
B.1 is directory-only, then operators pushed to jwks_uri or cimd are
not just losing domain binding and rotation per 4.3 - they cannot
produce the artifact the flagship enrollment gate requires. The BCP
would be pointing the operators least able to comply at a mechanism
that locks them out of the program that makes signing worthwhile in the
first place. Whichever scope is intended, the sentence belongs where
those operators will read it.

Shivdeep, your four-under-one deployment makes concrete what I was
about to describe hypothetically, so let me answer the
origin-per-operator cost from the other host seat. We run per-identity
origins, and the marginal cost has been a wildcard certificate and a
routing rule: a subdomain allocated once at onboarding, directory type
at its well-known path, rotation and B.1 intact. The same move covers
Vaibhav's per-purpose point, since a distinct purpose is just another
subdomain. -02's redirect prohibition pushed us into the adjacent shape
last week - each authority serving its own 200 with its own proof - and
that migration was a routing rule and a second proof, not new
infrastructure (details in my implementer's report from Aug 19). So
from where I sit the origin requirement is real but cheap.

The honest limit is a different one, and it may deserve a sentence
wherever the scope sentence lands: a minted origin is an origin under
the host's domain, so the 4.5 domain binding names the host, not the
operator. The operator gets directory-type mechanics - rotation, B.1,
the enrollment artifact - but the name those bind to is mine. Whether
that is acceptable depends on what domain binding is for, and the
answer is probably policy for the verifier rather than a rule here,
but the asymmetry should be visible rather than discovered.

And yes to your offer: if Thibault lands on directory-only, your
sentence in the place a jwks_uri or cimd operator would actually find
it is the right fix, and a PR beats leaving it with the editors. I
second Nick's ask for the scope confirmation before the BCP points
operators at the mechanism.

On the merits of the scope itself: as Nick observed, nothing in the
B.1 construction consumes a directory-format field - it applies
mechanically to any HTTPS response that serves key material - and a
proof mechanism gated on controlling an origin selects for who can
pay, not who holds the key, which is the same scaling problem this
protocol exists to retire.

Joshua Ashcroft

On Fri, Aug 21, 2026 12:17 AM, Shivdeep Singh <[e-mail]>
wrote:

> Nick, thank you, and the tag is the sharper version of this. I had the
> scope sentence and the consumption path in 5.5.3. That a conformant proof
> must carry tag=http-message-signatures-directory is what makes it concrete,
> since that value has no story on a jwks_uri response.
>
> Agreed that the ambiguity is the problem rather than either reading.
>
> This is not hypothetical at our end. We serve a hosted directory carrying
> four operators' keys under one identifier. Per-tenant authorities would
> resolve that, but they require an origin per operator, so the route for
> operators without one is per-tenant jwks_uri, which lands squarely in the
> gap.
>
> So if Thibault lands on directory-only, I am happy to draft the sentence
> for wherever a jwks_uri or cimd operator would actually find it: that key
> material published under those types cannot carry an Appendix B proof, and
> that a verifier receiving it redistributed falls back to the thumbprint
> identifier in 4.3, which does not survive rotation. Happy to open it as a
> PR, or leave it with the editors.
>
> Shivdeep
>
> On Fri, Aug 21, 2026 at 8:49 AM Nick Mathews <[e-mail]>
> wrote:
>
>> Shivdeep, Vaibhav, Gary,
>>
>> Speaking to the Appendix B question as the contributor of the test
>> vectors that validate it (E.2.3), not for the spec text itself.
>>
>> I think both of your readings have support in the text as written, which
>> is the problem. The framing sentence describes the concern: what a verifier
>> checks when it wants the domain a key is published under, rather than the
>> URL on its own. Nothing in the B.1 construction consumes a directory-format
>> field; it is a per-key signature over @authority and content-digest with
>> created and expires, and that applies mechanically to any HTTPS response
>> that serves key material, including a hosted jwks_uri. But the normative
>> hooks are directory-shaped: the required tag value is literally
>> http-message-signatures-directory, and the consumption path runs through
>> 5.5.3. So an implementer today cannot claim conformant proofs on a jwks_uri
>> response even though the mechanism would work there.
>>
>> Our verifier implements B.1 on directory responses only, because that is
>> what the text licenses and what the vectors cover. If the intent is
>> any-type, the appendix should say so and give the tag a story for
>> non-directory responses. If the intent is directory-only, then the
>> consequence you describe, jwks_uri and cimd operators falling back to
>> thumbprint identity with no rotation per 4.3, deserves a sentence where
>> those operators will find it. Either sentence is cheap and the ambiguity is
>> not. I would like to see Thibault confirm the intended scope before the BCP
>> points operators at the mechanism.
>>
>> On the point 3 convergence, one merchant-side datapoint: distinguishable
>> per-purpose identities are not just registry hygiene. Our policy layer keys
>> rules on the resolved identifier, so a crawler that separates search from
>> training from assistant retrieval gets separate policy rows a merchant can
>> treat differently. A single generic identity collapses that to one row, and
>> the measurement in the paper matches what we see from merchants: they want
>> to say yes to one purpose and no to another, and the identifier is the only
>> handle they have.
>>
>> Nick
>> On Aug 20, 2026 at 6:26 AM -0400, Shivdeep Singh <
>> [e-mail]>, wrote:
>>
>> Vaibhav, Gary, Nick,
>>
>> Point 3 and Nick's argument converge on the same mechanism, and I have a
>> scope question about it.
>>
>> Under draft-meunier-webbotauth-httpsig-protocol-02 an identity is the
>> resolved Signature-Agent URL, so distinguishable per-purpose crawlers mean
>> distinct URLs. An operator without a stable origin cannot use the directory
>> type, since 5.5 requires that member value to be an origin, which leaves
>> jwks_uri or cimd.
>>
>> Appendix B opens with "It applies to the directory type in Section 5.5."
>> I read that two ways. Either the possession proof in B.1 is available only
>> to the directory type, in which case jwks_uri and cimd material can never
>> satisfy 5.5.3 and falls back to thumbprint identity, which 4.3 says has no
>> rotation. Or the scope line is about domain binding specifically, the
>> concern of that appendix, and B.1's mechanism is available to any type, in
>> which case saying so would help implementers.
>>
>> I do not think this changes either recommendation. It affects who can act
>> on them, so it seemed worth asking before the BCP points operators at the
>> mechanism.
>>
>> Shivdeep Singh
>>
>> On Thu, Aug 20, 2026 at 2:53 PM Vaibhav Bajpai <[e-mail]>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> This paper just got published, and we believe would be
>>> valuable input to the BCP draft:
>>>
>>> >From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in
>>> the Age of AI
>>> https://dl.acm.org/doi/pdf/10.1145/3831956.3831960
>>>
>>> The measurements suggest that being identifiable and respecting
>>> robots.txt is
>>> necessary, but no longer sufficient. The harder emerging problem is
>>> ensuring
>>> that authenticated crawlers interpret a fragmented set of publisher
>>> signals
>>> consistently, predictably, and according to the intended type of AI use.
>>>
>>> Our measurements show that today’s system combines heavy dependence on
>>> robots.txt,
>>> fragmented AI-specific policies, inconsistent naming, and only partially
>>> coherent configurations.
>>>
>>>
>>> To this end, here is some input for the BCP:
>>>
>>> 1.) Define how crawlers should handle conflicting signals and precedence:
>>> Explicitly establish that discovery/descriptor files do not grant access
>>> and
>>> recommend conservative handling of contradictory permission signals.
>>>
>>> 2.) Separate crawler identity by purpose. Search, training, and
>>> user-triggered/assistant retrieval should be distinguishable where they
>>> represent materially different uses. Measurements show that that
>>> publishers
>>> are expressing use-specific preferences: e.g., search=yes, ai-train=no.
>>>
>>> 3.) Strengthen identity requirements and encourage a canonical crawler
>>> registry: Documentation alone isn’t solving crawler identity, as is
>>> apparent
>>> from the very large number of unrecognized User-Agent strings in our
>>> measurements. Plus, when an operator runs crawlers for materially
>>> different
>>> purposes, those crawlers should have distinguishable identities, e.g.:
>>>
>>>   ExampleSearchBot
>>>   ExampleTrainingBot
>>>   ExampleAssistantFetcher
>>>
>>>   rather than one generic ExampleBot.
>>>
>>> 4.) Explicitly prohibit identity evasion/policy shopping: A crawler
>>> shouldn’t
>>> spoof identities, rotate identifiers to evade rules, or choose whichever
>>> applicable policy gives it the most permissive result.
>>>
>>> 5.) Encourage policy revalidation rather than indefinitely caching
>>> permission
>>> decisions: Crawlers should periodically revalidate cached crawler-policy
>>> resources and must not assume that a previously observed permission
>>> remains
>>> valid indefinitely.
>>>
>>> 6.) Require transparent interpretation and validation. Crawler operators
>>> should
>>> document supported policy mechanisms, conflict behavior and caching, and
>>> ideally provide a URL testing tool showing publishers exactly how their
>>> crawler interprets the site’s policies, e.g., “This is how our crawler
>>> interprets your site.”
>>>
>>> Kind regards, Vaibhav
>>>
>>> > On 27. Jul 2026, at 14:42, Sauron <sauron=[e-mail]>
>>> wrote:
>>> >
>>> > Hi all,
>>> > Thanks for the engagement and support during the working group session
>>> at IETF126.
>>> > As mentioned during the meeting, we are looking for even more feedback
>>> on the Crawler Best Practices draft:
>>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/ /
>>> https://github.com/garyillyes/cbcp Specifically, input on the scope,
>>> rate control conventions, and handling of non-malicious crawlers would be
>>> really useful to help move the draft forward. Please send any comments to
>>> the list or open issues directly on GitHub.
>>> > Thanks,
>>> > Gary
>>> > PS: if you have ideas where to send the following two drafts that CBCP
>>> depends on, that would be hugely welcome
>>> > 1. JAFAR:
>>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
>>> > 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
>>> > _______________________________________________
>>> > Web-bot-auth mailing list -- [e-mail]
>>> > To unsubscribe send an email to [e-mail]
>>>
>>> _______________________________________________
>>> Web-bot-auth mailing list -- [e-mail]
>>> To unsubscribe send an email to [e-mail]
>>>
>> _______________________________________________
>> Web-bot-auth mailing list -- [e-mail]
>> To unsubscribe send an email to [e-mail]
>>
>> _______________________________________________
> Web-bot-auth mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>