[Web-bot-auth] Re: Follow-up from IETF 126: Comments on draft-illyes-webbotauth-cbcp

[Shivdeep Singh <[e-mail]>]

Joshua, Nick, thank you both. Two independent implementers landing on the
narrow reading from the text alone is the signal here, and I think the
repository explains why.

The intent looks like the broad reading, stated outside the draft. PR #98,
merged in June, says in its description: "We still strongly suggest that
jwks_uri response is signed to bound keys to a domain." And on issue #100
Thibault wrote that a signature over the requested authority "applies both
to the delegated and same-origin," and that he would not scope the signing
recommendation only to cross-origin discovery.

Reading Appendix B again with that in mind, I think the ambiguity is
structural rather than accidental. The appendix is titled "Validating the
domain binding," and domain binding genuinely is directory-only, since 4.5
rests on the well-known path being reserved. But B.1 is a possession proof,
and possession is not type-specific. Two different properties are sharing
one appendix, which is why the scope sentence reads as covering both.

If that is right, the fix is not to extend the appendix. It is to separate
the proof from the binding, so the proof is available to any response
serving key material while the binding stays where it belongs. Thibault
asked on #100 whether clarifying that distinction "would fit better" and I
do not think anyone answered, so I will put the proposed text there rather
than open a competing thread.

Joshua, on enrollment: Cloudflare's documentation does instruct signing the
directory response with one signature per key. Whether enrollment rejects
an unsigned directory in practice I have not tested, so I would put it no
stronger than that.

And your point that a minted origin binds 4.5 to the host rather than the
operator is the sharper version of what my draft calls principal mapping. I
will carry it into my revision with attribution.

Shivdeep

On Mon, Aug 24, 2026 at 11:46 AM Joshua Ashcroft <[e-mail]>
wrote:

> Shivdeep, Nick, Vaibhav,
>
> Second implementer datapoint on the scope question first: we implement
> B.1 on directory-type responses only, same reading as Nick's verifier,
> for the same reason - it is what the text licenses and what the vectors
> cover. So a production verifier and now two production directories have
> independently landed on the narrow reading, which makes the ambiguity
> worth a sentence whichever way it resolves.
>
> One consequence I have not seen raised: the scope decision has an
> enrollment multiplier. Cloudflare's enrollment requires a signature of
> your directory, the B.1 artifact, as a condition of registration. If
> B.1 is directory-only, then operators pushed to jwks_uri or cimd are
> not just losing domain binding and rotation per 4.3 - they cannot
> produce the artifact the flagship enrollment gate requires. The BCP
> would be pointing the operators least able to comply at a mechanism
> that locks them out of the program that makes signing worthwhile in the
> first place. Whichever scope is intended, the sentence belongs where
> those operators will read it.
>
> Shivdeep, your four-under-one deployment makes concrete what I was
> about to describe hypothetically, so let me answer the
> origin-per-operator cost from the other host seat. We run per-identity
> origins, and the marginal cost has been a wildcard certificate and a
> routing rule: a subdomain allocated once at onboarding, directory type
> at its well-known path, rotation and B.1 intact. The same move covers
> Vaibhav's per-purpose point, since a distinct purpose is just another
> subdomain. -02's redirect prohibition pushed us into the adjacent shape
> last week - each authority serving its own 200 with its own proof - and
> that migration was a routing rule and a second proof, not new
> infrastructure (details in my implementer's report from Aug 19). So
> from where I sit the origin requirement is real but cheap.
>
> The honest limit is a different one, and it may deserve a sentence
> wherever the scope sentence lands: a minted origin is an origin under
> the host's domain, so the 4.5 domain binding names the host, not the
> operator. The operator gets directory-type mechanics - rotation, B.1,
> the enrollment artifact - but the name those bind to is mine. Whether
> that is acceptable depends on what domain binding is for, and the
> answer is probably policy for the verifier rather than a rule here,
> but the asymmetry should be visible rather than discovered.
>
> And yes to your offer: if Thibault lands on directory-only, your
> sentence in the place a jwks_uri or cimd operator would actually find
> it is the right fix, and a PR beats leaving it with the editors. I
> second Nick's ask for the scope confirmation before the BCP points
> operators at the mechanism.
>
> On the merits of the scope itself: as Nick observed, nothing in the
> B.1 construction consumes a directory-format field - it applies
> mechanically to any HTTPS response that serves key material - and a
> proof mechanism gated on controlling an origin selects for who can
> pay, not who holds the key, which is the same scaling problem this
> protocol exists to retire.
>
> Joshua Ashcroft
>
> On Fri, Aug 21, 2026 12:17 AM, Shivdeep Singh <[e-mail]>
> wrote:
>
>> Nick, thank you, and the tag is the sharper version of this. I had the
>> scope sentence and the consumption path in 5.5.3. That a conformant proof
>> must carry tag=http-message-signatures-directory is what makes it concrete,
>> since that value has no story on a jwks_uri response.
>>
>> Agreed that the ambiguity is the problem rather than either reading.
>>
>> This is not hypothetical at our end. We serve a hosted directory carrying
>> four operators' keys under one identifier. Per-tenant authorities would
>> resolve that, but they require an origin per operator, so the route for
>> operators without one is per-tenant jwks_uri, which lands squarely in the
>> gap.
>>
>> So if Thibault lands on directory-only, I am happy to draft the sentence
>> for wherever a jwks_uri or cimd operator would actually find it: that key
>> material published under those types cannot carry an Appendix B proof, and
>> that a verifier receiving it redistributed falls back to the thumbprint
>> identifier in 4.3, which does not survive rotation. Happy to open it as a
>> PR, or leave it with the editors.
>>
>> Shivdeep
>>
>> On Fri, Aug 21, 2026 at 8:49 AM Nick Mathews <[e-mail]>
>> wrote:
>>
>>> Shivdeep, Vaibhav, Gary,
>>>
>>> Speaking to the Appendix B question as the contributor of the test
>>> vectors that validate it (E.2.3), not for the spec text itself.
>>>
>>> I think both of your readings have support in the text as written, which
>>> is the problem. The framing sentence describes the concern: what a verifier
>>> checks when it wants the domain a key is published under, rather than the
>>> URL on its own. Nothing in the B.1 construction consumes a directory-format
>>> field; it is a per-key signature over @authority and content-digest with
>>> created and expires, and that applies mechanically to any HTTPS response
>>> that serves key material, including a hosted jwks_uri. But the normative
>>> hooks are directory-shaped: the required tag value is literally
>>> http-message-signatures-directory, and the consumption path runs through
>>> 5.5.3. So an implementer today cannot claim conformant proofs on a jwks_uri
>>> response even though the mechanism would work there.
>>>
>>> Our verifier implements B.1 on directory responses only, because that is
>>> what the text licenses and what the vectors cover. If the intent is
>>> any-type, the appendix should say so and give the tag a story for
>>> non-directory responses. If the intent is directory-only, then the
>>> consequence you describe, jwks_uri and cimd operators falling back to
>>> thumbprint identity with no rotation per 4.3, deserves a sentence where
>>> those operators will find it. Either sentence is cheap and the ambiguity is
>>> not. I would like to see Thibault confirm the intended scope before the BCP
>>> points operators at the mechanism.
>>>
>>> On the point 3 convergence, one merchant-side datapoint: distinguishable
>>> per-purpose identities are not just registry hygiene. Our policy layer keys
>>> rules on the resolved identifier, so a crawler that separates search from
>>> training from assistant retrieval gets separate policy rows a merchant can
>>> treat differently. A single generic identity collapses that to one row, and
>>> the measurement in the paper matches what we see from merchants: they want
>>> to say yes to one purpose and no to another, and the identifier is the only
>>> handle they have.
>>>
>>> Nick
>>> On Aug 20, 2026 at 6:26 AM -0400, Shivdeep Singh <
>>> [e-mail]>, wrote:
>>>
>>> Vaibhav, Gary, Nick,
>>>
>>> Point 3 and Nick's argument converge on the same mechanism, and I have a
>>> scope question about it.
>>>
>>> Under draft-meunier-webbotauth-httpsig-protocol-02 an identity is the
>>> resolved Signature-Agent URL, so distinguishable per-purpose crawlers mean
>>> distinct URLs. An operator without a stable origin cannot use the directory
>>> type, since 5.5 requires that member value to be an origin, which leaves
>>> jwks_uri or cimd.
>>>
>>> Appendix B opens with "It applies to the directory type in Section 5.5."
>>> I read that two ways. Either the possession proof in B.1 is available only
>>> to the directory type, in which case jwks_uri and cimd material can never
>>> satisfy 5.5.3 and falls back to thumbprint identity, which 4.3 says has no
>>> rotation. Or the scope line is about domain binding specifically, the
>>> concern of that appendix, and B.1's mechanism is available to any type, in
>>> which case saying so would help implementers.
>>>
>>> I do not think this changes either recommendation. It affects who can
>>> act on them, so it seemed worth asking before the BCP points operators at
>>> the mechanism.
>>>
>>> Shivdeep Singh
>>>
>>> On Thu, Aug 20, 2026 at 2:53 PM Vaibhav Bajpai <
>>> [e-mail]> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> This paper just got published, and we believe would be
>>>> valuable input to the BCP draft:
>>>>
>>>> >From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in
>>>> the Age of AI
>>>> https://dl.acm.org/doi/pdf/10.1145/3831956.3831960
>>>>
>>>> The measurements suggest that being identifiable and respecting
>>>> robots.txt is
>>>> necessary, but no longer sufficient. The harder emerging problem is
>>>> ensuring
>>>> that authenticated crawlers interpret a fragmented set of publisher
>>>> signals
>>>> consistently, predictably, and according to the intended type of AI use.
>>>>
>>>> Our measurements show that today’s system combines heavy dependence on
>>>> robots.txt,
>>>> fragmented AI-specific policies, inconsistent naming, and only partially
>>>> coherent configurations.
>>>>
>>>>
>>>> To this end, here is some input for the BCP:
>>>>
>>>> 1.) Define how crawlers should handle conflicting signals and
>>>> precedence:
>>>> Explicitly establish that discovery/descriptor files do not grant
>>>> access and
>>>> recommend conservative handling of contradictory permission signals.
>>>>
>>>> 2.) Separate crawler identity by purpose. Search, training, and
>>>> user-triggered/assistant retrieval should be distinguishable where they
>>>> represent materially different uses. Measurements show that that
>>>> publishers
>>>> are expressing use-specific preferences: e.g., search=yes, ai-train=no.
>>>>
>>>> 3.) Strengthen identity requirements and encourage a canonical crawler
>>>> registry: Documentation alone isn’t solving crawler identity, as is
>>>> apparent
>>>> from the very large number of unrecognized User-Agent strings in our
>>>> measurements. Plus, when an operator runs crawlers for materially
>>>> different
>>>> purposes, those crawlers should have distinguishable identities, e.g.:
>>>>
>>>>   ExampleSearchBot
>>>>   ExampleTrainingBot
>>>>   ExampleAssistantFetcher
>>>>
>>>>   rather than one generic ExampleBot.
>>>>
>>>> 4.) Explicitly prohibit identity evasion/policy shopping: A crawler
>>>> shouldn’t
>>>> spoof identities, rotate identifiers to evade rules, or choose whichever
>>>> applicable policy gives it the most permissive result.
>>>>
>>>> 5.) Encourage policy revalidation rather than indefinitely caching
>>>> permission
>>>> decisions: Crawlers should periodically revalidate cached crawler-policy
>>>> resources and must not assume that a previously observed permission
>>>> remains
>>>> valid indefinitely.
>>>>
>>>> 6.) Require transparent interpretation and validation. Crawler
>>>> operators should
>>>> document supported policy mechanisms, conflict behavior and caching, and
>>>> ideally provide a URL testing tool showing publishers exactly how their
>>>> crawler interprets the site’s policies, e.g., “This is how our crawler
>>>> interprets your site.”
>>>>
>>>> Kind regards, Vaibhav
>>>>
>>>> > On 27. Jul 2026, at 14:42, Sauron <sauron=[e-mail]>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> > Thanks for the engagement and support during the working group
>>>> session at IETF126.
>>>> > As mentioned during the meeting, we are looking for even more
>>>> feedback on the Crawler Best Practices draft:
>>>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/ /
>>>> https://github.com/garyillyes/cbcp Specifically, input on the scope,
>>>> rate control conventions, and handling of non-malicious crawlers would be
>>>> really useful to help move the draft forward. Please send any comments to
>>>> the list or open issues directly on GitHub.
>>>> > Thanks,
>>>> > Gary
>>>> > PS: if you have ideas where to send the following two drafts that
>>>> CBCP depends on, that would be hugely welcome
>>>> > 1. JAFAR:
>>>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
>>>> > 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
>>>> > _______________________________________________
>>>> > Web-bot-auth mailing list -- [e-mail]
>>>> > To unsubscribe send an email to [e-mail]
>>>>
>>>> _______________________________________________
>>>> Web-bot-auth mailing list -- [e-mail]
>>>> To unsubscribe send an email to [e-mail]
>>>>
>>> _______________________________________________
>>> Web-bot-auth mailing list -- [e-mail]
>>> To unsubscribe send an email to [e-mail]
>>>
>>> _______________________________________________
>> Web-bot-auth mailing list -- [e-mail]
>> To unsubscribe send an email to [e-mail]
>>
>