[Web-bot-auth] Re: Follow-up from IETF 126: Comments on draft-illyes-webbotauth-cbcp

[Shivdeep Singh <[e-mail]>]

Nick,
Text is up as thibmeu/http-message-signatures-directory#131
<https://github.com/thibmeu/http-message-signatures-directory/issues/131>.
Say the word on the vectors.
Shivdeep Singh

On Tue, Aug 25, 2026 at 10:25 AM Nick Mathews <[e-mail]>
wrote:

> Shivdeep, Joshua,
>
> The repo evidence settles intent for me, and the structural diagnosis
> explains the text better than either reading did: the appendix holds two
> properties, possession and binding, and only the binding is
> directory-shaped. Separating them makes both readings correct at once.
> Putting the proposed text on #100, where Thibault already asked whether
> that distinction would fit better, is the right venue, and I will support
> it there.
>
> One offer so the change lands with its tests attached: the E.2.3 vectors
> currently exercise the possession proof on a directory response. If the
> separation is adopted, I will extend them to cover the proof on a jwks_uri
> response as well, so the type-generic claim ships with an executable
> example on each side of the split. Vectors follow text; say the word when
> yours is up.
>
> Nick
> On Aug 24, 2026 at 5:36 PM -0400, Shivdeep Singh <
> [e-mail]>, wrote:
>
> Joshua, Nick, thank you both. Two independent implementers landing on the
> narrow reading from the text alone is the signal here, and I think the
> repository explains why.
>
> The intent looks like the broad reading, stated outside the draft. PR #98,
> merged in June, says in its description: "We still strongly suggest that
> jwks_uri response is signed to bound keys to a domain." And on issue #100
> Thibault wrote that a signature over the requested authority "applies both
> to the delegated and same-origin," and that he would not scope the signing
> recommendation only to cross-origin discovery.
>
> Reading Appendix B again with that in mind, I think the ambiguity is
> structural rather than accidental. The appendix is titled "Validating the
> domain binding," and domain binding genuinely is directory-only, since 4.5
> rests on the well-known path being reserved. But B.1 is a possession proof,
> and possession is not type-specific. Two different properties are sharing
> one appendix, which is why the scope sentence reads as covering both.
>
> If that is right, the fix is not to extend the appendix. It is to separate
> the proof from the binding, so the proof is available to any response
> serving key material while the binding stays where it belongs. Thibault
> asked on #100 whether clarifying that distinction "would fit better" and I
> do not think anyone answered, so I will put the proposed text there rather
> than open a competing thread.
>
> Joshua, on enrollment: Cloudflare's documentation does instruct signing
> the directory response with one signature per key. Whether enrollment
> rejects an unsigned directory in practice I have not tested, so I would put
> it no stronger than that.
>
> And your point that a minted origin binds 4.5 to the host rather than the
> operator is the sharper version of what my draft calls principal mapping. I
> will carry it into my revision with attribution.
>
> Shivdeep
>
> On Mon, Aug 24, 2026 at 11:46 AM Joshua Ashcroft <[e-mail]>
> wrote:
>
>> Shivdeep, Nick, Vaibhav,
>>
>> Second implementer datapoint on the scope question first: we implement
>> B.1 on directory-type responses only, same reading as Nick's verifier,
>> for the same reason - it is what the text licenses and what the vectors
>> cover. So a production verifier and now two production directories have
>> independently landed on the narrow reading, which makes the ambiguity
>> worth a sentence whichever way it resolves.
>>
>> One consequence I have not seen raised: the scope decision has an
>> enrollment multiplier. Cloudflare's enrollment requires a signature of
>> your directory, the B.1 artifact, as a condition of registration. If
>> B.1 is directory-only, then operators pushed to jwks_uri or cimd are
>> not just losing domain binding and rotation per 4.3 - they cannot
>> produce the artifact the flagship enrollment gate requires. The BCP
>> would be pointing the operators least able to comply at a mechanism
>> that locks them out of the program that makes signing worthwhile in the
>> first place. Whichever scope is intended, the sentence belongs where
>> those operators will read it.
>>
>> Shivdeep, your four-under-one deployment makes concrete what I was
>> about to describe hypothetically, so let me answer the
>> origin-per-operator cost from the other host seat. We run per-identity
>> origins, and the marginal cost has been a wildcard certificate and a
>> routing rule: a subdomain allocated once at onboarding, directory type
>> at its well-known path, rotation and B.1 intact. The same move covers
>> Vaibhav's per-purpose point, since a distinct purpose is just another
>> subdomain. -02's redirect prohibition pushed us into the adjacent shape
>> last week - each authority serving its own 200 with its own proof - and
>> that migration was a routing rule and a second proof, not new
>> infrastructure (details in my implementer's report from Aug 19). So
>> from where I sit the origin requirement is real but cheap.
>>
>> The honest limit is a different one, and it may deserve a sentence
>> wherever the scope sentence lands: a minted origin is an origin under
>> the host's domain, so the 4.5 domain binding names the host, not the
>> operator. The operator gets directory-type mechanics - rotation, B.1,
>> the enrollment artifact - but the name those bind to is mine. Whether
>> that is acceptable depends on what domain binding is for, and the
>> answer is probably policy for the verifier rather than a rule here,
>> but the asymmetry should be visible rather than discovered.
>>
>> And yes to your offer: if Thibault lands on directory-only, your
>> sentence in the place a jwks_uri or cimd operator would actually find
>> it is the right fix, and a PR beats leaving it with the editors. I
>> second Nick's ask for the scope confirmation before the BCP points
>> operators at the mechanism.
>>
>> On the merits of the scope itself: as Nick observed, nothing in the
>> B.1 construction consumes a directory-format field - it applies
>> mechanically to any HTTPS response that serves key material - and a
>> proof mechanism gated on controlling an origin selects for who can
>> pay, not who holds the key, which is the same scaling problem this
>> protocol exists to retire.
>>
>> Joshua Ashcroft
>>
>> On Fri, Aug 21, 2026 12:17 AM, Shivdeep Singh <[e-mail]>
>> wrote:
>>
>>> Nick, thank you, and the tag is the sharper version of this. I had the
>>> scope sentence and the consumption path in 5.5.3. That a conformant proof
>>> must carry tag=http-message-signatures-directory is what makes it concrete,
>>> since that value has no story on a jwks_uri response.
>>>
>>> Agreed that the ambiguity is the problem rather than either reading.
>>>
>>> This is not hypothetical at our end. We serve a hosted directory
>>> carrying four operators' keys under one identifier. Per-tenant authorities
>>> would resolve that, but they require an origin per operator, so the route
>>> for operators without one is per-tenant jwks_uri, which lands squarely in
>>> the gap.
>>>
>>> So if Thibault lands on directory-only, I am happy to draft the sentence
>>> for wherever a jwks_uri or cimd operator would actually find it: that key
>>> material published under those types cannot carry an Appendix B proof, and
>>> that a verifier receiving it redistributed falls back to the thumbprint
>>> identifier in 4.3, which does not survive rotation. Happy to open it as a
>>> PR, or leave it with the editors.
>>>
>>> Shivdeep
>>>
>>> On Fri, Aug 21, 2026 at 8:49 AM Nick Mathews <[e-mail]>
>>> wrote:
>>>
>>>> Shivdeep, Vaibhav, Gary,
>>>>
>>>> Speaking to the Appendix B question as the contributor of the test
>>>> vectors that validate it (E.2.3), not for the spec text itself.
>>>>
>>>> I think both of your readings have support in the text as written,
>>>> which is the problem. The framing sentence describes the concern: what a
>>>> verifier checks when it wants the domain a key is published under, rather
>>>> than the URL on its own. Nothing in the B.1 construction consumes a
>>>> directory-format field; it is a per-key signature over @authority and
>>>> content-digest with created and expires, and that applies mechanically to
>>>> any HTTPS response that serves key material, including a hosted jwks_uri.
>>>> But the normative hooks are directory-shaped: the required tag value is
>>>> literally http-message-signatures-directory, and the consumption path runs
>>>> through 5.5.3. So an implementer today cannot claim conformant proofs on a
>>>> jwks_uri response even though the mechanism would work there.
>>>>
>>>> Our verifier implements B.1 on directory responses only, because that
>>>> is what the text licenses and what the vectors cover. If the intent is
>>>> any-type, the appendix should say so and give the tag a story for
>>>> non-directory responses. If the intent is directory-only, then the
>>>> consequence you describe, jwks_uri and cimd operators falling back to
>>>> thumbprint identity with no rotation per 4.3, deserves a sentence where
>>>> those operators will find it. Either sentence is cheap and the ambiguity is
>>>> not. I would like to see Thibault confirm the intended scope before the BCP
>>>> points operators at the mechanism.
>>>>
>>>> On the point 3 convergence, one merchant-side datapoint:
>>>> distinguishable per-purpose identities are not just registry hygiene. Our
>>>> policy layer keys rules on the resolved identifier, so a crawler that
>>>> separates search from training from assistant retrieval gets separate
>>>> policy rows a merchant can treat differently. A single generic identity
>>>> collapses that to one row, and the measurement in the paper matches what we
>>>> see from merchants: they want to say yes to one purpose and no to another,
>>>> and the identifier is the only handle they have.
>>>>
>>>> Nick
>>>> On Aug 20, 2026 at 6:26 AM -0400, Shivdeep Singh <
>>>> [e-mail]>, wrote:
>>>>
>>>> Vaibhav, Gary, Nick,
>>>>
>>>> Point 3 and Nick's argument converge on the same mechanism, and I have
>>>> a scope question about it.
>>>>
>>>> Under draft-meunier-webbotauth-httpsig-protocol-02 an identity is the
>>>> resolved Signature-Agent URL, so distinguishable per-purpose crawlers mean
>>>> distinct URLs. An operator without a stable origin cannot use the directory
>>>> type, since 5.5 requires that member value to be an origin, which leaves
>>>> jwks_uri or cimd.
>>>>
>>>> Appendix B opens with "It applies to the directory type in Section
>>>> 5.5." I read that two ways. Either the possession proof in B.1 is available
>>>> only to the directory type, in which case jwks_uri and cimd material can
>>>> never satisfy 5.5.3 and falls back to thumbprint identity, which 4.3 says
>>>> has no rotation. Or the scope line is about domain binding specifically,
>>>> the concern of that appendix, and B.1's mechanism is available to any type,
>>>> in which case saying so would help implementers.
>>>>
>>>> I do not think this changes either recommendation. It affects who can
>>>> act on them, so it seemed worth asking before the BCP points operators at
>>>> the mechanism.
>>>>
>>>> Shivdeep Singh
>>>>
>>>> On Thu, Aug 20, 2026 at 2:53 PM Vaibhav Bajpai <
>>>> [e-mail]> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> This paper just got published, and we believe would be
>>>>> valuable input to the BCP draft:
>>>>>
>>>>> >From robots.txt to ai.txt: Mapping the Evolution of Web Permissions
>>>>> in the Age of AI
>>>>> https://dl.acm.org/doi/pdf/10.1145/3831956.3831960
>>>>>
>>>>> The measurements suggest that being identifiable and respecting
>>>>> robots.txt is
>>>>> necessary, but no longer sufficient. The harder emerging problem is
>>>>> ensuring
>>>>> that authenticated crawlers interpret a fragmented set of publisher
>>>>> signals
>>>>> consistently, predictably, and according to the intended type of AI
>>>>> use.
>>>>>
>>>>> Our measurements show that today’s system combines heavy dependence on
>>>>> robots.txt,
>>>>> fragmented AI-specific policies, inconsistent naming, and only
>>>>> partially
>>>>> coherent configurations.
>>>>>
>>>>>
>>>>> To this end, here is some input for the BCP:
>>>>>
>>>>> 1.) Define how crawlers should handle conflicting signals and
>>>>> precedence:
>>>>> Explicitly establish that discovery/descriptor files do not grant
>>>>> access and
>>>>> recommend conservative handling of contradictory permission signals.
>>>>>
>>>>> 2.) Separate crawler identity by purpose. Search, training, and
>>>>> user-triggered/assistant retrieval should be distinguishable where they
>>>>> represent materially different uses. Measurements show that that
>>>>> publishers
>>>>> are expressing use-specific preferences: e.g., search=yes, ai-train=no.
>>>>>
>>>>> 3.) Strengthen identity requirements and encourage a canonical crawler
>>>>> registry: Documentation alone isn’t solving crawler identity, as is
>>>>> apparent
>>>>> from the very large number of unrecognized User-Agent strings in our
>>>>> measurements. Plus, when an operator runs crawlers for materially
>>>>> different
>>>>> purposes, those crawlers should have distinguishable identities, e.g.:
>>>>>
>>>>>   ExampleSearchBot
>>>>>   ExampleTrainingBot
>>>>>   ExampleAssistantFetcher
>>>>>
>>>>>   rather than one generic ExampleBot.
>>>>>
>>>>> 4.) Explicitly prohibit identity evasion/policy shopping: A crawler
>>>>> shouldn’t
>>>>> spoof identities, rotate identifiers to evade rules, or choose
>>>>> whichever
>>>>> applicable policy gives it the most permissive result.
>>>>>
>>>>> 5.) Encourage policy revalidation rather than indefinitely caching
>>>>> permission
>>>>> decisions: Crawlers should periodically revalidate cached
>>>>> crawler-policy
>>>>> resources and must not assume that a previously observed permission
>>>>> remains
>>>>> valid indefinitely.
>>>>>
>>>>> 6.) Require transparent interpretation and validation. Crawler
>>>>> operators should
>>>>> document supported policy mechanisms, conflict behavior and caching,
>>>>> and
>>>>> ideally provide a URL testing tool showing publishers exactly how their
>>>>> crawler interprets the site’s policies, e.g., “This is how our crawler
>>>>> interprets your site.”
>>>>>
>>>>> Kind regards, Vaibhav
>>>>>
>>>>> > On 27. Jul 2026, at 14:42, Sauron <sauron=
>>>>> [e-mail]> wrote:
>>>>> >
>>>>> > Hi all,
>>>>> > Thanks for the engagement and support during the working group
>>>>> session at IETF126.
>>>>> > As mentioned during the meeting, we are looking for even more
>>>>> feedback on the Crawler Best Practices draft:
>>>>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/ /
>>>>> https://github.com/garyillyes/cbcp Specifically, input on the scope,
>>>>> rate control conventions, and handling of non-malicious crawlers would be
>>>>> really useful to help move the draft forward. Please send any comments to
>>>>> the list or open issues directly on GitHub.
>>>>> > Thanks,
>>>>> > Gary
>>>>> > PS: if you have ideas where to send the following two drafts that
>>>>> CBCP depends on, that would be hugely welcome
>>>>> > 1. JAFAR:
>>>>> https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
>>>>> > 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
>>>>> > _______________________________________________
>>>>> > Web-bot-auth mailing list -- [e-mail]
>>>>> > To unsubscribe send an email to [e-mail]
>>>>>
>>>>> _______________________________________________
>>>>> Web-bot-auth mailing list -- [e-mail]
>>>>> To unsubscribe send an email to [e-mail]
>>>>>
>>>> _______________________________________________
>>>> Web-bot-auth mailing list -- [e-mail]
>>>> To unsubscribe send an email to [e-mail]
>>>>
>>>> _______________________________________________
>>> Web-bot-auth mailing list -- [e-mail]
>>> To unsubscribe send an email to [e-mail]
>>>
>>