[Web-bot-auth] Re: Follow-up from IETF 126: Comments on draft-illyes-webbotauth-cbcp

[Nick Mathews <[e-mail]>]

Hi Gary,

Comments on CBCP from the site side, in the three areas you asked about. I run AVA Pay, an open source merchant-side verifier for automated commerce traffic (Web Bot Auth, Visa TAP, AP2). We sit where these practices get enforced, so these observations come from operating the receiving end rather than from running a crawler.
1. Scope: user-initiated agents will be governed by this document whether or not they are in it.
Section 1 defines a crawler as retrieving resources "without direct human initiation of individual requests." That cleanly excludes the fastest-growing class of automated traffic we see: an agent dispatched by one person, for one task, at one site, thirty seconds ago. The exclusion is correct as a definition and unhelpful as a deployment reality, because site operators cannot tell the two apart at request time. What actually happens is that crawler policy gets applied to a shopping agent carrying a customer's intent, and a sale is refused.
Suggestion: say explicitly whether user-initiated agents are in scope, and if not, name the venue that covers them. A sentence acknowledging that sites cannot distinguish the classes without additional signals would also motivate the rest of the WG's work nicely.
2. Rate control: back-out logic keyed on 5xx misses the signal that rate limiters actually emit.
Section 2.3 requires back-out logic relying on "at least the standard signals defined by Section 15.6 of [HTTP-SEMANTICS]," which is the 5xx server error range. In practice the response to excessive automated traffic is usually 429 with Retry-After, not a 5xx, and often not even that: the more common failure is silent degradation, where a site's protection layer starts serving partial or challenge content while still returning 200.
We hit exactly this last week. Repeated automated fetches of one domain tripped that site's rate limiter; the visible symptom was degraded content well before any hard failure. A crawler keying only on 5xx would have kept going.
Suggestions: reference 429 and Retry-After explicitly as normative back-off signals; add guidance that a sudden change in response shape (challenge pages, truncated content, redirect to interstitials) should be treated as a back-off signal rather than as content; and consider whether a machine readable way for a site to state its own limit is in scope, since today a well behaved crawler is left to guess and a badly behaved one pays no price.
3. Identification: the document requires identity that cannot be verified, in the working group that is building verifiable identity.
Sections 2.2 and 2.5 rest on user-agent strings and published IP ranges. Both are self-asserted from the site's perspective. Any client can claim to be any crawler in a User-Agent header, and IP-range publication is operationally fragile in the presence of CDNs, cloud egress, and IPv6 churn. This is the concrete pain merchants describe: roughly half of commerce traffic is now automated, most sites have no policy beyond monitoring, and the identification they are offered is a string anyone can type.
Suggestion: keep 2.2 and 2.5 as baseline hygiene, but state plainly that user-agent and IP-range identification MUST NOT be treated as authentication, and point normatively to the signature-based work in this working group as the preferred mechanism where available. Same for the central registry idea in Section 1: a registry of names without proof-of-key reproduces the problem it is meant to solve. Anchoring it in the key-directory work being drafted here seems strictly better than a parallel list.
Happy to contribute text for any of this, or to open the issues on GitHub if you prefer them there.

Nick Mathews
AVA Pay, Agentic Verification Architecture LLC
https://github.com/AVA-PAY/ava-pay
On Jul 27, 2026 at 8:42 AM -0400, Sauron <sauron=[e-mail]>, wrote:
> Hi all,
> Thanks for the engagement and support during the working group session at IETF126.
> As mentioned during the meeting, we are looking for even more feedback on the Crawler Best Practices draft: https://datatracker.ietf.org/doc/draft-illyes-webbotauth-cbcp/ / https://github.com/garyillyes/cbcp
> Specifically, input on the scope, rate control conventions, and handling of non-malicious crawlers would be really useful to help move the draft forward. Please send any comments to the list or open issues directly on GitHub.
> Thanks,
> Gary
> PS: if you have ideas where to send the following two drafts that CBCP depends on, that would be hugely welcome
> 1. JAFAR: https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
> 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
> _______________________________________________
> Web-bot-auth mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]