Date: Tue, 28 Jul 2026 12:35:46 -0400
From: Nick Mathews <nick@lifelightlabs.com>
To: Sauron <sauron=40google.com@dmarc.ietf.org>, web-bot-auth@ietf.org
Message-ID: <b6abbcf2-76f6-4ace-add0-e9651512f5a7@Spark>
In-Reply-To: 
 <CADTQi=dSGTGJtczLTZ1ua77C8h4pvGhrVkz3MZV7SO280MJrKQ@mail.gmail.com>
References: 
 <CADTQi=dSGTGJtczLTZ1ua77C8h4pvGhrVkz3MZV7SO280MJrKQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="6a68da68_7545e146_53b"
Message-ID-Hash: VXAICRE5NS26JFOI3LTGEFXO75RXFAUJ
CC: "Mirja Kuehlewind (IETF)" <ietf@kuehlewind.net>
Precedence: list
Subject: =?utf-8?q?=5BWeb-bot-auth=5D_Re=3A_Follow-up_from_IETF_126=3A_Comments_on_dr?=
 =?utf-8?q?aft-illyes-webbotauth-cbcp?=
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/web-bot-auth/aNVOpQ2LcBAD3YPCLMNp1TS2vSE>

--6a68da68_7545e146_53b
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi Gary,

Comments on CBCP from the site side, in the three areas you asked about. =
I run AVA Pay, an open source merchant-side verifier for automated commer=
ce traffic (Web Bot Auth, Visa TAP, AP2). We sit where these practices ge=
t enforced, so these observations come from operating the receiving end r=
ather than from running a crawler.
1. Scope: user-initiated agents will be governed by this document whether=
 or not they are in it.
Section 1 defines a crawler as retrieving resources =22without direct hum=
an initiation of individual requests.=22 That cleanly excludes the fastes=
t-growing class of automated traffic we see: an agent dispatched by one p=
erson, for one task, at one site, thirty seconds ago. The exclusion is co=
rrect as a definition and unhelpful as a deployment reality, because site=
 operators cannot tell the two apart at request time. What actually happe=
ns is that crawler policy gets applied to a shopping agent carrying a cus=
tomer's intent, and a sale is refused.
Suggestion: say explicitly whether user-initiated agents are in scope, an=
d if not, name the venue that covers them. A sentence acknowledging that =
sites cannot distinguish the classes without additional signals would als=
o motivate the rest of the WG's work nicely.
2. Rate control: back-out logic keyed on 5xx misses the signal that rate =
limiters actually emit.
Section 2.3 requires back-out logic relying on =22at least the standard s=
ignals defined by Section 15.6 of =5BHTTP-SEMANTICS=5D,=22 which is the 5=
xx server error range. In practice the response to excessive automated tr=
affic is usually 429 with Retry-After, not a 5xx, and often not even that=
: the more common failure is silent degradation, where a site's protectio=
n layer starts serving partial or challenge content while still returning=
 200.
We hit exactly this last week. Repeated automated fetches of one domain t=
ripped that site's rate limiter; the visible symptom was degraded content=
 well before any hard failure. A crawler keying only on 5xx would have ke=
pt going.
Suggestions: reference 429 and Retry-After explicitly as normative back-o=
ff signals; add guidance that a sudden change in response shape (challeng=
e pages, truncated content, redirect to interstitials) should be treated =
as a back-off signal rather than as content; and consider whether a machi=
ne readable way for a site to state its own limit is in scope, since toda=
y a well behaved crawler is left to guess and a badly behaved one pays no=
 price.
3. Identification: the document requires identity that cannot be verified=
, in the working group that is building verifiable identity.
Sections 2.2 and 2.5 rest on user-agent strings and published IP ranges. =
Both are self-asserted from the site's perspective. Any client can claim =
to be any crawler in a User-Agent header, and IP-range publication is ope=
rationally fragile in the presence of CDNs, cloud egress, and IPv6 churn.=
 This is the concrete pain merchants describe: roughly half of commerce t=
raffic is now automated, most sites have no policy beyond monitoring, and=
 the identification they are offered is a string anyone can type.
Suggestion: keep 2.2 and 2.5 as baseline hygiene, but state plainly that =
user-agent and IP-range identification MUST NOT be treated as authenticat=
ion, and point normatively to the signature-based work in this working gr=
oup as the preferred mechanism where available. Same for the central regi=
stry idea in Section 1: a registry of names without proof-of-key reproduc=
es the problem it is meant to solve. Anchoring it in the key-directory wo=
rk being drafted here seems strictly better than a parallel list.
Happy to contribute text for any of this, or to open the issues on GitHub=
 if you prefer them there.

Nick Mathews
AVA Pay, Agentic Verification Architecture LLC
https://github.com/AVA-PAY/ava-pay
On Jul 27, 2026 at 8:42=E2=80=AFAM -0400, Sauron <sauron=3D40google.com=40=
dmarc.ietf.org>, wrote:
> Hi all,
> Thanks for the engagement and support during the working group session =
at IET=46126.
> As mentioned during the meeting, we are looking for even more feedback =
on the Crawler Best Practices draft: https://datatracker.ietf.org/doc/dra=
ft-illyes-webbotauth-cbcp/=C2=A0/=C2=A0https://github.com/garyillyes/cbcp=

> Specifically, input on the scope, rate control conventions, and handlin=
g of non-malicious crawlers would be really useful to help move the draft=
 forward. Please send any comments to the list or open issues directly on=
 GitHub.
> Thanks,
> Gary
> PS: if you have ideas where to send the following two drafts that CBCP =
depends on, that would be hugely welcome
> 1. JA=46AR:=C2=A0https://datatracker.ietf.org/doc/draft-illyes-webbotau=
th-jafar/
> 2. REP-ext https://datatracker.ietf.org/doc/draft-illyes-repext/
> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
> Web-bot-auth mailing list -- web-bot-auth=40ietf.org
> To unsubscribe send an email to web-bot-auth-leave=40ietf.org

--6a68da68_7545e146_53b
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

<html xmlns=3D=22http://www.w3.org/1999/xhtml=22>
<head>
<title></title>
</head>
<body>
<div name=3D=22messageBodySection=22>
<div>Hi Gary,</div>
<div>&=23160;</div>
<div>Comments on CBCP from the site side, in the three areas you asked ab=
out. I run AVA Pay, an open source merchant-side verifier for automated c=
ommerce traffic (Web Bot Auth, Visa TAP, AP2). We sit where these practic=
es get enforced, so these observations come from operating the receiving =
end rather than from running a crawler.</div>
<div><strong>1. Scope: user-initiated agents will be governed by this doc=
ument whether or not they are in it.</strong></div>
<div>Section 1 defines a crawler as retrieving resources =22without direc=
t human initiation of individual requests.=22 That cleanly excludes the f=
astest-growing class of automated traffic we see: an agent dispatched by =
one person, for one task, at one site, thirty seconds ago. The exclusion =
is correct as a definition and unhelpful as a deployment reality, because=
 site operators cannot tell the two apart at request time. What actually =
happens is that crawler policy gets applied to a shopping agent carrying =
a customer's intent, and a sale is refused.</div>
<div>Suggestion: say explicitly whether user-initiated agents are in scop=
e, and if not, name the venue that covers them. A sentence acknowledging =
that sites cannot distinguish the classes without additional signals woul=
d also motivate the rest of the WG's work nicely.</div>
<div><strong>2. Rate control: back-out logic keyed on 5xx misses the sign=
al that rate limiters actually emit.</strong></div>
<div>Section 2.3 requires back-out logic relying on =22at least the stand=
ard signals defined by Section 15.6 of =5BHTTP-SEMANTICS=5D,=22 which is =
the 5xx server error range. In practice the response to excessive automat=
ed traffic is usually 429 with Retry-After, not a 5xx, and often not even=
 that: the more common failure is silent degradation, where a site's prot=
ection layer starts serving partial or challenge content while still retu=
rning 200.</div>
<div>We hit exactly this last week. Repeated automated fetches of one dom=
ain tripped that site's rate limiter; the visible symptom was degraded co=
ntent well before any hard failure. A crawler keying only on 5xx would ha=
ve kept going.</div>
<div>Suggestions: reference 429 and Retry-After explicitly as normative b=
ack-off signals; add guidance that a sudden change in response <i>shape</=
i> (challenge pages, truncated content, redirect to interstitials) should=
 be treated as a back-off signal rather than as content; and consider whe=
ther a machine readable way for a site to state its own limit is in scope=
, since today a well behaved crawler is left to guess and a badly behaved=
 one pays no price.</div>
<div><strong>3. Identification: the document requires identity that canno=
t be verified, in the working group that is building verifiable identity.=
</strong></div>
<div>Sections 2.2 and 2.5 rest on user-agent strings and published IP ran=
ges. Both are self-asserted from the site's perspective. Any client can c=
laim to be any crawler in a User-Agent header, and IP-range publication i=
s operationally fragile in the presence of CDNs, cloud egress, and IPv6 c=
hurn. This is the concrete pain merchants describe: roughly half of comme=
rce traffic is now automated, most sites have no policy beyond monitoring=
, and the identification they are offered is a string anyone can type.</d=
iv>
<div>Suggestion: keep 2.2 and 2.5 as baseline hygiene, but state plainly =
that user-agent and IP-range identification MUST NOT be treated as authen=
tication, and point normatively to the signature-based work in this worki=
ng group as the preferred mechanism where available. Same for the central=
 registry idea in Section 1: a registry of names without proof-of-key rep=
roduces the problem it is meant to solve. Anchoring it in the key-directo=
ry work being drafted here seems strictly better than a parallel list.</d=
iv>
<div>Happy to contribute text for any of this, or to open the issues on G=
itHub if you prefer them there.</div>
<div>&=23160;</div>
<div>Nick Mathews&=23160;</div>
<div>AVA Pay, Agentic Verification Architecture LLC &=23160;</div>
<div><a href=3D=22https://github.com/AVA-PAY/ava-pay=22>https://github.co=
m/AVA-PAY/ava-pay</a></div>
</div>
<div name=3D=22messageReplySection=22>On Jul 27, 2026 at 8:42=E2=80=AFAM =
-0400, Sauron &lt;sauron=3D40google.com=40dmarc.ietf.org&gt;, wrote:<br /=
>
<blockquote type=3D=22cite=22>
<div dir=3D=22ltr=22>
<p>Hi all,</p>
<p>Thanks for the engagement and support during the working group session=
 at IET=46126.</p>
<p>As mentioned during the meeting, we are looking for even more feedback=
 on the Crawler Best Practices draft: <span class=3D=22gmail-no-md=22><sp=
an class=3D=22gmail-ng-star-inserted=22><a target=3D=22=5Fblank=22 rel=3D=
=22noopener=22 href=3D=22https://datatracker.ietf.org/doc/draft-illyes-we=
bbotauth-cbcp/=22 class=3D=22gmail-ng-star-inserted=22>https://datatracke=
r.ietf.org/doc/draft-illyes-webbotauth-cbcp/</a>&=23160;/&=23160;<a href=3D=
=22https://github.com/garyillyes/cbcp=22>https://github.com/garyillyes/cb=
cp</a>&=23160;</span></span></p>
<p>Specifically, input on the scope, rate control conventions, and handli=
ng of non-malicious crawlers would be really useful to help move the draf=
t forward. Please send any comments to the list or open issues directly o=
n GitHub.</p>
<p>Thanks,<br />
<span style=3D=22background-color:transparent=22>Gary</span></p>
<p>PS: if you have ideas where to send the following two drafts that CBCP=
 depends on, that would be hugely welcome<br />
1. JA=46AR:&=23160;<a href=3D=22https://datatracker.ietf.org/doc/draft-il=
lyes-webbotauth-jafar/=22>https://datatracker.ietf.org/doc/draft-illyes-w=
ebbotauth-jafar/</a></p>
<p>2. REP-ext <a href=3D=22https://datatracker.ietf.org/doc/draft-illyes-=
repext/=22>https://datatracker.ietf.org/doc/draft-illyes-repext/</a></p>
</div>
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F<br />
Web-bot-auth mailing list -- web-bot-auth=40ietf.org<br />
To unsubscribe send an email to web-bot-auth-leave=40ietf.org<br /></bloc=
kquote>
</div>
</body>
</html>

--6a68da68_7545e146_53b--