[Web-bot-auth] Re: Field data on Web Bot Auth from 40,003 origins, two vantages, and a measurement of #93's premise

[Nick Mathews <[e-mail]>]

Wenchong,

This is the most useful thing anyone has put in front of this working group
in a while, and the care shows: pairing the arms, re-running from a second
vantage, reporting the 38 to 18 cell that goes the wrong way, and declining
to settle it because settling it would mean hammering 40,003 real sites.
Two contributions, and then your three questions.

First, the observation I can speak to directly. You write that across
294,898 signed requests no origin fetched your published key, and that the
one production verifier you saw answered "401 unknown public key or unknown
verified bot ID for keyid" without fetching. I run the other kind. Our
verifier is merchant-side: it parses Signature-Agent, resolves the
directory, computes the RFC 7638 thumbprint, and reaches a typed decision
including an explicit could-not-check state when the directory is
unreachable. We moved it to -01 the weekend the revision landed. The honest
limit on that claim is that it is live in production but with almost no
merchant traffic yet, so it is a report about what a resolving verifier
does, not evidence that resolving verifiers are common. On your data, they
are not.

Second, and this is the part I think is load-bearing for the group: the
absence you measured has a cost that shows up on the other side of it. When
we did fetch the two production directories we know of, on 9 August, we
found three divergences from the -01 text. One sends a padded base64 nonce
where the grammar says base64url. One serves a bare JWK where 5.5.1
resolves to a JWK Set. Bare-string Signature-Agent is still common in the
wild, which E.1.3 already anticipates. Thibault has filed those as issues
108, 109 and 118. My point is not the divergences themselves. It is that
they survived in production because nobody fetches. An interop error that
no verifier ever exercises is invisible, and it stays invisible right up
until the first verifier that actually resolves arrives, which is exactly
the wrong moment to discover it. Your finding and ours are the same finding
seen from the two ends.

On your first question, whether the un-enrolled signing cost belongs in the
draft: I would say yes, as a deployment consideration, and I would keep it
descriptive. A protocol document cannot promise what bot-management
products do with unfamiliar headers, but it can warn an implementer that
sending them today may cost admission, and that the two mechanisms you
separate carefully, headers scoring badly versus a failed allowlist lookup
scoring worse than silence, have opposite trajectories. An implementer
deciding whether to ship signing this quarter needs that sentence more than
they need almost anything else in the document.

On your second, the FCrDNS sentence for issue 93: yes, and I would put it
in the fallback text itself rather than in security considerations, because
the person who needs it is reading the MUST. Your breakdown is the argument
in one line: of ten identities, two have a PTR that names the operator, both
Google's, three have none at all, and four forward-confirm to the hosting
provider. An ec2 name that round-trips perfectly and identifies nobody is
the case that will bite, because it looks like a pass. As a verifier
implementer I will say the stronger version plainly: if 93 lands as a MUST
without that distinction, it requires my verifier to run a check that
returns a false pass for four of the ten identities it will most often see,
and I would rather return an explicit could-not-identify than a
confidently wrong one.

On your third, the enrolment path: the thing I would want said is who may
enrol and at what granularity, and your position is the argument for it. An
enrolment gate that runs through an organisational account selects for
companies, not for correctness, and reproduces the scalability problem the
draft attributes to IP allowlisting with keys instead of addresses. From
the merchant side the same gate has a second cost that is easy to miss: an
allowlist can only express decisions about signers someone has already
heard of, so a merchant using one has no way to treat a correct but unknown
signer differently from a malformed one. That is the seat I work in, and it
is why we resolve rather than allowlist.

Separately, the Appendix B vectors are up as PR 121 on the draft
repository, in case they are useful to anyone else implementing against the
directory response.

Happy to compare notes on the directory-fetch path if it is useful for your
write-up. Our verifier is open source and the deployment divergences above
are documented with the captures they came from.

Nick Mathews
AVA Pay, Agentic Verification Architecture
https://github.com/AVA-PAY/ava-pay
On Aug 11, 2026 at 1:38 PM -0400, 董文冲 <dwc24=[e-mail]>, wrote:
> Hi all,
>
> Thibault suggested I bring this to the list.
>
> I am a PhD student at Tsinghua. For an academic measurement study I implemented the signing profile by hand, published a directory at agent.hiem.al, and ran signed and unsigned arms against a 40,003-origin frame from Tranco. This is written against draft-meunier-webbotauth-httpsig-protocol-01 of 6 August, and I have read the review opened on 7 August, so I know some of it is live. Issue numbers below are on the draft's repository: #93 is the FCrDNS fallback, #111 is "can't validate without a key", #104 and #106 are the closed enrolment issues.
>
> What this is not. It is not a conformance report, and I do not think "the mechanism is not deployed yet" is worth anyone's time: the document is a draft, adoption takes years, and absence of deployment at this stage is what you would expect. Three things I can offer instead, none of which depends on treating the current text as fixed. What it costs a signer to follow the draft today. What one specified mode does in the field. And a measurement of the premise under #93, which is about FCrDNS and not about this draft at all.
>
> Method, briefly, because three details decide how much of this you can believe. The frame is 40,003 origins from Tranco, stratified by rank. The client is a plain HTTP client and not a browser: no JavaScript, one TLS fingerprint throughout, and the arms are byte-identical except for the headers under test, so an arm cannot differ from another by anything but its declaration. The addresses are datacenter ones in Los Angeles, London and Tokyo, which I expect is itself part of any bot score I am measuring. robots.txt is fetched and obeyed per identity. "Admitted" means the origin returned at least 100 words; "refused" means fewer than 20, or a 4xx or 5xx; between those I record undecided, and undecided enters no comparison.
>
> Signing, un-enrolled, costs access. The arms are paired, so the same origins carry all of them and the discordant cells are the evidence; marginal counts are context, not the test. Against an otherwise identical unsigned request, adding Signature, Signature-Input and Signature-Agent lost admission at 657 origins and gained it at 34, of 26,548 where both decided. An earlier pass over the same frame, on the instrument we used before 5 August, gave 408 against 38, so the sign is not one run's fluke. Nobody is checking the signature: three unfamiliar headers are moving a bot-management score.
>
> Since the private note, I have run the same arms from a second vantage, because a measurement from one address invites the reading that it measured the address. This is the part I would most want a list to push on. On 11 August I re-ran the arm set from a different host, in a different country, on a 6,000-origin subset of the same frame, holding everything else identical -- same arms, same repeats, same ceilings, same delays, same library versions. Compared on those same 6,000 origins:
>
> | | first vantage, 6 Aug | second vantage, 11 Aug |
> |---|---:|---:|
> | lost admission when signed | 96 | 117 |
> | gained | 2 | 8 |
> | of origins deciding both | 4,021 | 4,035 |
> | discordance | 2.44% [2.00, 2.96] | 3.10% [2.61, 3.68] |
>
> The intervals overlap and both are far past any conventional threshold. What makes this more than a repetition is the contrast between the two rows of the experiment. Between those two runs the individual arms disagree with themselves about 5.0-5.5% of origins -- vantage plus five days of ordinary churn -- while the signed-versus-unsigned difference stays put. On one address a day apart that same self-disagreement is 0.5%, so I know what the floor looks like. The level of admission is a property of who is asking and when; the cost of signing is not. That is what I would expect if the effect is real, and it is the check I would have demanded of someone else's single-vantage number.
>
> Everything else replicates on the second vantage too. Valid and corrupt signatures remain indistinguishable: 6 origins against 2, p = 0.29. The cell an origin that actually verifies would have to land in -- valid admitted while corrupt is refused -- holds 2 origins, against 6 in the incoherent mirror of it, so it is negative on net and indistinguishable from noise. And signing costs 10.4 times this run's own in-frame floor.
>
> I want to be careful about what that does and does not say, because I think it says less than it first looks like. It is a measurement of signing while un-enrolled, which is not a measurement of what Web Bot Auth does. An enrolled crawler is presumably recognised and I cannot observe one: none of the crawler identities I impersonate and measure signs anything at all and I cannot acquire a verified key, so there is no enrolled signer anywhere in my data. Nor can I separate the two mechanisms that would produce this result, and they point in opposite directions. If unfamiliar headers are simply scoring badly, the effect should fade as the headers become common and it says little about the design. If instead a signature that fails an allowlist lookup is treated worse than no signature at all, that is a property of the gate and it will not fade. My unsigned arm is the same unknown crawler, so what follows is attributable to the headers and not to being unknown, but which of those two readings applies is not something this experiment can decide.
>
> On #111, "can't validate without a key", I happen to have run exactly that mode. The spec's position is that Signature-Agent is RECOMMENDED but not required and that verification still works where the verifier already holds the key. My no-Signature-Agent arm is that mode: a valid signature over ("@authority"), keyid thumbprint as the only identifier. On 6 August, over the same 40,003 origins:
>
> | arm | admitted |
> |---|---:|
> | unsigned | 18,501 |
> | signed | 17,893 |
> | signed, corrupt | 17,904 |
> | signed, unlisted key | 17,882 |
> | signed, no Signature-Agent | 17,823 |
>
> It is not neutral and it is not merely equal. It is the worst of the five, and against signed-with-header the difference is significant: 69 origins admit the header form where no-agent is refused, against 29 the other way. I offer that as data for the thread and not as a view on how the issue should close.
>
> Two caveats, and the first is the one I would raise if I were reading this. The four signed arms sit between 17,823 and 17,904 admitted, a spread of 81 on a 40,003-origin frame, which is 0.20%. Paired, the discordant cells among them are small and all of one order: valid against corrupt is 30 to 31 on 5 August and 38 to 18 on 6 August; corrupt against unlisted-key is 29 to 35 and then 36 to 14; valid against unlisted-key is 25 to 31 and then 26 to 23. Two of those move between runs in a way I cannot separate from noise, and one of them, 38 to 18, reaches p = 0.011 in the direction of the corrupt signature doing better. With six pairwise comparisons in that run the threshold is 0.0083, so it does not survive, and I am not going to build anything on it.
>
> I am also not going to resolve it, and the reason is ethical rather than technical. Settling a difference this small would mean repeating the same six-arm pass over the same 40,003 origins several times in a short window, and that is a real imposition on real sites for a question that is secondary to everything else here. So I am reporting the numbers and stopping there.
>
> The second caveat is my own noise floor, and here the direction matters. I measure noise as corrupt against unlisted-key, both invalid and so one stimulus to an origin that does not verify: 0.19% on 6 August, 0.24% on 5 August. But on 6 August those two separate at 36 to 14, p = 0.003, which does survive the correction and which they should not do if both are simply invalid. One candidate is the keyid string itself. My unlisted-key arm sends keyid="unpublished", eleven characters, where a real keyid is 43 of base64url. If that is what is being scored then my floor is inflated and every multiplier I quote against it is an understatement. I would rather tell you that than have you find it.
>
> The part I would most like views on is #93. I want to be careful about its state: it is open, it has no resolution, and what it proposes is that implementers MUST fall back to existing verification techniques. I am not describing that as settled. What I can add is a measurement of its premise, because a fallback is only worth mandating if the thing being fallen back to is performed. From two hosts whose PTR fails Google's published procedure, of 1,777 paired origins that served a browser, 1,632 admitted the Googlebot claim at one host and 1,632 at the other, 91.84% at each; 1,631 of them, 91.78%, admitted it at both, and the hosts disagreed about two origins.
>
> This is not a criticism of the procedure or of Google. Google's own procedure works: 12 of 12 addresses I sampled forward-confirm and the PTR names Google. The difficulty is that it is Google's, and for most of the crawlers a fallback would have to cover there is nothing to fall back to. Of the ten identities I could sample, two have a PTR that names the operator, both of them Google's. Three have no PTR at all: GPTBot, OAI-SearchBot and ChatGPT-User, where a reverse lookup returns nothing to check. The remaining four forward-confirm and name the hosting provider instead of the operator, which is the case I would most want a sentence about: an ec2 hostname of the compute-1.amazonaws.com form round-trips perfectly, is reproducible by anyone who rents an instance in that range, and passes an FCrDNS check while identifying nobody. So a verifier that falls back to FCrDNS gets a usable answer for two identities, a false pass for four, and nothing at all for three.
>
> Put beside the percentage, that is what I would offer to #93 rather than a position on it. Origins largely do not run the check, and for most crawlers there is not much of a check to run. If the MUST lands, a sentence saying that a forward-confirmed PTR may name the hosting provider rather than the operator would stop an implementer reading a pass as an identification. Requiring attribution to a domain the operator controls is the fix email settled on.
>
> One observation that supports the above rather than standing on its own. Across 294,898 signed requests no origin fetched our published key, and no response in 3,447,202 carried Accept-Signature. I do not offer that as a finding. Nobody fetching an unknown signer's directory is rational behaviour, not a gap, and a two-year-old draft not being widely deployed is what anyone should expect. What it does show is which path is actually in use: a production verifier answered "401 unknown public key or unknown verified bot ID for keyid", parsing and rejecting on absence without fetching. That is a key allowlist, and an allowlist is an enrolment gate under another name. A positive control says these origins were deciding on identity throughout: one AI crawler identity against a browser identity is 2,592 discordant against 262.
>
> And the enrolment path, which I raise knowing #104 and #106 are closed. I am not asking for them to be reopened; what I have is a measurement of the deployed consequence rather than a comment on the text, and it was not available while they were open. Reputational continuity is the right thing to build on, but nothing takes a signer from correct to known except a per-verifier application, which is the scalability problem the draft says IP allowlisting has, reproduced with keys. One narrow version of it selects on the wrong axis: an enrolment gate running through an organisational account is easy for a company and awkward for a PhD student, who cannot enter their university into a programme agreement. A line about who may enrol, and at what granularity, would help the people least able to comply. For context, we applied to one such programme on 31 July and have had no decision; the endpoint still answers 401. That is not a complaint and I am not chasing it, because a decision that arrives on its own makes the before-and-after honest.
>
> Three questions for the list. Does the un-enrolled signing cost belong anywhere in the draft as a deployment consideration, or is it out of scope for a protocol document? For #93, would a sentence distinguishing a forward-confirmed hosting-provider name from an operator identification be welcome, and where? And is there anything about the enrolment path itself that the drafts should say and currently do not?
>
> Happy to send the full numbers, the exact signature base, the directory, or the read-out code. The measurement is not finished and nothing above is load-bearing on a result I would mind correcting in public.
>
> Best,
> Wenchong Dong
>
> _______________________________________________
> Web-bot-auth mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]