[Web-bot-auth] Field data on Web Bot Auth from 40,003 origins, two vantages, and a measurement of #93's premise

[董文冲 <[e-mail]>]

Hi all,

Thibault suggested I bring this to the list.

I am a PhD student at Tsinghua. For an academic measurement study I implemented the signing profile by hand, published a directory at agent.hiem.al, and ran signed and unsigned arms against a 40,003-origin frame from Tranco. This is written against draft-meunier-webbotauth-httpsig-protocol-01 of 6 August, and I have read the review opened on 7 August, so I know some of it is live. Issue numbers below are on the draft's repository: #93 is the FCrDNS fallback, #111 is "can't validate without a key", #104 and #106 are the closed enrolment issues.

What this is not. It is not a conformance report, and I do not think "the mechanism is not deployed yet" is worth anyone's time: the document is a draft, adoption takes years, and absence of deployment at this stage is what you would expect. Three things I can offer instead, none of which depends on treating the current text as fixed. What it costs a signer to follow the draft today. What one specified mode does in the field. And a measurement of the premise under #93, which is about FCrDNS and not about this draft at all.

Method, briefly, because three details decide how much of this you can believe. The frame is 40,003 origins from Tranco, stratified by rank. The client is a plain HTTP client and not a browser: no JavaScript, one TLS fingerprint throughout, and the arms are byte-identical except for the headers under test, so an arm cannot differ from another by anything but its declaration. The addresses are datacenter ones in Los Angeles, London and Tokyo, which I expect is itself part of any bot score I am measuring. robots.txt is fetched and obeyed per identity. "Admitted" means the origin returned at least 100 words; "refused" means fewer than 20, or a 4xx or 5xx; between those I record undecided, and undecided enters no comparison.

Signing, un-enrolled, costs access. The arms are paired, so the same origins carry all of them and the discordant cells are the evidence; marginal counts are context, not the test. Against an otherwise identical unsigned request, adding Signature, Signature-Input and Signature-Agent lost admission at 657 origins and gained it at 34, of 26,548 where both decided. An earlier pass over the same frame, on the instrument we used before 5 August, gave 408 against 38, so the sign is not one run's fluke. Nobody is checking the signature: three unfamiliar headers are moving a bot-management score.

Since the private note, I have run the same arms from a second vantage, because a measurement from one address invites the reading that it measured the address. This is the part I would most want a list to push on. On 11 August I re-ran the arm set from a different host, in a different country, on a 6,000-origin subset of the same frame, holding everything else identical -- same arms, same repeats, same ceilings, same delays, same library versions. Compared on those same 6,000 origins:

| | first vantage, 6 Aug | second vantage, 11 Aug |
|---|---:|---:|
| lost admission when signed | 96 | 117 |
| gained | 2 | 8 |
| of origins deciding both | 4,021 | 4,035 |
| discordance | 2.44% [2.00, 2.96] | 3.10% [2.61, 3.68] |

The intervals overlap and both are far past any conventional threshold. What makes this more than a repetition is the contrast between the two rows of the experiment. Between those two runs the individual arms disagree with themselves about 5.0-5.5% of origins -- vantage plus five days of ordinary churn -- while the signed-versus-unsigned difference stays put. On one address a day apart that same self-disagreement is 0.5%, so I know what the floor looks like. The level of admission is a property of who is asking and when; the cost of signing is not. That is what I would expect if the effect is real, and it is the check I would have demanded of someone else's single-vantage number.

Everything else replicates on the second vantage too. Valid and corrupt signatures remain indistinguishable: 6 origins against 2, p = 0.29. The cell an origin that actually verifies would have to land in -- valid admitted while corrupt is refused -- holds 2 origins, against 6 in the incoherent mirror of it, so it is negative on net and indistinguishable from noise. And signing costs 10.4 times this run's own in-frame floor.

I want to be careful about what that does and does not say, because I think it says less than it first looks like. It is a measurement of signing while un-enrolled, which is not a measurement of what Web Bot Auth does. An enrolled crawler is presumably recognised and I cannot observe one: none of the crawler identities I impersonate and measure signs anything at all and I cannot acquire a verified key, so there is no enrolled signer anywhere in my data. Nor can I separate the two mechanisms that would produce this result, and they point in opposite directions. If unfamiliar headers are simply scoring badly, the effect should fade as the headers become common and it says little about the design. If instead a signature that fails an allowlist lookup is treated worse than no signature at all, that is a property of the gate and it will not fade. My unsigned arm is the same unknown crawler, so what follows is attributable to the headers and not to being unknown, but which of those two readings applies is not something this experiment can decide.

On #111, "can't validate without a key", I happen to have run exactly that mode. The spec's position is that Signature-Agent is RECOMMENDED but not required and that verification still works where the verifier already holds the key. My no-Signature-Agent arm is that mode: a valid signature over ("@authority"), keyid thumbprint as the only identifier. On 6 August, over the same 40,003 origins:

| arm | admitted |
|---|---:|
| unsigned | 18,501 |
| signed | 17,893 |
| signed, corrupt | 17,904 |
| signed, unlisted key | 17,882 |
| signed, no Signature-Agent | 17,823 |

It is not neutral and it is not merely equal. It is the worst of the five, and against signed-with-header the difference is significant: 69 origins admit the header form where no-agent is refused, against 29 the other way. I offer that as data for the thread and not as a view on how the issue should close.

Two caveats, and the first is the one I would raise if I were reading this. The four signed arms sit between 17,823 and 17,904 admitted, a spread of 81 on a 40,003-origin frame, which is 0.20%. Paired, the discordant cells among them are small and all of one order: valid against corrupt is 30 to 31 on 5 August and 38 to 18 on 6 August; corrupt against unlisted-key is 29 to 35 and then 36 to 14; valid against unlisted-key is 25 to 31 and then 26 to 23. Two of those move between runs in a way I cannot separate from noise, and one of them, 38 to 18, reaches p = 0.011 in the direction of the corrupt signature doing better. With six pairwise comparisons in that run the threshold is 0.0083, so it does not survive, and I am not going to build anything on it.

I am also not going to resolve it, and the reason is ethical rather than technical. Settling a difference this small would mean repeating the same six-arm pass over the same 40,003 origins several times in a short window, and that is a real imposition on real sites for a question that is secondary to everything else here. So I am reporting the numbers and stopping there.

The second caveat is my own noise floor, and here the direction matters. I measure noise as corrupt against unlisted-key, both invalid and so one stimulus to an origin that does not verify: 0.19% on 6 August, 0.24% on 5 August. But on 6 August those two separate at 36 to 14, p = 0.003, which does survive the correction and which they should not do if both are simply invalid. One candidate is the keyid string itself. My unlisted-key arm sends keyid="unpublished", eleven characters, where a real keyid is 43 of base64url. If that is what is being scored then my floor is inflated and every multiplier I quote against it is an understatement. I would rather tell you that than have you find it.

The part I would most like views on is #93. I want to be careful about its state: it is open, it has no resolution, and what it proposes is that implementers MUST fall back to existing verification techniques. I am not describing that as settled. What I can add is a measurement of its premise, because a fallback is only worth mandating if the thing being fallen back to is performed. From two hosts whose PTR fails Google's published procedure, of 1,777 paired origins that served a browser, 1,632 admitted the Googlebot claim at one host and 1,632 at the other, 91.84% at each; 1,631 of them, 91.78%, admitted it at both, and the hosts disagreed about two origins.

This is not a criticism of the procedure or of Google. Google's own procedure works: 12 of 12 addresses I sampled forward-confirm and the PTR names Google. The difficulty is that it is Google's, and for most of the crawlers a fallback would have to cover there is nothing to fall back to. Of the ten identities I could sample, two have a PTR that names the operator, both of them Google's. Three have no PTR at all: GPTBot, OAI-SearchBot and ChatGPT-User, where a reverse lookup returns nothing to check. The remaining four forward-confirm and name the hosting provider instead of the operator, which is the case I would most want a sentence about: an ec2 hostname of the compute-1.amazonaws.com form round-trips perfectly, is reproducible by anyone who rents an instance in that range, and passes an FCrDNS check while identifying nobody. So a verifier that falls back to FCrDNS gets a usable answer for two identities, a false pass for four, and nothing at all for three.

Put beside the percentage, that is what I would offer to #93 rather than a position on it. Origins largely do not run the check, and for most crawlers there is not much of a check to run. If the MUST lands, a sentence saying that a forward-confirmed PTR may name the hosting provider rather than the operator would stop an implementer reading a pass as an identification. Requiring attribution to a domain the operator controls is the fix email settled on.

One observation that supports the above rather than standing on its own. Across 294,898 signed requests no origin fetched our published key, and no response in 3,447,202 carried Accept-Signature. I do not offer that as a finding. Nobody fetching an unknown signer's directory is rational behaviour, not a gap, and a two-year-old draft not being widely deployed is what anyone should expect. What it does show is which path is actually in use: a production verifier answered "401 unknown public key or unknown verified bot ID for keyid", parsing and rejecting on absence without fetching. That is a key allowlist, and an allowlist is an enrolment gate under another name. A positive control says these origins were deciding on identity throughout: one AI crawler identity against a browser identity is 2,592 discordant against 262.

And the enrolment path, which I raise knowing #104 and #106 are closed. I am not asking for them to be reopened; what I have is a measurement of the deployed consequence rather than a comment on the text, and it was not available while they were open. Reputational continuity is the right thing to build on, but nothing takes a signer from correct to known except a per-verifier application, which is the scalability problem the draft says IP allowlisting has, reproduced with keys. One narrow version of it selects on the wrong axis: an enrolment gate running through an organisational account is easy for a company and awkward for a PhD student, who cannot enter their university into a programme agreement. A line about who may enrol, and at what granularity, would help the people least able to comply. For context, we applied to one such programme on 31 July and have had no decision; the endpoint still answers 401. That is not a complaint and I am not chasing it, because a decision that arrives on its own makes the before-and-after honest.

Three questions for the list. Does the un-enrolled signing cost belong anywhere in the draft as a deployment consideration, or is it out of scope for a protocol document? For #93, would a sentence distinguishing a forward-confirmed hosting-provider name from an operator identification be welcome, and where? And is there anything about the enrolment path itself that the drafts should say and currently do not?

Happy to send the full numbers, the exact signature base, the directory, or the read-out code. The measurement is not finished and nothing above is load-bearing on a result I would mind correcting in public.

Best,
Wenchong Dong