[SCITT] Re: A page for independent runs of the Vaara conformance vectors

[Joel Hillier <[e-mail]>]

Hi Emek, Henri, all,

The row first, because I said I'd file one and a promise isn't a row.

@conarium-ai/core@0.2.41
tarball sha256  3e4f34e679acb84539001a38e084321cc6e45867b9552be85438197987d93314
Node v22.22.2, Linux, extracted into an empty directory
13 of 13 agree with the published manifest.json exit codes

You ran it on Node 24, I ran it on Node 22 on a different operating system, and every case landed on its documented code. No install, no arguments beyond the file, exactly as you said.

Kind: reproduction of the author-supplied checkers and vectors. Your binary over your cases. It establishes that the tarball runs and is exit-stable on a second Node major and a second OS, and it establishes nothing about your wire format, because I didn't implement anything from it. Naming that is cheap now and impossible after somebody cites the row.

Then I probed the thing you disclosed. You named three routes to exit 20. The probe reaches two more, and the second is worse than an exit code.

exit=20  no file argument         "missing <file|dir>"                     named by you
exit=20  unknown flag             "unknown flag: --not-a-real-flag"        named by you
exit=20  schema-invalid receipt   007-schema-invalid                       the documented meaning
exit=20  path not found           "path not found: does-not-exist.jsonl"   not named
exit=20  valid JSON, not JSONL    "invalid JSON in <file>:1"               not named

That last row is the one I'd fix first, because the message is false and not merely the code. The parser is line-oriented, so a pretty-printed JSON file gets its first line, an opening brace on its own, handed to JSON.parse as though it were a whole document. Isolated by construction rather than inferred from the symptom:

pretty.json    {
                 "a": 1
               }
  -> exit 20, invalid JSON in pretty.json:1: Expected property name or '}' at position 1

oneline.json   {"a":1}
  -> parses, treated as a receipt

pretty.json is valid JSON. Python's json.load takes it without complaint. The tool tells a user their file is invalid JSON when it is valid JSON, and the real fact is that it isn't JSONL. Two different things, and the one it reports is the one that isn't true.

Where a user meets it is directory mode. Pointed at test-vectors/ it reaches for expected-hashes.json, one of your own metadata files, and reports it as malformed. Pointed at a directory of receipts it's perfectly happy, so the intended path is fine and the adjacent one isn't.

Exit 13 names two events as well, which I think is worse, because 13 is a claim about cryptography.

exit=13  VALID file,   no --pubkey    "no --pubkey given (refusing to skip signature checks)"
exit=13  missing file, no --pubkey    same message, and the file is never opened
exit=20  missing file, WITH --pubkey  "path not found"

The first line is the control that separates the two explanations: 13 for a missing pubkey fires on a perfectly good receipt file, so it isn't an artefact of the missing path. Documented as "signature invalid or missing while a pubkey was supplied", and returned when no pubkey was supplied. The refusal itself is right and I'd keep it, and the message says plainly what happened. It's the code that's wrong, and wrong in the direction that costs, because a caller scripting on exit codes reads 13 as the cryptography having failed and it means a flag is missing.

The second line adds the ordering. The pubkey check fires before the path is resolved, so on an invocation wrong in two ways at once, 13 wins over 20 and the run answers with a signature verdict about a file it never opened. Which of the two a caller sees is a fact about check order rather than about the run, and that's your own short-circuiting finding arriving inside your own CLI.

Two clean results, since a clean result is a result and the probe was built to find failures.

--anchor-check on a chain carrying no anchor returns exit 15, not 0: 0/1 anchored, nothing was compared. That's the rule you described to this list working exactly as described. The probe was written expecting a 0 there and would have reported one.

The empty chain returns 0 with a warning saying in terms that this is not a verification that nothing was deleted. The warning carries what the exit code can't, which is the honest arrangement once a code is published and can't move.

Your drift guard's blind spot is the fourth arrival of one shape this week. It compares the set of codes in the binaries against the set in the table, both directions, and 20 is in both, so it's green. A code naming two events is invisible to a check over the set of codes.

The same shape landed on my side today in a different vocabulary, and you found that one: ABSENT covered a digest that could not be checked and an instruction a recipient cannot follow, and a checker over the set of verdicts couldn't see it. Same defect, same week, two tools, neither author looking for it. A check over the members of a set cannot see a member that means two things. That belongs in the shared text beside the weakest-rung rule, because it's the rule that says why the weakest-rung rule has to be enforced somewhere other than in the enumeration.

________________________________

Now the defect you found in what I shipped, which is the better half of this message.

You're right that runners/verify_manifest.py isn't in the archive while REPRODUCE.md still tells a reader to run it, and right that a reader following the file in order hits it at exactly the step whose stated purpose is catching stale digests in that file.

The part worth reporting is where the check stood at the time. check_reproduce.py did report it, among ten ABSENT invocations, and the packaging step passed it through, because ABSENT was a single verdict covering two unrelated conditions. A pinned digest over a file a subset doesn't contain genuinely could not be checked, which is the case the three-way verdict was built for. An instruction to a recipient naming a file that isn't there is a different condition entirely and belongs to the package, not to the digests. One label, two conditions, and the run gave a reader no way to tell them apart. Which is the finding above, one layer down, in my tree rather than yours.

So: a fourth verdict rather than a note. SUBSET.txt lists every path deliberately excluded from a package with the reason a recipient is owed. An invocation naming a file that isn't present is EXCLUDED if it's declared there and MISSING if it isn't, and MISSING exits non-zero. Silence is no longer an available answer. It carries a mutant: remove verify_manifest.py from the declarations and the run must fail naming it. That mutant is your finding, restored as a test. Six mutants on that runner now, six killed.

Your network correction stands and the wording is repaired. "No network at run time" is wrong. The existence-oracle runner stands up an HTTP server on 127.0.0.1 and connects to it over loopback, because a timing discriminator without a real round trip measures nothing. No connection leaves the machine, which is the narrower and correct claim, and REPRODUCE.md now makes it in both places.

What I'd keep is your method rather than the correction. You read the imports and the call sites before running any of it rather than take the sentence on the word of the person who sent you the archive. That's the only thing that makes such a sentence worth anything.

One thing in your message that doesn't hold. You wrote "Six is what your message listed." It listed eight, checked against the sent bytes rather than from memory:

run_deterministic_encoding_vectors.py
run_existence_oracle_vectors.py --timing-samples 60
ecdsa_malleability_probe.py
merkle_equiv.py
check_self_claims.py
mutate_self_claims.py
check_reproduce.py
(cd runners && mutate_reproduce.py)

Your self-correction is right in substance and you found the two you'd missed without being told, which is the part that counts. The count you attributed it to is the one thing that isn't, and the shape of it is worth a line: a correction about taking a count instead of deriving it took a count instead of deriving it. That's the failure mode being unusually well-behaved as a specimen.

Your kind field, applied back at my own run too. You're right that your ARP run is the first kind, my runners over my vectors, establishing that the archive runs somewhere other than my machine and nothing about the ARP text. What's worth pointing at is that you filed vaaraio/vaara#612 with the kind already named, because Iman's scoping was in front of you. That's the field being used before it exists in any format, which is the strongest evidence it's the right field. Henri, two rows now map cleanly to the first kind with no amendment from either of them.

On your freeze, and the part you don't have. Refusing to write the interpretation notes now is the right call. A note written after reading my concession is a reading that has already met mine, and hashing it would produce an artefact whose whole claimed value is a property it doesn't have. The absence, stated, is worth more than the document. That's the weakest-rung rule applied to a piece of evidence you'd have benefited from producing, which is the hard direction to apply it in. Writing the notes first and hashing them before opening a class nobody has seen is the version that works, and that's the shape I'd take next round rather than the one I proposed.

And on clause three, since both implementations failed it today. Yours carried a sentence removed from your specification hours earlier for being false. Mine said three named fields were in a document where they appeared zero times. Your release gate caught yours in the document and nothing caught it in the mail. My self-claim checker caught mine in the document and nothing caught it in the mail. Same clause, same day, two implementations, no mechanism on either side.

There's one now and it's in the archive, so borrow it or improve it rather than build a second. check_correspondence.py over a claim table, which is your design: one row per claim made to somebody, rows added by hand because deciding which clause is a claim is a judgement a runner would make badly. Nineteen rows, zero failing, five mutants, five killed, and mutant 1 is the clocks defect restored.

Building it produced three results worth more than the runner.

One. It survived its own mutant on the first attempt. PRESENT is satisfied by any mention, so renaming a mechanism's defining occurrence while leaving a cross-reference standing passed. DEFINED was added, and the mutant now kills.

Two. Three rows were added for claims in this message and two failed immediately. They were claims about the tooling being evaluated against the specification, because a row had no way to name the artefact it was about. A claim about one artefact verified against another is not a check. The table now carries a target per row.

Three, and this one only surfaced because the first target assignment was also wrong: REPRODUCE.md documented neither SUBSET.txt nor the MISSING verdict. The mechanism existed and the file a reader is told to follow never mentioned it. Now it does.

The standing gap, stated because it's what will bite next: the table holds the claims somebody wrote down. A claim made and never tabled is invisible, and the runner says so on every run rather than letting a pass imply coverage it doesn't have.

________________________________

New archive, carrying your seventh point. All earlier ones are superseded.

package digest   123710c156a1e0995542c2b97d00a2370719fdeeda189e95a816766f440da6cd
archive          fc6db0d463875ef1fd2a0e12896362a8c81db973b083095719a0cd2a46fdd6b0
members          27
root             the package root, the directory containing
                 draft-hillier-scitt-arp.md and conformance/

The root is the first line of MANIFEST.txt and inside the digest, so it travels with the bytes and can't be restated afterwards without moving the value. The working for why it was needed is on the other thread.

self-claims     11 checks, 0 failing     11 mutants, 11 killed
reproduce       43 pins, 0 stale         6 mutants, 6 killed
correspondence  19 claims, 0 failing     5 mutants, 5 killed
section 3       10 runnable, 10 declared excluded, 0 MISSING

Extracted into an empty directory and all ten commands run from there before sending, including the two-minute one, with both pinned run records reproducing byte-identically from that extraction. Stated at that precision deliberately: the previous message said "run from there" on a package where four of the ten had been, so the claim now names the number.

Emek, the Windows note is useful. The runners are clean on a console without forcing UTF-8 because nothing in them prints outside ASCII, which was untested until you named it and is now a rule.

Joel