[SCITT] Re: A page for independent runs of the Vaara conformance vectors

[[e-mail]]

Joel, all,

A correction to my message this afternoon, before anyone runs it.

I wrote that our locale guard "said 324 lines across 22 files". That figure
does not reproduce. Checked out at v0.2.42 in a clean worktree, that release's
own pack_locale.mjs prints 310 across 20.

The cause is worth more than the correction, and it is not specific to us.

  git worktree add --detach /tmp/v0242 v0.2.42
  cd /tmp/v0242 && node test/pack_locale.mjs
  -> 310 Turkish word occurrence(s) across 20 packed doc file(s)

  cp <two gitignored files under docs/audit/> docs/audit/
  node test/pack_locale.mjs
  -> 324 ... across 22 ...

Two files under docs/audit/ are gitignored, present on the machine the figure
was taken on, and in no commit. .gitignore does not bind npm: the files
allow-list in package.json is evaluated against the working directory, so
anything sitting under an allowed path is packed whether or not any commit
records it. The measurement was of a disk, not of the package.

It reached a changelog and then this list, which is the part I would have
wanted told to me.

We had met this before and drawn the boundary too narrowly. npm publish from a
machine that had run the demo once shipped examples/_keys/audit-ed25519.pem, a
private key. The repair pinned that path -- a check that refuses if that one
file appears. The class was left open, so it came back as a number instead of a
key, and a number is the version that gets published and quoted.

0.2.46 adds test/pack_tracked.mjs -- in the repository, since test/ is not
packed, so extracting the tarball will not show it to you. Every path npm would
ship is tracked by git or under one declared generated prefix, dist/, which the
build reproduces from source. Anything else fails the run and is named. It
fails closed -- if git cannot answer, the check does not pass, because a check
that could not run must not print the same word as one that ran and found
nothing.

It caught something on its first real use, which is the only recommendation I
can give it: the claims review for the release it ships in was itself untracked
when I ran the suite, and the guard refused. That file is packed.

Worth ten minutes on your own side if you cut releases from a working
directory. The question is not whether your ignore file is right. It is whether
anything you ship exists only where you built it, and whether any number you
have published was measured over that.

Two smaller repairs in the same release, both declared here yesterday and both
the same shape. exit_contract.mjs said "27 invocation(s)" and ran 25 processes;
27 was the assertion counter. Both figures now come from the run and both are
printed. And pack_locale.mjs carried "1041" in a comment where the changelog
said "1034" -- two hand-written declarations of one number inside the check
whose subject is hand-written declarations drifting. The comment states no
figure now; the live one is pinned in a file a run compares.

For the record, since the numbers in that paragraph were also wrong in kind:
on the 0.2.42 packed set the letter pass is 971 across 26, the word pass 310
across 20, and the union 1034 across 29. 1034 was the union, never the letter
pass, and I called it the letter pass twice.

Emek

On Sat, Aug 22, 2026 at 5:08 PM <[e-mail]> wrote:

Joel,

The row first, since you filed one and the answer to a row is a run.

@conarium-ai/core@0.2.45
tarball sha256 d642a9bf4b959325df60fc7b04ff723b93a33e353091bb8bb2c809628cb49d4e
npm pack, extracted into an empty directory, run from there
Node v24.16.0, Windows, no install
your five routes:

  no file argument            2   was 20
  unknown flag                2   was 20
  007-schema-invalid         20   the documented meaning, unmoved
  path not found              2   was 20
  valid JSON, not JSONL       2   was 20, and the message was false

Thirteen of thirteen still agree with manifest.json. Rows one, two and four
moved in 0.2.44; row five and the exit-13 ordering below are 0.2.45. Row
three never moved and should not have.

You were right about which half of the JSONL row matters. The message was
the defect and the code was downstream of it. The loader splits on newlines
unless the file opens with a bracket, so a pretty-printed object hands its
opening brace to JSON.parse alone, and the tool told a caller their valid
file was invalid JSON. It now says the file is JSON and not JSONL, on one
line, with the conversion command. A caller acts on the sentence, and the
sentence is what was wrong.

Worse than you reported, and I found it only because you pointed at
directory mode. Aimed at our own published test-vectors directory, the
verifier reached expected-hashes.json and stopped -- the artefacts we ship
for other people to run, refused by the tool we ship to run them. A
directory is now asked for the receipts in it. A neighbour that is valid
JSON in the wrong shape is skipped and named on stderr; skipped quietly
would be reporting success over something never examined, which is the
defect this thread keeps arriving at from every direction.

On exit 13, and I answered the wrong argument the first time.

I wrote it up as a decision and defended fail-closed. You had already said
the refusal is right and you would keep it. Your point was narrower and I
argued past it: manifest.json described 13 as "signature invalid or missing
while a pubkey was supplied", and 13 is what you get when no pubkey was
supplied, so the machine-readable description was false in the one case it
was returned for. Rejecting a finding is fair. Rejecting a finding I had
misread produces the same paragraph and settles nothing, and it is harder to
notice from the inside for exactly that reason.

The ordering, which is the half I deferred on a reason that does not hold. I
wrote that reordering a fail-closed check belongs in a release about the
fail-closed posture. It does not. Fail-closed means never reporting success
without verifying signatures; resolving the target first exits 2 on a
missing path, which is not success, and a target that is present still
requires a key. I had classified a harmless reorder as a safety change,
which is how work gets deferred for a reason that reads well and is not
true. Target before key now, in verify and in coverage:

  valid file, no --pubkey       13   unchanged, fail-closed
  missing file, no --pubkey      2   was 13
  missing file, with --pubkey    2

So which answer a caller gets is a fact about their invocation again rather
than about our check order.

The one I would keep, and it is yours.

A check over the members of a set cannot see a member that means two things.
test/exit_code_descriptions.mjs reads the three places a code is described
-- the tables in RECEIPT-SPEC.md, the exitCodes map in the published
manifest.json, and the generator that writes it -- and requires the two
machine-readable copies to be identical and each manifest description to be
the leading clause of the specification's row. The manifest may be shorter
than the row. It may not say something else.

It failed on two live defects the first time it ran: your 13, and code 2
described differently by the manifest and the specification inside the same
release, which I had found that morning and had no mechanism for. The set
comparison was green through both, as it had been green while 20 named two
events.

Its limit, in the same breath, because a check that is described only by
what it catches is being sold rather than reported: it establishes that the
three texts agree with each other. It does not establish that any of them is
true of the binary. exit_contract.mjs asks that question of the codes, over
twenty-five invocations of four binaries, and cannot ask it of the
sentences. Nothing in the tree closes that gap and I would rather write it
down than let a green run imply otherwise.

Twenty-five, and the check itself says twenty-seven. I found that writing
this paragraph. The success line prints the pass counter, which is
twenty-three cases plus four assertions over two of them, and calls the
total invocations. It makes twenty-five processes. A number that names
something it is not, printed by the file I added to stop numbers naming
things they are not, in the release that was about exactly that. It is
0.2.46 along with the second one below.

Three things I would rather tell you than have you find.

One. A property test decided where the JSONL split fell, against my first
answer. I had sent every parse failure to 2, on the reasoning that no
receipt had been read, which is literally true.
src/invariants.property.test.ts mutates one byte of a valid signed chain and
asserts the result is a verdict code; it failed. It was right. A corrupted
receipts file is the artefact examined and rejected, and telling an operator
the command could not be run sends them to check their arguments instead of
their evidence. Valid JSON in the wrong shape is 2. Anything else is 20, in
a directory as well as out of
one, because skipping a corrupt neighbour would make reporting evidence
depend on whether the caller named a file or a folder. The test was written
long before the change and caught it on the first run, which is the only
kind of evidence I have that the split is in the right place.

Two. exit 1 is now published in four tables and observed in none. It is the
uncaught-exception handler, which used to be 20 -- a schema verdict for a
crash. No CLI input reaches it; I tried four candidates against the verifier
and every one was caught by the code that owns it, which is the right result
for a last-resort handler and also the reason the row is unmeasured. The
evidence for that line is that it reads correctly. That is reading
substituted for running, which is the substitution that hid the original
defect from me, and the release note says so rather than letting 67 green
checks imply otherwise. The alternative is a deliberate crash path in a
security tool so a test can observe it, and I think that trade is worse than
the gap. If you disagree I would take the argument.

Three, and it touches your row directly. Your row is 0.2.41, and 0.2.43
changed what the package contains, so anyone re-running it on a current
version is running against a differently composed tarball. That is worth a
line in the register rather than a surprise for whoever files next.

Four directories of internal design notes are no longer shipped:
docs/superpowers, docs/plans, docs/specs, docs/audit, fourteen files.
Extracted, the published tarball goes from 508 files at 0.2.42 to 496 at
0.2.43, the difference being twelve rather than fourteen because that
release also added two. They were dated Turkish planning documents, one of
which carries an instruction addressed to an agent rather than to a reader
in its opening paragraph, and they were going to anyone running npm i.

The guard that was supposed to keep that visible had been reporting a third
of it. It counted with the pass written for Turkish that carries no Turkish
letter,
which is the wrong instrument for Turkish prose: it said 324 lines across
22 files, and counting letters as well over the same packed set gives 1034
across 29. The remainder is 34 across 14 now, and pinned, so the next drift
is a red run rather than a figure in a note.

One number in that repair is already inconsistent in the shipped tree, and
I would rather say it than have you diff it. The guard's own comment says
the letter pass finds 1041 where the changelog says 1034. One of them is an
earlier measurement over a slightly different file filter, and they are two
hand-written declarations of one figure sitting inside the check whose
whole subject is hand-written declarations drifting apart. 0.2.46, with the
counter above. Same shape as the set comparison: a number nobody compares
against anything.

Emek

On Sat, Aug 22, 2026 at 1:56 AM Joel Hillier <jhillier=[e-mail]> wrote:

Hi Emek, Henri, all,

The row first, because I said I'd file one and a promise isn't a row.

@conarium-ai/core@0.2.41
tarball sha256  3e4f34e679acb84539001a38e084321cc6e45867b9552be85438197987d93314
Node v22.22.2, Linux, extracted into an empty directory
13 of 13 agree with the published manifest.json exit codes

You ran it on Node 24, I ran it on Node 22 on a different operating system, and every case landed on its documented code. No install, no arguments beyond the file, exactly as you said.

Kind: reproduction of the author-supplied checkers and vectors. Your binary over your cases. It establishes that the tarball runs and is exit-stable on a second Node major and a second OS, and it establishes nothing about your wire format, because Ididn't implement anything from it. Naming that is cheap now and impossible after somebody cites the row.

Then I probed the thing you disclosed. You named three routes to exit 20. The probe reaches two more, and the second is worse than an exit code.

exit=20  no file argument         "missing <file|dir>"                     named by you
exit=20  unknown flag             "unknown flag: --not-a-real-flag"        named by you
exit=20  schema-invalid receipt   007-schema-invalid                       the documented meaning
exit=20  path not found           "path not found: does-not-exist.jsonl"   not named
exit=20  valid JSON, not JSONL    "invalid JSON in <file>:1"               not named

That last row is the one I'd fix first, because the message is false and not merely the code. The parser is line-oriented, so a pretty-printed JSON file gets its first line, an opening brace on its own, handed to JSON.parse as though it were a whole document. Isolated by construction rather than inferred from the symptom:

pretty.json    {
                 "a": 1
               }
  -> exit 20, invalid JSON in pretty.json:1: Expected property name or '}' at position 1

oneline.json   {"a":1}
  -> parses, treated as a receipt

pretty.json is valid JSON. Python's json.load takes it without complaint. The tool tells a user their file is invalid JSON when it is valid JSON, and the real fact is that it isn't JSONL. Two different things, and the one it reportsis the one that isn't true.

Where a user meets it is directory mode. Pointed at test-vectors/ it reaches for expected-hashes.json, one of your own metadata files, and reports it as malformed. Pointed at a directory of receipts it's perfectly happy, so the intended path is fine and the adjacent one isn't.

Exit 13 names two events as well, which I think is worse, because 13 is a claim about cryptography.

exit=13  VALID file,   no --pubkey    "no --pubkey given (refusing to skip signature checks)"
exit=13  missing file, no --pubkey    same message, and the file is never opened
exit=20  missing file, WITH --pubkey  "path not found"

The first line is the control that separates the two explanations: 13 for a missing pubkey fires on a perfectly good receipt file, so it isn't an artefact of the missing path. Documented as "signature invalid or missing while a pubkey was supplied", and returnedwhen no pubkey was supplied. The refusal itself is right and I'd keep it, and the message says plainly what happened. It's the code that's wrong, and wrong in the direction that costs, because a caller scripting on exit codes reads 13 as the cryptography havingfailed and it means a flag is missing.

The second line adds the ordering. The pubkey check fires before the path is resolved, so on an invocation wrong in two ways at once, 13 wins over 20 and the run answers with a signature verdict about a file it never opened. Which of the two a caller sees isa fact about check order rather than about the run, and that's your own short-circuiting finding arriving inside your own CLI.

Two clean results, since a clean result is a result and the probe was built to find failures.

--anchor-check on a chain carrying no anchor returns exit 15, not 0: 0/1 anchored, nothing was compared. That's the rule you described to this list working exactly as described. The probe was written expecting a 0 there and would have reported one.

The empty chain returns 0 with a warning saying in terms that this is not a verification that nothing was deleted. The warning carries what the exit code can't, which is the honest arrangement once a code is published and can't move.

Your drift guard's blind spot is the fourth arrival of one shape this week. It compares the set of codes in the binaries against the set in the table, both directions, and 20 is in both, so it's green. A code naming two events is invisible to a checkover the set of codes.

The same shape landed on my side today in a different vocabulary, and you found that one: ABSENT covered a digest that could not be checked and an instruction a recipient cannot follow, and a checker over the set of verdicts couldn't see it. Same defect, sameweek, two tools, neither author looking for it. A check over the members of a set cannot see a member that means two things. That belongs in the shared text beside the weakest-rung rule, because it's the rule that says why the weakest-rung rule hasto be enforced somewhere other than in the enumeration.

---

Now the defect you found in what I shipped, which is the better half of this message.

You're right that runners/verify_manifest.py isn't in the archive while REPRODUCE.md still tells a reader to run it, and right that a reader following the file in order hits it at exactly the step whose stated purpose is catching stale digestsin that file.

The part worth reporting is where the check stood at the time. check_reproduce.py did report it, among ten ABSENT invocations, and the packaging step passed it through, because ABSENT was a single verdict covering two unrelated conditions.A pinned digest over a file a subset doesn't contain genuinely could not be checked, which is the case the three-way verdict was built for. An instruction to a recipient naming a file that isn't there is a different condition entirely and belongs to the package,not to the digests. One label, two conditions, and the run gave a reader no way to tell them apart. Which is the finding above, one layer down, in my tree rather than yours.

So: a fourth verdict rather than a note. SUBSET.txt lists every path deliberately excluded from a package with the reason a recipient is owed. An invocation naming a file that isn't present is EXCLUDED if it's declared there and MISSING if it isn't, and MISSING exits non-zero. Silence is no longer an available answer. It carries a mutant: remove verify_manifest.py from the declarations and the run must fail naming it. That mutant is your finding, restored as a test. Six mutants on that runner now, six killed.

Your network correction stands and the wording is repaired. "No network at run time" is wrong. The existence-oracle runner stands up an HTTP server on 127.0.0.1 and connects to it over loopback, because a timing discriminator without a real round tripmeasures nothing. No connection leaves the machine, which is the narrower and correct claim, and REPRODUCE.md now makes it in both places.

What I'd keep is your method rather than the correction. You read the imports and the call sites before running any of it rather than take the sentence on the word of the person who sent you the archive. That's the only thing that makes such a sentence worthanything.

One thing in your message that doesn't hold. You wrote "Six is what your message listed." It listed eight, checked against the sent bytes rather than from memory:

run_deterministic_encoding_vectors.py
run_existence_oracle_vectors.py --timing-samples 60
ecdsa_malleability_probe.py
merkle_equiv.py
check_self_claims.py
mutate_self_claims.py
check_reproduce.py
(cd runners && mutate_reproduce.py)

Your self-correction is right in substance and you found the two you'd missed without being told, which is the part that counts. The count you attributed it to is the one thing that isn't, and the shape of it is worth a line: a correction about taking a countinstead of deriving it took a count instead of deriving it. That's the failure mode being unusually well-behaved as a specimen.

Your kind field, applied back at my own run too. You're right that your ARP run is the first kind, my runners over my vectors, establishing that the archive runs somewhere other than my machine and nothing about the ARP text. What's worth pointing atis that you filed vaaraio/vaara#612 with the kind already named, because Iman's scoping was in front of you. That's the field being used before it exists in any format, which is the strongest evidence it's the right field. Henri, two rows nowmap cleanly to the first kind with no amendment from either of them.

On your freeze, and the part you don't have. Refusing to write the interpretation notes now is the right call. A note written after reading my concession is a reading that has already met mine, and hashing it would produce an artefact whose whole claimedvalue is a property it doesn't have. The absence, stated, is worth more than the document. That's the weakest-rung rule applied to a piece of evidence you'd have benefited from producing, which is the hard direction to apply it in. Writing the notes firstand hashing them before opening a class nobody has seen is the version that works, and that's the shape I'd take next round rather than the one I proposed.

And on clause three, since both implementations failed it today. Yours carried a sentence removed from your specification hours earlier for being false. Mine said three named fields were in a document where they appeared zero times. Your release gatecaught yours in the document and nothing caught it in the mail. My self-claim checker caught mine in the document and nothing caught it in the mail. Same clause, same day, two implementations, no mechanism on either side.

There's one now and it's in the archive, so borrow it or improve it rather than build a second. check_correspondence.py over a claim table, which is your design: one row per claim made to somebody, rows added by hand because deciding which clause is a claim is a judgement a runner would make badly. Nineteen rows, zero failing, five mutants,five killed, and mutant 1 is the clocks defect restored.

Building it produced three results worth more than the runner.

One. It survived its own mutant on the first attempt. PRESENT is satisfied by any mention, so renaming a mechanism's defining occurrence while leaving a cross-reference standing passed. DEFINED was added, and the mutant now kills.

Two. Three rows were added for claims in this message and two failed immediately. They were claims about the tooling being evaluated against the specification, because a row had no way to name the artefact it was about. A claim about one artefact verified against another is not a check. The table now carries a target per row.

Three, and this one only surfaced because the first target assignment was also wrong: REPRODUCE.md documented neither SUBSET.txt nor the MISSING verdict. The mechanism existed and the file a reader is told to follow never mentioned it. Now it does.

The standing gap, stated because it's what will bite next: the table holds the claims somebody wrote down. A claim made and never tabled is invisible, and the runner says so on every run rather than letting a pass imply coverage it doesn't have.

---

New archive, carrying your seventh point. All earlier ones are superseded.

package digest   123710c156a1e0995542c2b97d00a2370719fdeeda189e95a816766f440da6cd
archive          fc6db0d463875ef1fd2a0e12896362a8c81db973b083095719a0cd2a46fdd6b0
members          27
root             the package root, the directory containing
                 draft-hillier-scitt-arp.md and conformance/

The root is the first line of MANIFEST.txt and inside the digest, so it travels with the bytes and can't be restated afterwards without moving the value. The working for why it was needed is on the other thread.

self-claims     11 checks, 0 failing     11 mutants, 11 killed
reproduce       43 pins, 0 stale         6 mutants, 6 killed
correspondence  19 claims, 0 failing     5 mutants, 5 killed
section 3       10 runnable, 10 declared excluded, 0 MISSING

Extracted into an empty directory and all ten commands run from there before sending, including the two-minute one, with both pinned run records reproducing byte-identically from that extraction. Stated at that precision deliberately: the previous message said"run from there" on a package where four of the ten had been, so the claim now names the number.

Emek, the Windows note is useful. The runners are clean on a console without forcing UTF-8 because nothing in them prints outside ASCII, which was untested until you named it and is now a rule.

Joel