[SCITT] Re: A page for independent runs of the Vaara conformance vectors

[[e-mail]]

Joel, Henri,

Two applications of the kind field, one of them backwards at myself.

The row I filed this afternoon (vaaraio/vaara#612) already names its kind, because

Iman's scoping was in front of me when I wrote it: a reproduction of the

author-supplied checkers and vectors at commit a209864e, not an independent

implementation of the receipt draft and not a validation of the specification

text. So the field is fillable and that is what a filled one looks like. Henri, if

you add it, my row maps to your first kind and needs no amendment from me.

Backwards at myself, twice.

First, the ARP run I reported earlier is the same kind and I did not say so. Your

runners over your vectors, on my machine — it establishes that the archive runs

and is byte-stable somewhere other than yours, and nothing about the ARP text,

because I did not implement anything from it. By your own rule, a report that does

not name its kind reads as the strongest one available, and mine did not name it.

Second, and this is the worse one: I reported six runners and said the bundle was

run. Six is what your message listed. REPRODUCE.md documents more than six that

need no arguments, and I took your count instead of deriving it — on the same day

you corrected yourself for exactly that, in a message I had read. I have since run

the two I missed:

    check_reproduce exit 0

    mutate_reproduce exit 0 5 mutants, 5 killed on their designed set

Both pass. The coverage claim in my earlier message was six of eight, stated as

though it were all of them.

And running them surfaced something in the archive. REPRODUCE.md step (k) is:

    # (k) check this file against the tree it describes. Two digests here

    # were stale before this existed, and nothing caught them.

    python3 runners/verify_manifest.py

runners/verify_manifest.py is not in the self-contained archive. So the check

written because two digests in that document went stale, with nothing to catch

them, is the one missing from the tree it shipped in. A reader following

REPRODUCE.md in order hits it at step (k) and finds nothing there — and the

failure mode is the one the step exists to prevent, since what they cannot run is

the thing that would have told them the document's own digests are current.

I do not know whether it is excluded deliberately as part of the subset or left

out by accident. Either way the invocation is still in the file a recipient is

told to follow.

The CAP-1 runs are the one place I would claim the second kind, and only because

the two readers were written from the -00 prose before the vectors were locatable

— which is what the digests in the other thread now pin, whatever they are worth

about the past.

One thing your taxonomy does that I would keep. It makes what a run establishes a

property of who wrote the verifier and what they read, rather than of the result.

Two green runs can be worth entirely different amounts, and until now nothing in

any of our registers could say which was which. A class digest fixes the bytes; it

was never going to fix that.

Emek

On Fri, Aug 21, 2026 at 6:30 PM Joel Hillier <jhillier=[e-mail]> wrote:

Hi Iman, Henri,

Iman, the last sentence of that is the most useful thing anyone has put on the runs page, and I don't think it should sit inside one message.

You wrote your own row down before anyone asked you to: a reproduction of the author-supplied checkers and vectors, not an independent implementation or validation of the specification text. Nobody made you add that. It's a passing result you had every incentiveto file as a plain 41 of 44, and you named the ground it stands on instead.

That's the weakest-rung rule arriving at a register entry. We have now watched it turn up in bounds, in populations, in verdicts, in exit codes, in a freshness floor, and yesterday inside a statistical estimator. Every one of those was a rule about anartefact. Yours is the first at the point where a result gets filed, which is the point where the loss actually happens, because a row outlives the message that explains it.

So the specific suggestion, and it's smaller than it sounds: the row carries what kind of run it was. Three kinds cover what this list has produced in a week, and they aren't degrees of the same thing:

A reproduction of the author's checkers over the author's vectors. Establishes that the artefact runs and is byte-stable somewhere other than the author's machine, which is worth having and is not nothing.

An independent implementation from the text, run against the author's vectors. Establishes something about the text, because a second reader had to decide what the sentences meant.

An independent implementation run against independently constructed vectors. Establishes something about both.

And the rule that makes it load-bearing rather than decorative: a row that doesn't name its kind reads as the first one. Henri, that's the same argument you already accepted for disagreement rows. A register that can only record agreement is a marketing surface.A register that can't distinguish a reproduction from an independent implementation will get cited for the second when it holds the first, and the person citing it won't be lying.

Applied to what I sent this list an hour ago, before anybody files anything. The five runners in that archive are my checkers over my vectors. If Henri runs them and gets my two pinned digests, that is a reproduction row and nothing stronger. It showsthe package works outside the tree it was built in, which is the thing the 11 August package failed, and it shows nothing whatever about whether ARP's text says what I think it says. I'd rather write that down now than have it inferred from a green row later.

And it's already load-bearing on the other draft. CAP-1 has picked up runs of three different kinds this week and the numbers don't distinguish them. Walter's is an independent implementation from the prose plus the published schema. Emek's two are independentimplementations from the prose alone, and one of them refused all fifteen documents on shape because the member names aren't in the draft at all. Yours is a reproduction. Those are three different pieces of evidence, and "three implementations agree" flattensall three into a sentence that was already the weakest claim in the document.

The class digest fixes which bytes were run. It records nothing about who wrote the verifier or what they read to write it, and that's the axis that decides what a run is worth.

Henri, if the row format takes one more field, I'd make it that one rather than anything I suggested earlier.

Best,

Joel

________________________________________
From: Iman Schrock <[e-mail]>
Sent: Friday, 21 August 2026 15:00:19
To: [e-mail]
Cc: [e-mail]; Joel Hillier
Subject: Re: [SCITT] Re: A page for independent runs of the Vaara conformance vectors

Henri,

Thank you for taking this seriously and fixing both the claim and the mechanism.

The corrected language is accurate. The retained-head checks and the
external witness close the issue I raised without overstating what the
hash chain alone provides.

You may file my run exactly as scoped:

41 passed, 0 failed, 3 skipped across 44 suites at commit
62a7080b7c854173c7d6b8ee51ce4dd724d59227. This is a reproduction of
the author-supplied checkers and vectors, not an independent
implementation or validation of the specification text.

Best,
Iman