[SCITT] Re: A page for independent runs of the Vaara conformance vectors

[Henri Sirkkavaara <[e-mail]>]

Joel,

Yes to both directions, and the reciprocal half is the part I want.

Send me the ARP vectors and the runner invocation and I will run them and report what I get, in the same terms: what passed, what did not, and the reason where it did not. If my run disagrees with yours, that disagreement is the result, not a problem to resolve privately first.

You are right that the terms belong on the page rather than only on the list. I have put them there: a run that disagrees is a row, with the reason stated, and the form publishes without me in the loop. Nobody has to take my word for either.
The point about an author's own negative controls is the one I would like to quote back at people, because it applies to me first. My eight negative cases are fault injection into an implementation written from my reading of my own text. They demonstrate that the implementation matches my reading. They cannot demonstrate anything about the text. Only your run can do that.

Henri Sirkkavaara
Vaara - Runtime execution layer for AI agents
Built to see over the noise.
vaara.ioHelsinki, Finland

On Friday, August 21st, 2026 at 00:08, Joel Hillier <jhillier=[e-mail]> wrote:

> Hi Henri,
>
> I'll run them and file a row, and if it disagrees with yours then that's the row I file.
>
> The sentence that makes this worth more than a page is "a run that disagrees with mine is a row too, with the reason stated." A conformance register that can only record agreement is a marketing surface, and every reader knows it, so it carries no information either way. The disagreement row is the whole load-bearing part. And the form publishing without you in the loop is what makes it credible rather than promised, because a curator who can decline a row is a curator whose agreements mean nothing. Both of those are worth stating on the page itself as its terms, not just on the list where they'll scroll away.
>
> Why I think this is the scarcest instrument in the area right now. An author's own negative controls are fault injection into an implementation written from that author's reading of that author's text. It's the one configuration that structurally can't surface a specification defect, because the specification and the implementation share a mind, so a passing suite proves the two agree, which was never in doubt. Only a second implementer working from the text alone produces evidence about the text. ARP carries this as a named open item rather than a satisfied one, and the reason it's still open is precisely that there was nowhere for such a run to be recorded and no norm that it should be.
>
> So the page isn't a Vaara artefact. It's the missing half of everybody's conformance claim, and I'd like to reciprocate rather than only consume it. ARP's vectors and runners are public, they take no arguments, and I'll stand up the same kind of register on the same terms: disagreements published, no gate, no blacklist. A run of yours by me and a run of mine by you are worth more than either suite growing by a hundred cases.
>
> One suggestion for the row format, straight out of the type-table thread. A row records that a checker agreed. What a checker pins varies, and your own example this morning was a suite that pins the decoded field mapping and never the scope digest, because ingestion happens on the decoded map while identity is over the encoded bytes. Those are different agreements, and a row that says only "passed" merges them.
>
> I'd have the row carry the suite names and the commit, so a reader can tell whether an independent run confirmed identity reproducibility or ingestion compatibility. Two implementations can agree on every field mapping and still compute different digests over the same input, and a register that can't show the difference will eventually get cited for the stronger claim.
>
> Nobody owes you a run, as you say. This one's owed in the other direction: the check I posted yesterday asks other people to audit their own constructions, and the first person to build somewhere for the answers to be recorded has done the harder half.
>
> Joel