[SCITT] Re: New draft: The Coverage Attestation Profile (draft-hillier-coverage-attestation-00), and a request to break it

[[e-mail]]

Joel,

I found the vectors through the datatracker's additional-resources link and ran

both text-derived verifiers against your fifteen. Neither was touched after the

vectors were located — that was the whole value of writing them first, and I did

not want to spend it. Walter's pinned digests reproduce here exactly once line

endings are normalised, so we are talking about the same bytes:

    cap-1/src/vectors/manifest.json 170aa81e…

    cap-1/src/CAP-1.schema.json 4453f216…

    (Certisyn-Inc/certisyn-drafts @ 0980d320)

The result first, then the one finding in it that I think is yours to act on.

                     verdicts strict attribution

    reader A 14/15 9/10

    reader B 10/15 0/10

    A vs B disagree on 15 of 15

Two readings of the same document, run against the author's own class, agree on

nothing. Earlier today I put the cost of the missing schema at eight

disagreements in twenty documents on our own cross-run. On yours it is total,

and it has a single root: the draft does not name the members.

**The schema makes Section 9's MUST unsatisfiable in two of its three digest

positions.**

This is the one I would not have found by reading, and it is the root of the

string-versus-object split I described earlier in this thread.

Section 9 says a producer "SHOULD use an algorithm that is unbroken at time of

production and MUST state which was used." The published schema carries a digest

in three places and in two different shapes:

    subject.digest object {algorithm, value} — states it

    catalogue_digest string ^[0-9a-f]{32,128}$ — cannot

    withheld_digest string ^[0-9a-f]{32,128}$ — cannot

The range 32–128 hex spans MD5 through SHA-512, so the algorithm is not

recoverable from the value. And both enclosing objects — `basis` and the

`unexamined` item — are `additionalProperties: false`, so a producer who wants to

comply has nowhere to put the statement. It is not a style inconsistency between

two shapes; under the published schema the normative requirement cannot be met at

two of the three sites where R3 and R4 rely on it.

The first shape is the one you got right. I would carry `{algorithm, value}` to

the other two rather than the reverse, because the string form is what the

weakest-rung argument is about: a digest whose algorithm is unstated is a claim

whose strength the reader cannot evaluate, and R3/R4 are exactly where that

matters.

**Two smaller ones, both the same absence.**

`subject` is an object with `kind` and `ref`. Neither word occurs in the

Internet-Draft. Reader B took the draft's "named" at face value, expected a

string, and refused all fifteen documents under R0 — which is why its strict

attribution is 0/10. It is not a careless reading; there was nothing in the text

to read otherwise, and a class where every document fails the shape check tells

its author nothing about the rules underneath.

`withheld_digest` is reader A's only lost verdict, on PV-04. It looked for

`digest` and for `hash` — the two names the prose suggests — and found neither.

One member name, and a conforming document reads as non-conforming.

**NC-05, on your side of the fence.**

Reader A refuses NC-05 under R1 *and* R5. That is the analytic point from this

morning arriving empirically, in your own negative control, without either of us

arranging it: a document violating R5's second sentence has already violated

R1, so the mutant survives R5's removal. I am not re-arguing it — only noting

that the vector you wrote shows it, which is better evidence than my reading of

it was.

**Our own version of this, since it would be cheap to point only outward.**

While the above was running, an implementer built a verifier against our

published receipt vectors and reached 13/13 only after opening our source. Four

things were nowhere in our documents: what the Ed25519 signature actually covers

(the UTF-8 bytes of the prefixed hash string, not the raw digest — they tried all

three), that the sequence counter advances by exactly one, which absent fields are

schema errors rather than tampering, and the order the checks run in. The last is

the one I had not considered a wire fact at all, and it is: run the signature

check first and a deleted receipt reports as a bad signature. True verdict, wrong

reason.

That is the same failure as `kind` and `ref` — a document whose author knows what

it means. Ours is written down now, and four parts of it are asserted against the

shipped verifier, so a change to the code that leaves one of those sentences

standing fails the build rather than a reader.

The rest of that section is prose, and the document says so. I am spelling that

out because the first version of it claimed every sentence was checked, and the

release gate caught that before it shipped: a section written against asserting a

property over a sample, asserting a property over a sample. It is corrected in

the published text rather than here.

Publish your schema and vectors in the draft's own tree and both verifiers here

can be pointed at them the same day, and at whatever you revise after.

Emek

On Fri, Aug 21, 2026 at 3:32 AM Walter Hawkins <[e-mail]> wrote:

Joel, Tiago, Emek,

The fourth implementation exists and has been run against the published
fifteen. Tiago, the bytes are findable: the datatracker page for the draft
lists the repository under additional resources — Certisyn-Inc/certisyn-drafts,
cap-1/src. You're right that the -00 text itself doesn't pin them, and your
point stands as written; but the run can be done today, so I did it, and I'll
pin what I ran so it meets your bar:

repo commit 0980d3201aa2caab3cbad5c6e9bc99b422370b43
manifest.json sha256 170aa81efc74c5278a2fb6e3bcc22bc91fb1706e9fb8faf1ce6d575e5ce3d965
CAP-1.schema sha256 4453f216089543780bfecc4295cc4a61462fdc585b88d1e35b7d1aba79716b4a
class digest sha256 d895d24b5a41ec27f17c0318098e1fb2fee1994f2d0aee9b55a7e57a239cab10
(sha256 over the sorted per-file sha256 list of the fifteen —
the enumeration-digest move from my first message, eating its
own cooking)

Provenance: one file, Node built-ins, no network, written from the -00 prose
plus CAP-1.schema.json, which section 4 declares normative for shape — so I
read the schema as spec and left verify.mjs, verify.py, verify.html, the
crosscheck harness and REPRODUCE.md unread. Emek, that means your missing-shape
finding is right where the -00 is concerned and the schema in the repo settles
your {alg, value}-versus-string divergence: catalogue_digest is a bare hex
string, pattern ^[0-9a-f]{32,128}$, no algorithm member anywhere in it. Your
second reader's reading was reasonable and is wrong per the published schema —
which is exactly your point, since neither of you could reach it from the
draft.

Results: verdicts fifteen for fifteen — five conform, ten refuse. Strict
attribution nine of ten, and the tenth is Emek's finding wearing empirical
clothes: NC-05 (eligible 9, examined 11, unexamined empty) refuses under R5
AND R1 in a text-derived verifier, because raising examined breaks R1's
equation at the same stroke. Emek got there analytically — the
examined-exceeds-eligible clause is entailed by R1, so no document violates it
alone — and the published R5 control confirms it: the "single mutation"
cannot exist as the rules are written. His second reader's examined:-1
construction shows clause one of R5 is independently testable; clause two is
decoration by the draft's own 7.2 standard. One of them has to move.

A related ambiguity the mutation protocol itself has: whether 7.2's "the class
MUST fail" incorporates 7.1's refused-by-the-rule-it-targets requirement.
Under attribution-aware re-running, my R5 mutant dies (NC-05 stops naming R5).
Under mere-refusal re-running it survives (R1 still refuses the document) —
which is the survival Emek measured. Same protocol, two readings, opposite
kill counts. Worth one sentence in 7.2.

Tiago, your withheld contradiction the class also answers empirically, in
both halves. PV-04 carries eligible 7, examined 4, unexamined 3 with the
withheld unit among the three — so the class counts withheld as UNexamined,
against section 6's "Examined, but the result is not disclosed." And the
withheld entry names its unit ("selector.ts"), against the privacy rationale
of accounting without naming. The repair you sketched — separate the
examined axis from the disclosed axis — is what the vectors already
practice on the first axis and refuse on the second. The text should follow
the class or the class the text; today they disagree.

Two findings I believe are new:

Section 7.4's production fixtures contain no cap/1 document. Both published
fixtures are certisyn.verdict-document/1.0.0 hosts whose coverage member is
the older shape — catalogued, executed, not_run_units. grep for "cap/1"
returns nothing in either file. So "correctly refused under R1" has no
referent in the artifacts it cites: no CAP-1 verifier can refuse the before
fixture under R1 (mine refuses both under shape) or conform the after. The
claims most likely to be false are the ones nothing was pointed at; these
fixtures were never pointed at a CAP-1 verifier.

And the nine-versus-eight numbering both of you flagged has a concrete form:
silencing R1 through R8 in my verifier kills the class each time, eight for
eight, your count. Silencing R0 leaves the class passing — no vector
exercises shape alone, so a verifier with no shape checking at all agrees on
all fifteen. One R0-only negative control closes it.

The verifier and the run record — implementation source, hashes above,
environment, per-vector output — are anyone's on request; the file imports
nothing of mine either. Emek, with the schema location above your two can run
the same fifteen today, and then the class has the independent runs it was
asking for instead of one.

Walter

On Thu, Aug 20, 2026 07:24 PM, [e-mail] wrote:

Joel,

Walter and Tiago have the three break-tests, and on the two we both reached they got further than I did, so I will not restate those. Two things are left that I can add, and one of them is a measurement rather than a reading.

Two implementations, and what they disagree about.

Tiago says he cannot do the independent run honestly because the conformance bytes are not identifiable from the -00 text. He is right, and I can put a number on what that costs.

Two verifiers were written from the -00 text and nothing else: one by me, and a second produced independently on my side by a reader who had not seen the first and did not have it. Neither was changed after the other was read. Standard library, no network, R0 through R8. Your own distinction applies to them exactly as it applies to your three: they are implementation independent, not author independent. What they are independent of is each other, and of you.

Each was run against the other's conformance class, twenty documents:

    cases: 20   agreements: 12   disagreements: 8

Ten of the twelve agreements are the second class, ten documents, including all nine of its negative controls, each refused by the rule it targets. The other two agreements are the shape control and the basis control in the first class. On that evidence the rules are implementable from the prose.

The eight disagreements are one disagreement, propagated:

    A/positive-base    A: conforming    B: refused under R4

Both readings implement "a catalogue basis MUST carry a catalogue digest". The first wrote the digest as a string, sha256:... . The second read Section 9, which does not mandate an algorithm but requires the producer to state which was used, and took that to call for an object, {alg, value}. Both forms state the algorithm and neither is a misreading. The JSON Schema named in Section 4 would have settled it in one line, and it is not in the Internet-Draft, so that member has no shape anywhere a reader can reach. The second verifier stops at its first refusal, so the single divergence then masks the intended rule in seven further documents, which is a smaller finding in its own right: "refused by the rule it targets rather than by some other rule" is only checkable against a verifier that reports every refusal. Against a short-circuiting one it is a fact about check order.

So the cost of the missing schema is not that an implementation is hard. It is that two careful readings of the same sentence produce different verdicts on a conforming document, and neither reader can tell which one you meant. Publish the schema and the vectors and there are two verifiers here that can be run against your fifteen the same day.

R5 has a clause no negative control can isolate.

    R5 control, every rule live   : ['R1', 'R5']

    R5 control, R5 silenced       : ['R1']

R1 requires eligible == examined + the number of individually accounted unexamined units, and that number cannot be negative, so any document where examined exceeds eligible already violates R1. Silencing R5 does not stop the class refusing that document; the mutant survives.

The second reading found the way through, and it is worth stating because it is the difference between the two clauses. Its R5 control sets examined to -1 while keeping R1's equality intact, and both verifiers refuse that under R5 alone. So R5's first sentence, counts are non-negative integers, is testable. Its second sentence, examined MUST NOT exceed eligible, has no document that violates it alone, and by your own standard in 7.2 a rule no control exercises is decoration. It is entailed by R1. Either it goes, or R1 weakens to a bound and R5 carries the equality; as the two stand, one of them is unreachable.

Adjacent and probably the same thread to pull: Section 5 numbers nine rules, R0 through R8. Section 7.2 says eight. Both readings applied nine and both flagged it independently. A reader cannot tell which rule is not mutation-tested, and the mutation claim is the load-bearing one for the whole conformance section.

One door Walter's construction leaves standing.

Walter moves the denominator's edge after the answers are known. R8 lets you leave the edge alone and move the claim instead. A stratum cited by an absence assertion must state which classes of claim it supports; it need not state truthfully, and nothing requires the cited claim to fall inside what it states. Five TLS configuration checks may declare supports: ["absence-of-any-vulnerability"] and bound "no vulnerability was observed in the release". Both verifiers call that conforming. R8 as written checks presence. The failure it names is containment.

On Tiago's withheld contradiction, one case beside it rather than inside it. His unit was examined and has a result that is not disclosed. Mine was examined and has no result: dispatched, ran, terminated normally, determined nothing. failed says it errored and it did not; unavailable says it was never dispatched and it was. examined counts units, not verdicts, so that unit is counted as examined and the document is conforming while a reader concludes a determination exists. I have an interest here and will say so rather than let it be noticed: indeterminate is the word my own draft uses for this, and I would rather be wrong about the gap than right about the borrowing.

Both verifiers, both classes and the cross-run are yours if you want them.

Emek

On Fri, Aug 21, 2026 at 3:05 AM Tiago Pinto <tiago=[e-mail]> wrote:

Joel,

I took the four break-tests in your message literally. I reviewed
`draft-hillier-coverage-attestation-00`, dated 20 August 2026, against
the exact bytes with SHA-256
`7a9eeb1fbdb1fee95697622546d2ae7efba762fff193d6ee34765233539ac353`.

I think the core construct survives, but I found four things worth
putting back to you, in the order of your tests.

1. `withheld` has no semantically consistent accounting under the
current model

The Terminology definition says a Disposition is the reason a unit was
not examined. Section 4.1 says `unexamined` contains one entry per unit
not examined. Section 6 then defines `withheld` as "Examined, but the
result is not disclosed in this document." R1 requires `eligible` to
equal `examined` plus the number of individually accounted unexamined
units.

For a withheld unit, I do not see a count assignment that preserves all
four statements. If the unit is included in `examined`, as the
definition of `withheld` requires, and also appears in `unexamined`, R1
counts it twice. If it is excluded from `examined`, the arithmetic can
reconcile but the `examined` count is false by the definition of
`withheld`.

There is a second contradiction in the same mechanism: R2 says every
unexamined entry MUST name a unit, while the Privacy Considerations say
`withheld` exists so that a sensitive unit can be accounted for without
being named in the document.

The obvious defence would be that `unexamined` really means something
like "not examined with a disclosed result". I do not think that saves
the current text; it is the repair. It requires changing the Terminology
definition, the meaning or name of the array, and the R1 accounting
semantics. My instinct would be to separate two axes: whether the unit
was examined, and whether the unit/result is disclosed.

2. I can produce a legitimate non-examination state that fits none of
the closed dispositions

Suppose a unit applies to the subject, is supported, is authorised, has
all dependencies/licences/network available, and was intended to run,
but the overall examination is aborted before that unit is dispatched.
The abort could be operator cancellation, fail-fast after an earlier
condition, or a global threshold.

I cannot place that unit cleanly in any of the eight dispositions. It is
not `not_applicable`, `disabled_by_policy`, `unsupported_input`,
`resource_exhausted`, `failed`, `unavailable`, `out_of_scope`, or
`withheld` under their current definitions.

Section 4.3 makes this sharper: `integrity` says whether every
dispatched unit reached a recorded outcome. In a clean operator
cancellation, every unit that was dispatched may have completed, so
`complete = true` remains consistent with that definition even though
part of the eligible population was never reached. R7 does not repair
this, because it only forces `complete = false` for `failed`,
`resource_exhausted`, or `unavailable`, and none accurately describes
the undispatched unit.

Reducing `eligible` after the abort to the units actually reached would
be worse: that is precisely denominator selection after the fact, the
class of hidden-bound problem the draft is trying to prevent.

I think this needs either a distinct state such as `not_reached` /
`aborted_before_dispatch`, or an explicit widening of an existing
disposition plus corresponding integrity semantics.

3. The denominator basis distinction breaks on a derived population

Consider a denominator constructed by taking external catalogue C at
digest D, enumerating properties of the subject, and applying a
deterministic applicability predicate to C using that enumeration. The
resulting eligible population is a subset E.

`catalogue` does not fully describe the basis because E was not simply
taken from C. `enumeration` does not fully describe it because the
candidate universe came from C. `declared` is plainly weaker than what
happened.

The same issue appears with catalogue unions, catalogue subtraction by
authorisation scope, and other deterministic filters. These seem like
ordinary derived denominators rather than edge cases.

A composable basis, or a `derived` basis that identifies its source
bases and derivation method, would preserve the information that
choosing any one of the current three loses.

4. I cannot yet do the independent implementation/run honestly because
the conformance bytes are not identifiable from the -00 text alone

The draft says the normative shape is given by an accompanying JSON
Schema, that there are fifteen vectors, that three implementations agree
on them, and that production fixtures are published. In the -00 text I
have, I cannot find a stable URI plus digest identifying the exact
schema, vector set, implementations, and fixtures to which those
statements refer.

I would not call that an R4 violation; R4 governs CAP-1 documents, not
the Internet-Draft. But it reproduces the same epistemic shape that R4
is designed to avoid: a set is asserted and described while the exact
revision of the set is not recoverable from the claim itself.

That also blocks the contribution you say would be most useful. I can
build a fourth implementation from the prose rather than porting yours,
but I cannot honestly claim an independent run against "the fifteen
vectors" until I can identify exactly which fifteen bytes/objects
constitute that class. If you point me to a bundle or manifest with
stable identities, I will run it independently and record the
implementation source, environment, hashes, and results.

I also have smaller findings around duplicate unit identifiers, R0-R8
versus "eight mutants", technique/depth being part of the stated bound
but not represented in the normative object, the hiding property of the
`withheld` digest, and one sentence in the XCCDF comparison. I held
those back here to keep this aligned with the four tests you actually
asked for.

If any of the four above is closed by a sidecar rule that is not visible
in the -00 text, send me the exact bytes and I will rerun the analysis
against them.

Best,
Tiago


Note added before sending, after Walter's message arrived. The review
above had already been frozen and submitted to OpenTimestamps before I
read his message, and I am leaving it unchanged rather than editing it
in hindsight. The stamped file has SHA-256
`7717c82bba7842257cd0757945b8cbd2445390d47b2a29cb4d305671c2fbecce`. I
can provide the exact file and its OpenTimestamps receipt so that the
commitment to those bytes can be verified independently.

Walter independently reached the same derived-denominator problem as my
third finding, but he also identified an adjacent gap I had not: an
enumeration basis records its method without binding the exact
identifier set that method returned. His proposal to digest the ordered
identifier set closes that output-binding gap. I think the two repairs
are complementary rather than alternatives: a composable or derived
basis preserves how the population was obtained, and the digest fixes
exactly which population was obtained. Neither substitutes for the
other, since a digest of the set does not let anyone re-derive it
without the source and the predicate.

His `not_sampled` and `not_yet_due` cases also reach the
closed-vocabulary problem independently of my `aborted_before_dispatch`
case. That makes me more confident this is a boundary in the current
taxonomy rather than one missing label. His first test also finds a
different denominator attack, fixing a declared population after the
results are known. That is not in my frozen review, and I would rather
leave it visibly as his finding than retrofit it into mine.

On the fourth implementation, Walter and I are blocked on the same
missing artefact. He has said he wants to write from the draft text
alone, without reading your implementations. I think the same
discipline has to hold between the two of us: neither should read the
other's code or results before both are frozen. Ideally we each hash
the implementation and our interpretation notes first, then run
independently against the same pinned conformance bundle. Agreement
would then be worth something. Disagreement would be worth more.

Em quinta-feira, 20 de agosto de 2026 às 23:29, Joel Hillier <jhillier=[e-mail]> escreveu:

> Hi all,
>
> I posted this today. It isn't a SCITT draft, it makes no reference to SCITT, COSE, receipts or transparency logs, but I reckon it still belongs in front of this list.
>
> draft-hillier-coverage-attestation-00, Informational.
>
> The abstract is one sentence and then the consequence: a report can be complete and still silent about its own scope. A statement that something was not observed gets recorded in a form that reads as a claim about the world, when what was established was a claim about a bounded population examined to a stated depth. Nothing in the record distinguishes the two, and no relying party can recover the difference afterwards.
>
> Why it should still land here. Four of us have spent a few days converging on a wall, from four designs in four subject matters, and the wall is that completeness of what was anchored is not proof that nothing was withheld from anchoring. This draft sits one layer up from that, and it's the layer where the loss actually happens: before you can ask whether a record set is complete, somebody has to have said what population it was complete of. Almost nothing says.
>
> Specifically, against what each of you has put on the list:
>
> Henri, your sequence catches a record removed from a held set and says nothing about a record never created, and you said so directly. CAP-1 is the vocab for the second half. A conforming document declares a population, a denominator whose basis is itself declared, and an individual accounting for every unit not examined, drawn from a closed set of eight dispositions. A remainder that reconciles only by arithmetic is refused. A count that comes out right and can't say which units it is right about is exactly the artefact this declines to accept.
>
> Emek, the disposition vocabulary is your rung ladder now aimed at a very different question. `not_applicable`, `disabled_by_policy`, `unsupported_input`, `resource_exhausted`, `failed`, `unavailable`, `out_of_scope` and `withheld` are not eight ways of saying "no". They're eight different grounds, and collapsing them into a fraction is the same information loss as reading a result at an unnamed rung. Your rule that an unnamed ground reads at the weakest one has a direct parallel or analogue here, and I think citing yours is the best option.
>
> Walter and Todd, the receiver vantage needs the executor's record set to say what it purports to cover. As things stand - a short set, and a complete set over a narrower population, are the same bytes.
>
> Pablo, three denominator bases are defined - `catalogue`, `enumeration` and `declared` - and which one you're on is your placement question asked at the level of the population rather than the anchor. Another thing worth recognising IMO.
>
> What I'm actually asking for, and it isn't just a good read.
>
> Fifteen vectors ship with it: five positive, ten negative controls, each negative control carrying exactly one mutation and required to fail under the rule it targets rather than under some other rule that happened to catch it. The verifier is mutation-tested: each of the eight normative rules is silenced in turn and the class re-run, and the class must fail each time. Eight rules, eight mutants, eight kills. Three implementations accompany it, one on Node built-ins, one on the Python standard library, one a single-file HTML verifier, and there's a production run at a scale of 227 catalogued check identifiers with both conforming and non-conforming fixtures published.
>
> Please try to break it. Specifically:
>
> Construct a document that satisfies all eight rules and still misleads a reader about what was examined. That's the failure the draft exists to really prevent, and it's the one I'm least able to test myself for obvious reasons.
>
> Find a disposition that belongs in the closed set and is not in it. A closed vocabulary is a strong claim and I think eight is a suspiciously round number.
>
> Break the denominator basis distinction. If `catalogue`, `enumeration` and `declared` do not partition the real cases, the whole accounting rests on a trichotomy that leaks.
>
> Here's the claim I'd probably attack first if it were somebody else's. "Eight rules, eight mutants, eight kills" and "three implementations agree on all fifteen vectors" are claims about my own rigour, and this list has spent the last week or so establishing that those are the claims most likely to be false, because nothing was pointed at them. Three implementations written by one author from one reading of one text agree that the author is self-consistent. That's not nothing, and it also isn't what the sentence sounds like. The mutation result is the stronger of the two, since a silenced rule the class still passes is a real finding, but it tests the verifier against the rules and not the rules against the world.
>
> So the thing that would actually move it is a fourth implementation by somebody who has never actually spoken to me IRL, working from the text alone, disagreeing with the other three about a vector. Henri, that is the shape of what you built the run register for, and I would rather CAP-1 acquired one independent run than another rule.
>
> What it deliberately isn't. It carries no action log, no transformation sealing, no quality assessment and no trust establishment, and it doesn't tell you whether to believe the examiner. It's basically just coverage accounting and nothing else. Section 1.2 states the relationship to existing practice before the draft makes any claim of its own: coverage accounting with a declared denominator is settled in configuration assessment and in vulnerability scanning, XCCDF has had it since 2012 and the PCI ASV programme since 2006. What is new is separating the vocabulary at three points where those formats lose information, not the idea of counting what you looked at.
>
> Which is also why I think it composes with the work here rather than competing. A transparency layer records that something was attested. CAP-1 gives the thing being attested a legible scope. A receipt over a document that can't genuinely or specifically say what it covered is a receipt for an unbounded claim, and this list has three drafts whose payloads would be more useful bounded.
>
> Wrong answers welcome, and preferred.
>
> Joel

--
SCITT mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]