[SCITT] Re: New draft: The Coverage Attestation Profile (draft-hillier-coverage-attestation-00), and a request to break it

[[e-mail]]

Joel,

Walter and Tiago have the three break-tests, and on the two we both reached they got further than I did, so I will not restate those. Two things are left that I can add, and one of them is a measurement rather than a reading.

Two implementations, and what they disagree about.

Tiago says he cannot do the independent run honestly because the conformance bytes are not identifiable from the -00 text. He is right, and I can put a number on what that costs.

Two verifiers were written from the -00 text and nothing else: one by me, and a second produced independently on my side by a reader who had not seen the first and did not have it. Neither was changed after the other was read. Standard library, no network, R0 through R8. Your own distinction applies to them exactly as it applies to your three: they are implementation independent, not author independent. What they are independent of is each other, and of you.

Each was run against the other's conformance class, twenty documents:

    cases: 20   agreements: 12   disagreements: 8

Ten of the twelve agreements are the second class, ten documents, including all nine of its negative controls, each refused by the rule it targets. The other two agreements are the shape control and the basis control in the first class. On that evidence the rules are implementable from the prose.

The eight disagreements are one disagreement, propagated:

    A/positive-base    A: conforming    B: refused under R4

Both readings implement "a catalogue basis MUST carry a catalogue digest". The first wrote the digest as a string, sha256:... . The second read Section 9, which does not mandate an algorithm but requires the producer to state which was used, and took that to call for an object, {alg, value}. Both forms state the algorithm and neither is a misreading. The JSON Schema named in Section 4 would have settled it in one line, and it is not in the Internet-Draft, so that member has no shape anywhere a reader can reach. The second verifier stops at its first refusal, so the single divergence then masks the intended rule in seven further documents, which is a smaller finding in its own right: "refused by the rule it targets rather than by some other rule" is only checkable against a verifier that reports every refusal. Against a short-circuiting one it is a fact about check order.

So the cost of the missing schema is not that an implementation is hard. It is that two careful readings of the same sentence produce different verdicts on a conforming document, and neither reader can tell which one you meant. Publish the schema and the vectors and there are two verifiers here that can be run against your fifteen the same day.

R5 has a clause no negative control can isolate.

    R5 control, every rule live   : ['R1', 'R5']

    R5 control, R5 silenced       : ['R1']

R1 requires eligible == examined + the number of individually accounted unexamined units, and that number cannot be negative, so any document where examined exceeds eligible already violates R1. Silencing R5 does not stop the class refusing that document; the mutant survives.

The second reading found the way through, and it is worth stating because it is the difference between the two clauses. Its R5 control sets examined to -1 while keeping R1's equality intact, and both verifiers refuse that under R5 alone. So R5's first sentence, counts are non-negative integers, is testable. Its second sentence, examined MUST NOT exceed eligible, has no document that violates it alone, and by your own standard in 7.2 a rule no control exercises is decoration. It is entailed by R1. Either it goes, or R1 weakens to a bound and R5 carries the equality; as the two stand, one of them is unreachable.

Adjacent and probably the same thread to pull: Section 5 numbers nine rules, R0 through R8. Section 7.2 says eight. Both readings applied nine and both flagged it independently. A reader cannot tell which rule is not mutation-tested, and the mutation claim is the load-bearing one for the whole conformance section.

One door Walter's construction leaves standing.

Walter moves the denominator's edge after the answers are known. R8 lets you leave the edge alone and move the claim instead. A stratum cited by an absence assertion must state which classes of claim it supports; it need not state truthfully, and nothing requires the cited claim to fall inside what it states. Five TLS configuration checks may declare supports: ["absence-of-any-vulnerability"] and bound "no vulnerability was observed in the release". Both verifiers call that conforming. R8 as written checks presence. The failure it names is containment.

On Tiago's withheld contradiction, one case beside it rather than inside it. His unit was examined and has a result that is not disclosed. Mine was examined and has no result: dispatched, ran, terminated normally, determined nothing. failed says it errored and it did not; unavailable says it was never dispatched and it was. examined counts units, not verdicts, so that unit is counted as examined and the document is conforming while a reader concludes a determination exists. I have an interest here and will say so rather than let it be noticed: indeterminate is the word my own draft uses for this, and I would rather be wrong about the gap than right about the borrowing.

Both verifiers, both classes and the cross-run are yours if you want them.

Emek

On Fri, Aug 21, 2026 at 3:05 AM Tiago Pinto <tiago=[e-mail]> wrote:

Joel,

I took the four break-tests in your message literally. I reviewed
`draft-hillier-coverage-attestation-00`, dated 20 August 2026, against
the exact bytes with SHA-256
`7a9eeb1fbdb1fee95697622546d2ae7efba762fff193d6ee34765233539ac353`.

I think the core construct survives, but I found four things worth
putting back to you, in the order of your tests.

1. `withheld` has no semantically consistent accounting under the
current model

The Terminology definition says a Disposition is the reason a unit was
not examined. Section 4.1 says `unexamined` contains one entry per unit
not examined. Section 6 then defines `withheld` as "Examined, but the
result is not disclosed in this document." R1 requires `eligible` to
equal `examined` plus the number of individually accounted unexamined
units.

For a withheld unit, I do not see a count assignment that preserves all
four statements. If the unit is included in `examined`, as the
definition of `withheld` requires, and also appears in `unexamined`, R1
counts it twice. If it is excluded from `examined`, the arithmetic can
reconcile but the `examined` count is false by the definition of
`withheld`.

There is a second contradiction in the same mechanism: R2 says every
unexamined entry MUST name a unit, while the Privacy Considerations say
`withheld` exists so that a sensitive unit can be accounted for without
being named in the document.

The obvious defence would be that `unexamined` really means something
like "not examined with a disclosed result". I do not think that saves
the current text; it is the repair. It requires changing the Terminology
definition, the meaning or name of the array, and the R1 accounting
semantics. My instinct would be to separate two axes: whether the unit
was examined, and whether the unit/result is disclosed.

2. I can produce a legitimate non-examination state that fits none of
the closed dispositions

Suppose a unit applies to the subject, is supported, is authorised, has
all dependencies/licences/network available, and was intended to run,
but the overall examination is aborted before that unit is dispatched.
The abort could be operator cancellation, fail-fast after an earlier
condition, or a global threshold.

I cannot place that unit cleanly in any of the eight dispositions. It is
not `not_applicable`, `disabled_by_policy`, `unsupported_input`,
`resource_exhausted`, `failed`, `unavailable`, `out_of_scope`, or
`withheld` under their current definitions.

Section 4.3 makes this sharper: `integrity` says whether every
dispatched unit reached a recorded outcome. In a clean operator
cancellation, every unit that was dispatched may have completed, so
`complete = true` remains consistent with that definition even though
part of the eligible population was never reached. R7 does not repair
this, because it only forces `complete = false` for `failed`,
`resource_exhausted`, or `unavailable`, and none accurately describes
the undispatched unit.

Reducing `eligible` after the abort to the units actually reached would
be worse: that is precisely denominator selection after the fact, the
class of hidden-bound problem the draft is trying to prevent.

I think this needs either a distinct state such as `not_reached` /
`aborted_before_dispatch`, or an explicit widening of an existing
disposition plus corresponding integrity semantics.

3. The denominator basis distinction breaks on a derived population

Consider a denominator constructed by taking external catalogue C at
digest D, enumerating properties of the subject, and applying a
deterministic applicability predicate to C using that enumeration. The
resulting eligible population is a subset E.

`catalogue` does not fully describe the basis because E was not simply
taken from C. `enumeration` does not fully describe it because the
candidate universe came from C. `declared` is plainly weaker than what
happened.

The same issue appears with catalogue unions, catalogue subtraction by
authorisation scope, and other deterministic filters. These seem like
ordinary derived denominators rather than edge cases.

A composable basis, or a `derived` basis that identifies its source
bases and derivation method, would preserve the information that
choosing any one of the current three loses.

4. I cannot yet do the independent implementation/run honestly because
the conformance bytes are not identifiable from the -00 text alone

The draft says the normative shape is given by an accompanying JSON
Schema, that there are fifteen vectors, that three implementations agree
on them, and that production fixtures are published. In the -00 text I
have, I cannot find a stable URI plus digest identifying the exact
schema, vector set, implementations, and fixtures to which those
statements refer.

I would not call that an R4 violation; R4 governs CAP-1 documents, not
the Internet-Draft. But it reproduces the same epistemic shape that R4
is designed to avoid: a set is asserted and described while the exact
revision of the set is not recoverable from the claim itself.

That also blocks the contribution you say would be most useful. I can
build a fourth implementation from the prose rather than porting yours,
but I cannot honestly claim an independent run against "the fifteen
vectors" until I can identify exactly which fifteen bytes/objects
constitute that class. If you point me to a bundle or manifest with
stable identities, I will run it independently and record the
implementation source, environment, hashes, and results.

I also have smaller findings around duplicate unit identifiers, R0-R8
versus "eight mutants", technique/depth being part of the stated bound
but not represented in the normative object, the hiding property of the
`withheld` digest, and one sentence in the XCCDF comparison. I held
those back here to keep this aligned with the four tests you actually
asked for.

If any of the four above is closed by a sidecar rule that is not visible
in the -00 text, send me the exact bytes and I will rerun the analysis
against them.

Best,
Tiago


Note added before sending, after Walter's message arrived. The review
above had already been frozen and submitted to OpenTimestamps before I
read his message, and I am leaving it unchanged rather than editing it
in hindsight. The stamped file has SHA-256
`7717c82bba7842257cd0757945b8cbd2445390d47b2a29cb4d305671c2fbecce`. I
can provide the exact file and its OpenTimestamps receipt so that the
commitment to those bytes can be verified independently.

Walter independently reached the same derived-denominator problem as my
third finding, but he also identified an adjacent gap I had not: an
enumeration basis records its method without binding the exact
identifier set that method returned. His proposal to digest the ordered
identifier set closes that output-binding gap. I think the two repairs
are complementary rather than alternatives: a composable or derived
basis preserves how the population was obtained, and the digest fixes
exactly which population was obtained. Neither substitutes for the
other, since a digest of the set does not let anyone re-derive it
without the source and the predicate.

His `not_sampled` and `not_yet_due` cases also reach the
closed-vocabulary problem independently of my `aborted_before_dispatch`
case. That makes me more confident this is a boundary in the current
taxonomy rather than one missing label. His first test also finds a
different denominator attack, fixing a declared population after the
results are known. That is not in my frozen review, and I would rather
leave it visibly as his finding than retrofit it into mine.

On the fourth implementation, Walter and I are blocked on the same
missing artefact. He has said he wants to write from the draft text
alone, without reading your implementations. I think the same
discipline has to hold between the two of us: neither should read the
other's code or results before both are frozen. Ideally we each hash
the implementation and our interpretation notes first, then run
independently against the same pinned conformance bundle. Agreement
would then be worth something. Disagreement would be worth more.

Em quinta-feira, 20 de agosto de 2026 às 23:29, Joel Hillier <jhillier=[e-mail]> escreveu:

> Hi all,
>
> I posted this today. It isn't a SCITT draft, it makes no reference to SCITT, COSE, receipts or transparency logs, but I reckon it still belongs in front of this list.
>
> draft-hillier-coverage-attestation-00, Informational.
>
> The abstract is one sentence and then the consequence: a report can be complete and still silent about its own scope. A statement that something was not observed gets recorded in a form that reads as a claim about the world, when what was established was a claim about a bounded population examined to a stated depth. Nothing in the record distinguishes the two, and no relying party can recover the difference afterwards.
>
> Why it should still land here. Four of us have spent a few days converging on a wall, from four designs in four subject matters, and the wall is that completeness of what was anchored is not proof that nothing was withheld from anchoring. This draft sits one layer up from that, and it's the layer where the loss actually happens: before you can ask whether a record set is complete, somebody has to have said what population it was complete of. Almost nothing says.
>
> Specifically, against what each of you has put on the list:
>
> Henri, your sequence catches a record removed from a held set and says nothing about a record never created, and you said so directly. CAP-1 is the vocab for the second half. A conforming document declares a population, a denominator whose basis is itself declared, and an individual accounting for every unit not examined, drawn from a closed set of eight dispositions. A remainder that reconciles only by arithmetic is refused. A count that comes out right and can't say which units it is right about is exactly the artefact this declines to accept.
>
> Emek, the disposition vocabulary is your rung ladder now aimed at a very different question. `not_applicable`, `disabled_by_policy`, `unsupported_input`, `resource_exhausted`, `failed`, `unavailable`, `out_of_scope` and `withheld` are not eight ways of saying "no". They're eight different grounds, and collapsing them into a fraction is the same information loss as reading a result at an unnamed rung. Your rule that an unnamed ground reads at the weakest one has a direct parallel or analogue here, and I think citing yours is the best option.
>
> Walter and Todd, the receiver vantage needs the executor's record set to say what it purports to cover. As things stand - a short set, and a complete set over a narrower population, are the same bytes.
>
> Pablo, three denominator bases are defined - `catalogue`, `enumeration` and `declared` - and which one you're on is your placement question asked at the level of the population rather than the anchor. Another thing worth recognising IMO.
>
> What I'm actually asking for, and it isn't just a good read.
>
> Fifteen vectors ship with it: five positive, ten negative controls, each negative control carrying exactly one mutation and required to fail under the rule it targets rather than under some other rule that happened to catch it. The verifier is mutation-tested: each of the eight normative rules is silenced in turn and the class re-run, and the class must fail each time. Eight rules, eight mutants, eight kills. Three implementations accompany it, one on Node built-ins, one on the Python standard library, one a single-file HTML verifier, and there's a production run at a scale of 227 catalogued check identifiers with both conforming and non-conforming fixtures published.
>
> Please try to break it. Specifically:
>
> Construct a document that satisfies all eight rules and still misleads a reader about what was examined. That's the failure the draft exists to really prevent, and it's the one I'm least able to test myself for obvious reasons.
>
> Find a disposition that belongs in the closed set and is not in it. A closed vocabulary is a strong claim and I think eight is a suspiciously round number.
>
> Break the denominator basis distinction. If `catalogue`, `enumeration` and `declared` do not partition the real cases, the whole accounting rests on a trichotomy that leaks.
>
> Here's the claim I'd probably attack first if it were somebody else's. "Eight rules, eight mutants, eight kills" and "three implementations agree on all fifteen vectors" are claims about my own rigour, and this list has spent the last week or so establishing that those are the claims most likely to be false, because nothing was pointed at them. Three implementations written by one author from one reading of one text agree that the author is self-consistent. That's not nothing, and it also isn't what the sentence sounds like. The mutation result is the stronger of the two, since a silenced rule the class still passes is a real finding, but it tests the verifier against the rules and not the rules against the world.
>
> So the thing that would actually move it is a fourth implementation by somebody who has never actually spoken to me IRL, working from the text alone, disagreeing with the other three about a vector. Henri, that is the shape of what you built the run register for, and I would rather CAP-1 acquired one independent run than another rule.
>
> What it deliberately isn't. It carries no action log, no transformation sealing, no quality assessment and no trust establishment, and it doesn't tell you whether to believe the examiner. It's basically just coverage accounting and nothing else. Section 1.2 states the relationship to existing practice before the draft makes any claim of its own: coverage accounting with a declared denominator is settled in configuration assessment and in vulnerability scanning, XCCDF has had it since 2012 and the PCI ASV programme since 2006. What is new is separating the vocabulary at three points where those formats lose information, not the idea of counting what you looked at.
>
> Which is also why I think it composes with the work here rather than competing. A transparency layer records that something was attested. CAP-1 gives the thing being attested a legible scope. A receipt over a document that can't genuinely or specifically say what it covered is a receipt for an unbounded claim, and this list has three drafts whose payloads would be more useful bounded.
>
> Wrong answers welcome, and preferred.
>
> Joel

--
SCITT mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]