[SCITT] Re: New draft: The Coverage Attestation Profile (draft-hillier-coverage-attestation-00), and a request to break it

[Tiago Pinto <[e-mail]>]

Joel,

I took the four break-tests in your message literally. I reviewed
`draft-hillier-coverage-attestation-00`, dated 20 August 2026, against
the exact bytes with SHA-256
`7a9eeb1fbdb1fee95697622546d2ae7efba762fff193d6ee34765233539ac353`.

I think the core construct survives, but I found four things worth
putting back to you, in the order of your tests.

1. `withheld` has no semantically consistent accounting under the
current model

The Terminology definition says a Disposition is the reason a unit was
not examined. Section 4.1 says `unexamined` contains one entry per unit
not examined. Section 6 then defines `withheld` as "Examined, but the
result is not disclosed in this document." R1 requires `eligible` to
equal `examined` plus the number of individually accounted unexamined
units.

For a withheld unit, I do not see a count assignment that preserves all
four statements. If the unit is included in `examined`, as the
definition of `withheld` requires, and also appears in `unexamined`, R1
counts it twice. If it is excluded from `examined`, the arithmetic can
reconcile but the `examined` count is false by the definition of
`withheld`.

There is a second contradiction in the same mechanism: R2 says every
unexamined entry MUST name a unit, while the Privacy Considerations say
`withheld` exists so that a sensitive unit can be accounted for without
being named in the document.

The obvious defence would be that `unexamined` really means something
like "not examined with a disclosed result". I do not think that saves
the current text; it is the repair. It requires changing the Terminology
definition, the meaning or name of the array, and the R1 accounting
semantics. My instinct would be to separate two axes: whether the unit
was examined, and whether the unit/result is disclosed.

2. I can produce a legitimate non-examination state that fits none of
the closed dispositions

Suppose a unit applies to the subject, is supported, is authorised, has
all dependencies/licences/network available, and was intended to run,
but the overall examination is aborted before that unit is dispatched.
The abort could be operator cancellation, fail-fast after an earlier
condition, or a global threshold.

I cannot place that unit cleanly in any of the eight dispositions. It is
not `not_applicable`, `disabled_by_policy`, `unsupported_input`,
`resource_exhausted`, `failed`, `unavailable`, `out_of_scope`, or
`withheld` under their current definitions.

Section 4.3 makes this sharper: `integrity` says whether every
dispatched unit reached a recorded outcome. In a clean operator
cancellation, every unit that was dispatched may have completed, so
`complete = true` remains consistent with that definition even though
part of the eligible population was never reached. R7 does not repair
this, because it only forces `complete = false` for `failed`,
`resource_exhausted`, or `unavailable`, and none accurately describes
the undispatched unit.

Reducing `eligible` after the abort to the units actually reached would
be worse: that is precisely denominator selection after the fact, the
class of hidden-bound problem the draft is trying to prevent.

I think this needs either a distinct state such as `not_reached` /
`aborted_before_dispatch`, or an explicit widening of an existing
disposition plus corresponding integrity semantics.

3. The denominator basis distinction breaks on a derived population

Consider a denominator constructed by taking external catalogue C at
digest D, enumerating properties of the subject, and applying a
deterministic applicability predicate to C using that enumeration. The
resulting eligible population is a subset E.

`catalogue` does not fully describe the basis because E was not simply
taken from C. `enumeration` does not fully describe it because the
candidate universe came from C. `declared` is plainly weaker than what
happened.

The same issue appears with catalogue unions, catalogue subtraction by
authorisation scope, and other deterministic filters. These seem like
ordinary derived denominators rather than edge cases.

A composable basis, or a `derived` basis that identifies its source
bases and derivation method, would preserve the information that
choosing any one of the current three loses.

4. I cannot yet do the independent implementation/run honestly because
the conformance bytes are not identifiable from the -00 text alone

The draft says the normative shape is given by an accompanying JSON
Schema, that there are fifteen vectors, that three implementations agree
on them, and that production fixtures are published. In the -00 text I
have, I cannot find a stable URI plus digest identifying the exact
schema, vector set, implementations, and fixtures to which those
statements refer.

I would not call that an R4 violation; R4 governs CAP-1 documents, not
the Internet-Draft. But it reproduces the same epistemic shape that R4
is designed to avoid: a set is asserted and described while the exact
revision of the set is not recoverable from the claim itself.

That also blocks the contribution you say would be most useful. I can
build a fourth implementation from the prose rather than porting yours,
but I cannot honestly claim an independent run against "the fifteen
vectors" until I can identify exactly which fifteen bytes/objects
constitute that class. If you point me to a bundle or manifest with
stable identities, I will run it independently and record the
implementation source, environment, hashes, and results.

I also have smaller findings around duplicate unit identifiers, R0-R8
versus "eight mutants", technique/depth being part of the stated bound
but not represented in the normative object, the hiding property of the
`withheld` digest, and one sentence in the XCCDF comparison. I held
those back here to keep this aligned with the four tests you actually
asked for.

If any of the four above is closed by a sidecar rule that is not visible
in the -00 text, send me the exact bytes and I will rerun the analysis
against them.

Best,
Tiago


Note added before sending, after Walter's message arrived. The review
above had already been frozen and submitted to OpenTimestamps before I
read his message, and I am leaving it unchanged rather than editing it
in hindsight. The stamped file has SHA-256
`7717c82bba7842257cd0757945b8cbd2445390d47b2a29cb4d305671c2fbecce`. I
can provide the exact file and its OpenTimestamps receipt so that the
commitment to those bytes can be verified independently.

Walter independently reached the same derived-denominator problem as my
third finding, but he also identified an adjacent gap I had not: an
enumeration basis records its method without binding the exact
identifier set that method returned. His proposal to digest the ordered
identifier set closes that output-binding gap. I think the two repairs
are complementary rather than alternatives: a composable or derived
basis preserves how the population was obtained, and the digest fixes
exactly which population was obtained. Neither substitutes for the
other, since a digest of the set does not let anyone re-derive it
without the source and the predicate.

His `not_sampled` and `not_yet_due` cases also reach the
closed-vocabulary problem independently of my `aborted_before_dispatch`
case. That makes me more confident this is a boundary in the current
taxonomy rather than one missing label. His first test also finds a
different denominator attack, fixing a declared population after the
results are known. That is not in my frozen review, and I would rather
leave it visibly as his finding than retrofit it into mine.

On the fourth implementation, Walter and I are blocked on the same
missing artefact. He has said he wants to write from the draft text
alone, without reading your implementations. I think the same
discipline has to hold between the two of us: neither should read the
other's code or results before both are frozen. Ideally we each hash
the implementation and our interpretation notes first, then run
independently against the same pinned conformance bundle. Agreement
would then be worth something. Disagreement would be worth more.

Em quinta-feira, 20 de agosto de 2026 às 23:29, Joel Hillier <jhillier=[e-mail]> escreveu:

> Hi all,
> 
> I posted this today. It isn't a SCITT draft, it makes no reference to SCITT, COSE, receipts or transparency logs, but I reckon it still belongs in front of this list.
> 
> draft-hillier-coverage-attestation-00, Informational.
> 
> The abstract is one sentence and then the consequence: a report can be complete and still silent about its own scope. A statement that something was not observed gets recorded in a form that reads as a claim about the world, when what was established was a claim about a bounded population examined to a stated depth. Nothing in the record distinguishes the two, and no relying party can recover the difference afterwards.
> 
> Why it should still land here. Four of us have spent a few days converging on a wall, from four designs in four subject matters, and the wall is that completeness of what was anchored is not proof that nothing was withheld from anchoring. This draft sits one layer up from that, and it's the layer where the loss actually happens: before you can ask whether a record set is complete, somebody has to have said what population it was complete of. Almost nothing says.
> 
> Specifically, against what each of you has put on the list:
> 
> Henri, your sequence catches a record removed from a held set and says nothing about a record never created, and you said so directly. CAP-1 is the vocab for the second half. A conforming document declares a population, a denominator whose basis is itself declared, and an individual accounting for every unit not examined, drawn from a closed set of eight dispositions. A remainder that reconciles only by arithmetic is refused. A count that comes out right and can't say which units it is right about is exactly the artefact this declines to accept.
> 
> Emek, the disposition vocabulary is your rung ladder now aimed at a very different question. `not_applicable`, `disabled_by_policy`, `unsupported_input`, `resource_exhausted`, `failed`, `unavailable`, `out_of_scope` and `withheld` are not eight ways of saying "no". They're eight different grounds, and collapsing them into a fraction is the same information loss as reading a result at an unnamed rung. Your rule that an unnamed ground reads at the weakest one has a direct parallel or analogue here, and I think citing yours is the best option.
> 
> Walter and Todd, the receiver vantage needs the executor's record set to say what it purports to cover. As things stand - a short set, and a complete set over a narrower population, are the same bytes.
> 
> Pablo, three denominator bases are defined - `catalogue`, `enumeration` and `declared` - and which one you're on is your placement question asked at the level of the population rather than the anchor. Another thing worth recognising IMO.
> 
> What I'm actually asking for, and it isn't just a good read.
> 
> Fifteen vectors ship with it: five positive, ten negative controls, each negative control carrying exactly one mutation and required to fail under the rule it targets rather than under some other rule that happened to catch it. The verifier is mutation-tested: each of the eight normative rules is silenced in turn and the class re-run, and the class must fail each time. Eight rules, eight mutants, eight kills. Three implementations accompany it, one on Node built-ins, one on the Python standard library, one a single-file HTML verifier, and there's a production run at a scale of 227 catalogued check identifiers with both conforming and non-conforming fixtures published.
> 
> Please try to break it. Specifically:
> 
> Construct a document that satisfies all eight rules and still misleads a reader about what was examined. That's the failure the draft exists to really prevent, and it's the one I'm least able to test myself for obvious reasons.
> 
> Find a disposition that belongs in the closed set and is not in it. A closed vocabulary is a strong claim and I think eight is a suspiciously round number.
> 
> Break the denominator basis distinction. If `catalogue`, `enumeration` and `declared` do not partition the real cases, the whole accounting rests on a trichotomy that leaks.
> 
> Here's the claim I'd probably attack first if it were somebody else's. "Eight rules, eight mutants, eight kills" and "three implementations agree on all fifteen vectors" are claims about my own rigour, and this list has spent the last week or so establishing that those are the claims most likely to be false, because nothing was pointed at them. Three implementations written by one author from one reading of one text agree that the author is self-consistent. That's not nothing, and it also isn't what the sentence sounds like. The mutation result is the stronger of the two, since a silenced rule the class still passes is a real finding, but it tests the verifier against the rules and not the rules against the world.
> 
> So the thing that would actually move it is a fourth implementation by somebody who has never actually spoken to me IRL, working from the text alone, disagreeing with the other three about a vector. Henri, that is the shape of what you built the run register for, and I would rather CAP-1 acquired one independent run than another rule.
> 
> What it deliberately isn't. It carries no action log, no transformation sealing, no quality assessment and no trust establishment, and it doesn't tell you whether to believe the examiner. It's basically just coverage accounting and nothing else. Section 1.2 states the relationship to existing practice before the draft makes any claim of its own: coverage accounting with a declared denominator is settled in configuration assessment and in vulnerability scanning, XCCDF has had it since 2012 and the PCI ASV programme since 2006. What is new is separating the vocabulary at three points where those formats lose information, not the idea of counting what you looked at.
> 
> Which is also why I think it composes with the work here rather than competing. A transparency layer records that something was attested. CAP-1 gives the thing being attested a legible scope. A receipt over a document that can't genuinely or specifically say what it covered is a receipt for an unbounded claim, and this list has three drafts whose payloads would be more useful bounded.
> 
> Wrong answers welcome, and preferred.
> 
> Joel