[SCITT] Consistency of the registry append-only log.

[Ray Lutz <[e-mail]>]

We were talking today about how to deal with various related records 
that may be submitted at various times in the log.
I will outline some thoughts here with the caveat that I have to spend a 
night thinking about it.

I thought at first that the log could be quite brain-dead about this, 
and just accept a payload and secure it with a receipt. If the user 
wants to group them together, then the submitter can submit additional 
entries that provide the grouping. But I don't think this is good enough 
for most use cases.

First, we must admit that records related to specific supply chain 
products will not all be submitted at the same time. In the SW use case, 
there are various points in the development and release and testing 
cycle when submissions about the same product must be submitted 
separately, but thought of as a set.

For a simple sw product, there may be a release of the source code, and 
also later several related submissions, such as testing results.

For the election use case I am currently working to resolve, there 
appear to be several release stages:
1. Release of a set of all the public keys of the devices that will be 
deployed.
2. Incremental releases of cryptographic evidence related to scanning of 
ballots (without releasing the ballots or the vote).
3. Release of all the evidence for auditing and review.
4. Final release of the result of the election upon certification.
5. Submission of auditing results based on all the evidence found.

There are similar scenarios in the sw space, usually related to critical 
code that must be tested to some agreed criteria. The example I used in 
the call was of say self-driving vehicle software that must be tested to 
a standard test suite or the mfr will be liable for crashes. We can't 
trust them to tell the truth about prior submissions, that is one of the 
great things about the append-only log.

These are all logically all part of the same election, so we need a way 
to submit incrementally.
But one threat scenario is a compromised official or hacker, who may 
want to hope that we will forget about the original public keys, and 
substitute a compromised set so the results can be modified. Thus, we 
can't rely on the submitter to be honest about what was submitted, but 
must build that in.

this is a useful Chat GPT response.

The attribute you are describing is called consistency in the context of 
a registry built on an append-only log. Consistency ensures that the 
registry maintains a reliable and accurate state, where all related 
records are present and there are no additional or missing entries.

In an append-only log, new entries are added to the end of the log, 
preserving the order in which they were appended. To achieve 
consistency, the registry must adhere to certain principles:

 1. Append-only: The registry strictly follows the append-only rule,
    meaning that once a record is appended to the log, it cannot be
    modified or deleted. This ensures that the log maintains a reliable
    and immutable history of all entries.

 2. Atomicity: Operations on the registry should be atomic, meaning they
    are treated as indivisible units of work. This guarantees that each
    operation either succeeds completely or fails, preventing partial
    updates that could compromise consistency.

 3. Strong consistency guarantees: The registry should provide strong
    consistency guarantees to ensure that all related records are found
    and there are no additional or missing entries. This means that
    queries made to the registry will always reflect the most up-to-date
    state of the log, and the registry will not return stale or
    inconsistent results.

 4. Indexing and querying: The registry should support efficient
    indexing and querying mechanisms to allow users to find relevant
    entries quickly and accurately. This can be achieved through various
    techniques, such as maintaining indexes or using data structures
    optimized for querying, like balanced search trees or hash tables.

==========
The other important aspect is that the data can be transparently 
inspected. Others can check that the log isn't lying. So the consistency 
guarantee will be important, esp. bc we need to know for sure that there 
are no forgotten entries related to the product of concern.

Sure, the system could provide completely obscured usage, which would be 
fine if there were no incremental submissions for the same product.

Thus, we will need to also provide some exposed metadata with enough 
structure to at least identify the submitter and the product, when 
consistency attribute is desired.

This is not fully thought through but it is a start.

--Ray





-- 
-------
Ray Lutz
Citizens' Oversight Projects (COPs)
http://www.citizensoversight.org
619-820-5321