[jose] DARE: PKCS#7, Blockchain in JSON/JOSE

[Phillip Hallam-Baker <[e-mail]>]

[cc's to the JOSE list since this is extending JOSE]

UDF is one of the Mesh technologies that could be used independently to
build other systems. DARE is another. The spec is available from the IETF
tools or in HTML form as:

http://mathmesh.com/Documents/draft-hallambaker-mesh-dare.html

This format is used within the Mesh protocols to support authentication at
the Message and presentation layers. It is also used as the basis for the
Mesh persistence stores such as an end-to-end encrypted password catalog.

Using append only logs authenticated by means of a Merkle tree as the basis
for the persistence model turns out to be very powerful. Devices,
Passwords, Tasks are added to or removed from a catalog by adding entries
to a log. The same approach is used to implement message spools, the
approach bearing some similarity to ideas used in the IBM CICS system (or
at least my interpretation of said system as described in a conversation
mediated by many pints of Marstons)

The same approach could be applied to many other applications. For test
purposes only, I developed a ZIP archive type format out of it which seems
to be functional. I am also planning to use this format to encrypt the logs
kept by Mesh Services. The incremental encryption scheme avoids the need to
perform a key agreement for every incremental update which would slow down
reading and writing greatly and bloat the log size. A process need only
perform a key exchange when it starts writing to the log and refresh it as
policy dictates (e.g hourly). Also two different processes can write to the
same log simultaneously, encrypting their entries under different key
agreements.


The starting point for the design was the observation that the Mesh needs a
PKCS#7/CMS type cryptographic envelope and a Merkle Tree type
authentication mechanism. As I worked on the two independently, I realized
that instead of there being two systems, these needed to be designed to
work together so that a series of envelopes could be concatenated in an
append only log to make a stack or container.

The result is DARE which applies the work of JSON Signature and Encryption
but with some restrictions. I am a big fan of the Rails part of Ruby on
Rails. Sticking to one way to identify things and applying it consistently
is very powerful. DARE requires that every key be identified by its UDF
fingerprint and only its UDF fingerprint. Any other information
(certificates, SAML assertions, the key value etc. is merely provided to
assist).

So first design decision, why JSON?

1) No matter how much argument is made, a solid majority of IETF-ers
despise ASN.1 and will refuse to do any new project based on ASN.1

2) JSON is clearly the favorite data serialization at present and has clear
advantages over XML while allowing for a document-friendly text encoding.

3) The main drawback to JSON is the lack of a binary encoding. This is not
actually much of a problem for the Mesh as the data items are actually
quite small.

So the decision made was to make use of JSON and address the encoding
efficiency issue by proposing an OPTIONAL extension to JSON encoding that
drops in length/ prefixed data sequences for binary and character data:

http://mathmesh.com/Documents/draft-hallambaker-jsonbcd.html

The reason I prefer extending JSON to moving to CBOR is that extending JSON
means that an implementation can use a single decoder for JSON and JSON-B.
That greatly simplifies the design of the code and removes the need to
negotiate encodings and the number of different code paths to test.

Another, secondary issue is that I wanted to be able to encode streams of
data without buffering. This meant I had to extend any of the options on
offer io


The second big concern was how to support separate encryption of a content
body and related content data in a secure and flexible fashion. Consider
the S/MIME problem of encrypting the subject line on a message with a
powerpoint document of 3MB. It is obvious we want to encrypt the subject
line for security. It is also obvious that we don't want to have to decrypt
the whole document just to read the subject line. And for proper hygiene, I
want each piece of data encrypted under a completely different key.

The approach I took was to make use of KDF to generate encryption keys, MAC
keys and IVs.

One concern that was raised is that it is frequently desirable to erase
information such as a file from a container. Overwriting a 3MB powerpoint
file with zeros means modifying a large chunk of the file and does not
provide atomicity. Most O/S will guarantee that a 32KB file update is
atomic but no more.

To overcome this issue, I introduced the option of adding in a nonce as a
mix-in to the KDF. If a process needs to delete an entry, they only need to
erase the nonce and the whole encrypted file becomes unrecoverable.

This in turn lead to the idea of incremental encryption. If I have a
sequence of envelopes, I can use a single key exchange to generate a master
key which is then used together with a unique nonce per as input to the KDF
used to encrypt each envelope.


The envelope format is designed to support single pass, unbuffered encoding
with optional encryption and authentication. Each envelope has a header
(required), a body and an optional trailer.

The container format is almost just a sequence of enveloped but with one
important twist. Each envelope begins and ends with a length indicator. And
the final length indicator is written in reverse (another break with CBOR).

This approach makes it possible to read containers in either the forward or
reverse direction with equal efficiency. This is very useful because 90% of
the time, the most interesting material in an append only log file is to be
found at or near the end.

The spec proposes that processes could periodically add indexes to a
container allowing O(1) retrieval of specific records by index. This has
not (yet) been implemented.


While the DARE format was originally designed to allow the use of the
meta-cryptography techniques I am using in the Mesh, it does not actually
introduce any new crypto. This is because the split key techniques I use
affect the implementation of decryption. The encryption approach is
unchanged.