[Acme] Signed JSON document / Json Content Metaheader / JSON Container

[Phillip Hallam-Baker <[e-mail]>]

All,

I would like to propose a spec similar to the JSON log file spec but with
the purpose of being a container for JSON signatures. The immediate
application for this might be for ACME or other JSON/Rest Web services.

The elevator pitch for the idea is that a JSON Container would be a JSON
Metadata blob with the content being described appended to the end in a way
that makes the separation between content and metadata simple and
unambiguous.

<JSON-BLOB> [Separator] <content-blob>

Advantages:

* Can read the metadata for a file etc with a plain ASCII editor or command
line tools like cat, more, etc.

* Avoids the need to BASE64 armor the content. So if the content is JSON or
other ASCII/Unicode text it remains readable in an ASCII/Unicode editor.

* Can add signatures, digests and other metadata items to content in a
simple regular fashion.

* Content and metadata can be transported as a simple regular blob that is
fully abstracted from and independent of the transport and without the need
for canonicalization, XPath or other complications.


Right now, the IETF spec for describing content metadata is based on
RFC822. And by 'based on', I mean that it is a find the needle in the RFC
haystack approach. RFC822 style content headers as currently used have
several disadvantages:

* It is a flat structure, headers can have attributes but trying to add
more structure is painful.

* It is a historical artifact and content metadata headers are mixed in
with protocol metadata headers without rhyme or reason.

* It is not JSON and JSON is the spec we are now converging on for this
type of work. It has all the expressive capabilities of XML and ASM.1
without the insanities and stupid. It is as easy to read and write as
RFC822 but without the limitations.


The spec itself would be simple, its
basically draft-ietf-json-text-sequence applied to a sequence of one json
object and one content blob.

JSON sequence specifies that "A JSON text sequence consists of any number
of JSON texts, all encoded in UTF-8, each prefixed by an ASCII Record
Separator (0x1E), and each ending with an ASCII Line Feed character (0x1A)"

This is not ideal for JSON Container. We would prefer that the character
which is illegal in a JSON document be the one that is the separator
between the header and content.

All indexes used in the metadata header are calculated from the Record
Separator byte, the first byte following being byte 0 and so on.


Applying this in a Web Service is very simple, our messages now have the
form:

POST / HTTP/1.1
Host: example.com
Content-Type: application/json-container
Content-Length: 666

{ "Signature" : "wefwkjefkljwehfjklwhejkflh" }

<0x1E>{ "Service-Type" : "acme.ws",
   "Transaction-ID" : "2h23roih23oih23orh",
   "Register" : { ....<web service parameters here> ... } }

Where <0x1E> is the record separator character.

The scope of the signature is the content, the whole content, AND NOTHING
BUT THE CONTENT. Signing headers is a mugs game. Put all the information
that matters into one blob and sign the blob. Do not present developers
with an IKEA self-assembly job.

So to prevent protocol substitution attacks, the service identifier and the
method name wrap the method parameters. In a response, the service
identifier, response code and transaction ID wrap the response data for the
same reason.


Applying the scheme to stored data in a file system is a little different.
First we have to develop a file extension convention:

file.jpg  becomes file.jpg.jsonc

Many jsonc aware applications will have to share data with applications
that are not aware. So we need the option of storing metadata in a separate
file when that is convenient:

The metadata for file.jpg is stored in file.jsonm

In some circumstances we might want to keep the metadata for all the files
in a directory or tree in one file, a metadata container. This could be
jsonmc.jsonmc appearing in either the directory a file is in or one of the
directories above it.

A metadata container would be a JSON container and begin with the usual
metadata header for that file followed by a JSON sequence of entries per
file. If this contains a signature, the file entries can just be digests.


Notice how we have just developed a format that allows us to sign a
complete code distribution including content data very easily. This is
obviously out of scope for ACME but the fact that an approach transfers so
neatly to a completely different problem suggests that it is the right
track.