Re: [Acme] Signed JSON document / Json Content Metaheader / JSON Container

"Manger, James" <> Thu, 29 January 2015 02:51 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id BA1471A8AE1; Wed, 28 Jan 2015 18:51:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 0.299
X-Spam-Status: No, score=0.299 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_AU=0.377, HOST_EQ_AU=0.327, HTML_MESSAGE=0.001, J_CHICKENPOX_45=0.6, J_CHICKENPOX_66=0.6, RCVD_IN_DNSWL_LOW=-0.7, RELAY_IS_203=0.994] autolearn=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id LYn17R88D8Sg; Wed, 28 Jan 2015 18:51:17 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id CA3FC1A8ADC; Wed, 28 Jan 2015 18:51:16 -0800 (PST)
X-IronPort-AV: E=Sophos; i="5.09,484,1418043600"; d="scan'208,217"; a="238511957"
Received: from unknown (HELO ([]) by with ESMTP; 29 Jan 2015 13:32:56 +1100
X-IronPort-AV: E=McAfee;i="5600,1067,7695"; a="229447894"
Received: from ([]) by with ESMTP; 29 Jan 2015 13:51:15 +1100
Received: from ([]) by ([]) with mapi; Thu, 29 Jan 2015 13:51:15 +1100
From: "Manger, James" <>
To: Phillip Hallam-Baker <>, JSON WG <>, "" <>
Date: Thu, 29 Jan 2015 13:51:13 +1100
Thread-Topic: [Acme] Signed JSON document / Json Content Metaheader / JSON Container
Thread-Index: AdA7Jll5lOzfaDYxToiTs6qbsClkAAAOI9oA
Message-ID: <>
References: <>
In-Reply-To: <>
Accept-Language: en-US, en-AU
Content-Language: en-US
acceptlanguage: en-US, en-AU
Content-Type: multipart/alternative; boundary="_000_255B9BB34FB7D647A506DC292726F6E1284ED9AA38WSMSG3153Vsrv_"
MIME-Version: 1.0
Archived-At: <>
Subject: Re: [Acme] Signed JSON document / Json Content Metaheader / JSON Container
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Automated Certificate Management Environment <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 29 Jan 2015 02:51:21 -0000

A signed JAR file meets some of these requirements.
Metadata and signatures are in extra files in the ZIP archive: META-INF/MANIFEST.MF, META-INF/MYKEY.SF, META-INF/MYKEY.RSA.
Content is the other files in the archive.
It is not JSON of course, and the signature & certs are packaged in ASN.1, but it is a useful comparison. It avoids BASE64 on the content; can adds signatures, digests, and other metadata; can transport content and metadata as a regular blob (*.jar file); can sign complete code distributions.

James Manger

From: Acme [] On Behalf Of Phillip Hallam-Baker
Sent: Thursday, 29 January 2015 5:15 AM
Subject: [Acme] Signed JSON document / Json Content Metaheader / JSON Container


I would like to propose a spec similar to the JSON log file spec but with the purpose of being a container for JSON signatures. The immediate application for this might be for ACME or other JSON/Rest Web services.

The elevator pitch for the idea is that a JSON Container would be a JSON Metadata blob with the content being described appended to the end in a way that makes the separation between content and metadata simple and unambiguous.

<JSON-BLOB> [Separator] <content-blob>


* Can read the metadata for a file etc with a plain ASCII editor or command line tools like cat, more, etc.

* Avoids the need to BASE64 armor the content. So if the content is JSON or other ASCII/Unicode text it remains readable in an ASCII/Unicode editor.

* Can add signatures, digests and other metadata items to content in a simple regular fashion.

* Content and metadata can be transported as a simple regular blob that is fully abstracted from and independent of the transport and without the need for canonicalization, XPath or other complications.

Right now, the IETF spec for describing content metadata is based on RFC822. And by 'based on', I mean that it is a find the needle in the RFC haystack approach. RFC822 style content headers as currently used have several disadvantages:

* It is a flat structure, headers can have attributes but trying to add more structure is painful.

* It is a historical artifact and content metadata headers are mixed in with protocol metadata headers without rhyme or reason.

* It is not JSON and JSON is the spec we are now converging on for this type of work. It has all the expressive capabilities of XML and ASM.1 without the insanities and stupid. It is as easy to read and write as RFC822 but without the limitations.

The spec itself would be simple, its basically draft-ietf-json-text-sequence applied to a sequence of one json object and one content blob.

JSON sequence specifies that "A JSON text sequence consists of any number of JSON texts, all encoded in UTF-8, each prefixed by an ASCII Record Separator (0x1E), and each ending with an ASCII Line Feed character (0x1A)"

This is not ideal for JSON Container. We would prefer that the character which is illegal in a JSON document be the one that is the separator between the header and content.

All indexes used in the metadata header are calculated from the Record Separator byte, the first byte following being byte 0 and so on.

Applying this in a Web Service is very simple, our messages now have the form:

Content-Type: application/json-container
Content-Length: 666

{ "Signature" : "wefwkjefkljwehfjklwhejkflh" }

<0x1E>{ "Service-Type" : "<>",
   "Transaction-ID" : "2h23roih23oih23orh",
   "Register" : { ....<web service parameters here> ... } }

Where <0x1E> is the record separator character.

The scope of the signature is the content, the whole content, AND NOTHING BUT THE CONTENT. Signing headers is a mugs game. Put all the information that matters into one blob and sign the blob. Do not present developers with an IKEA self-assembly job.

So to prevent protocol substitution attacks, the service identifier and the method name wrap the method parameters. In a response, the service identifier, response code and transaction ID wrap the response data for the same reason.

Applying the scheme to stored data in a file system is a little different. First we have to develop a file extension convention:

file.jpg  becomes file.jpg.jsonc

Many jsonc aware applications will have to share data with applications that are not aware. So we need the option of storing metadata in a separate file when that is convenient:

The metadata for file.jpg is stored in file.jsonm

In some circumstances we might want to keep the metadata for all the files in a directory or tree in one file, a metadata container. This could be jsonmc.jsonmc appearing in either the directory a file is in or one of the directories above it.

A metadata container would be a JSON container and begin with the usual metadata header for that file followed by a JSON sequence of entries per file. If this contains a signature, the file entries can just be digests.

Notice how we have just developed a format that allows us to sign a complete code distribution including content data very easily. This is obviously out of scope for ACME but the fact that an approach transfers so neatly to a completely different problem suggests that it is the right track.