Re: [apps-discuss] Concise Binary Object Representation (CBOR) -- support streaming

Phillip Hallam-Baker <> Thu, 23 May 2013 02:04 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 6E2C511E812A for <>; Wed, 22 May 2013 19:04:57 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, NO_RELAYS=-0.001]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id LnGgjokq9AZn for <>; Wed, 22 May 2013 19:04:56 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4010:c03::236]) by (Postfix) with ESMTP id BF1FA21F826B for <>; Wed, 22 May 2013 19:04:49 -0700 (PDT)
Received: by with SMTP id eg20so2686881lab.27 for <>; Wed, 22 May 2013 19:04:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=u7pUK676CuSvCHA/G6CAh4WM63Tzi7Y3ZzMNfaYIQiU=; b=lP2rXlFqZxTXLkOvZt2BlDggENoioErEBMld0Xpc+3ljK6GZ/6nO8slAsB2Ak73VTh +M2IKUOTN2P2gbDw/1HdQhvapNeQV5xMRY7k/UTLJM7nuo7kzehthYTQGJIlSZAe1mY0 yWJ/iXvH2VJydmIrCnjPuRdX8GnIc2Gr8VUWg97gVSW+iP4dKdzcbb/CXrdV/EDhLXNr G94htGceMoAcASjcwuNJhjwnbRhf3VY+PsT+RFp6PZnYCjdGunMyjjw/moj5LQxVcLX1 tOei//pcUHCIn4x7KplCpqw1yIoJo8l8GhEiBIQnwuqboogEf/HGzQXMa6XFQ9x1vVUO /Ajg==
MIME-Version: 1.0
X-Received: by with SMTP id l7mr5371250lbv.42.1369274681606; Wed, 22 May 2013 19:04:41 -0700 (PDT)
Received: by with HTTP; Wed, 22 May 2013 19:04:41 -0700 (PDT)
In-Reply-To: <>
References: <> <>
Date: Wed, 22 May 2013 22:04:41 -0400
Message-ID: <>
From: Phillip Hallam-Baker <>
To: "Manger, James H" <>
Content-Type: multipart/alternative; boundary="14dae93d8eacff4bfb04dd591a20"
Cc: "" <>
Subject: Re: [apps-discuss] Concise Binary Object Representation (CBOR) -- support streaming
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion of application-layer protocols <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 23 May 2013 02:04:57 -0000

On Wed, May 22, 2013 at 9:49 PM, Manger, James H <> wrote:

> CBOR [draft-bormann-cbor] looks great.
> The only big problem I see is that sizes have to be known upfront. You
> need to know how many bytes, array elements, or map pairs you have BEFORE
> you start encoding them. This really hinders streaming content. JSON does
> not have this limitation so CBOR should not introduce it. Removing this
> limitation would better meet objective 5: "be applicable to both
> constrained nodes and high-volume applications".
> This should not be hard to solve, nor to implement.
> For instance define initial bytes for "Array", "Map", "Byte array in
> parts", and "End". An array is the items between "Array" and "End". A map
> is an even number of items between "Map" and "End". There would be no need
> for the 64 initial bytes assigned to known-size versions of arrays and maps.
> The 32 initial bytes for known-length byte arrays would remain. The "Byte
> array in parts" initial byte would be an additional option. Between "Byte
> array in parts" and "End" can be any number of known-length byte array
> items.

That is the reason that DER encoding is such a major pain. Only DER makes
it even harder as the internal sizes are indeterminate...

It has to be possible to serialize and deserialize without any additional
state or pre-computation other than maintaining the position in the tree

Having to know the number of objects in a list would make transcoding from
JSON to binary form a real pig. It would be necessary to buffer objects
before you can push out the first byte.

This is the type of issue that is critical for me for an on the wire
format. Being able to index into specific content is the sort of thing I
might care about in a storage format. I might care about things like byte
alignment as well. Or maybe not.