Re: [apps-discuss] Concise Binary Object Representation (CBOR) -- support streaming

[Carsten Bormann <[e-mail]>]

On May 23, 2013, at 03:49, "Manger, James H" <[e-mail]> wrote:

> sizes have to be known upfront

Indeed.  The old "counted vs. delimited" debate.

JSON is delimited because is HAS TO BE -- counted is a non-starter for text formats (as amply demonstrated by FORTAN Hollerith data).

CBOR is counted because that is the best fit for a binary format.
Fast deserializers really benefit from knowing counts upfront (and that also helps with constrained deserializers).
(The number of things they will have to create, e.g., the number of data items; not the byte size of their representation -- counting bytes here was a mistake RFC 731 and ASN.1 BER shared and that creates much pain in BER.)
With a byte-oriented encoding, counting also happens to require fewer bytes than delimiting for small counts.
So if the serializer is in a position to provide count information in time, we should use it.

CBOR supports streaming by having self-delimited data items -- you can send many of these in sequence for a stream (as in, say, an XMPP-like protocol).

So streaming *of* data items is not a problem with CBOR.
Now let's look at "streaming" *within* a data item.

The actual need for this is a fringe case, but one that is worth looking at.
The only significant use case I know is a state-limited (on-the-fly) converter from a delimited format (such as JSON) to CBOR.
(Unfortunately, that is not just a use case in gateways; using CBOR as a plug-in replacement for JSON may saddle one with an API that makes the CBOR serializer such a converter.)
Putting together an aquarium when being handed fish sticks.

Anything that addresses this specific use case needs to be a hybrid.
(Purely delimited is possible for binary, but tends to be more taxing on the deserializer; it also often requires additional work at the serializer to enable data transparency.)
I tried to avoid designing a hybrid.
Counted is the way things have been moving to in much of CS for the last 10, 15 years.

As you say, delimiters (push/pop, or maybe some form of continuation scheme) can be easily hacked into CBOR, the only victim being its architecture.
Should we?

Grüße, Carsten