Re: [apps-discuss] Concise Binary Object Representation (CBOR) -- support streaming

["Manger, James H" <[e-mail]>]

> > sizes have to be known upfront
> 
> Indeed.  The old "counted vs. delimited" debate.
> 
> JSON is delimited because is HAS TO BE -- counted is a non-starter for
> text formats (as amply demonstrated by FORTAN Hollerith data).
> 
> CBOR is counted because that is the best fit for a binary format.
> Fast deserializers really benefit from knowing counts upfront (and that
> also helps with constrained deserializers).

Is this because the deserializer can allocate exactly the right memory for an array (of pointers to elements), instead of having to guess then resize as necessary?

> (The number of things they will have to create, e.g., the number of
> data items; not the byte size of their representation -- counting bytes
> here was a mistake RFC 731 and ASN.1 BER shared and that creates much
> pain in BER.)

Ok. I can see than an item count can avoid *some* of the difficulties with a count of encoded bytes.

> With a byte-oriented encoding, counting also happens to require fewer
> bytes than delimiting for small counts.

Save 1 byte per map and array that has fewer than 28 items. Sounds nice, but not that compelling.

> So if the serializer is in a position to provide count information in
> time, we should use it.
> 
> CBOR supports streaming by having self-delimited data items -- you can
> send many of these in sequence for a stream (as in, say, an XMPP-like
> protocol).
> 
> So streaming *of* data items is not a problem with CBOR.

Supporting streaming -- but only at the top layer of a protocol -- doesn't sound good. If it is useful at a "top-layer" surely it is useful at other layers as well. Protocols are often embedded in other protocols. Once you have something that relies on a stream *of* data items, and you want to embed it in another layer you have to stop using CBOR. That's bad.

> Now let's look at "streaming" *within* a data item.
> 
> The actual need for this is a fringe case, but one that is worth
> looking at.
> The only significant use case I know is a state-limited (on-the-fly)
> converter from a delimited format (such as JSON) to CBOR.
> (Unfortunately, that is not just a use case in gateways; using CBOR as
> a plug-in replacement for JSON may saddle one with an API that makes
> the CBOR serializer such a converter.)
> Putting together an aquarium when being handed fish sticks.

This doesn't feel like a "fringe case". Converting from JSON is one of the listed objectives (#6). The draft has a section on how to do it.

It is generally only for potentially large things that you need to bother with something like CBOR for efficiency, instead of staying with JSON. It is these large things that are going to have unknown sizes at the start.

Plenty of languages pass around iterators, instead of arrays or maps, because they are a useful abstraction that covers more situations -- but they don't give you a size up front.

> Anything that addresses this specific use case needs to be a hybrid.
> (Purely delimited is possible for binary, but tends to be more taxing
> on the deserializer; it also often requires additional work at the
> serializer to enable data transparency.)

What does "enable data transparency" mean?

> I tried to avoid designing a hybrid.
> Counted is the way things have been moving to in much of CS for the
> last 10, 15 years.
> 
> As you say, delimiters (push/pop, or maybe some form of continuation
> scheme) can be easily hacked into CBOR, the only victim being its
> architecture.
> Should we?

Yes.

I'm not convinced that situations where you don't know the size of a byte array, text, array, or map up front are so rare that CBOR can ignore them.

I am not familiar enough with squeezing performance from a deserializer to judge whether a hybrid is worth the complexity cost over just delimiters for maps & arrays.

--
James Manger