[apps-discuss] binarypack: streaming, very large values, indefinite lengths

["Manger, James H" <[e-mail]>]

Carsten,

Binarypack [draft-bormann-apparea-bpack] looks fairly simple and efficient.

One limitation is that you have to know the length of a string, the number of items in an array, and the number of key/value pairs in a table (aka object or map) before you start encoding them.

JSON does not have this limitation.

This is likely to make binarypack unsuitable for really large values, for values created in a streaming mode, or for tools that want to filter a stream as it passes.

One solution would be to allow values of these types to be sent in pieces. A new "partial" representation for these types would be defined (using the same syntax as the representation that uses a 4-byte length field). Any number of those can be concatenated to convey a full value (terminated with a non-partial representation of the same type, or defining a special terminator).

So "Hello, World!" could be encoded as:
  D9 0000000D 48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21
Or (picking D7 to indicate a "partial" string)
  D7 00000005 48 65 6C 6C 6F
  D7 00000008 2C 20 57  6F 72 6C 64 21
  B0

P.S. I believe protocol buffers have the same scalability limitation.

P.S. ASN.1 BER supports indefinite-length encodings to overcome this limitation.

P.S. Being able to encode parts of a larger field is also useful for occasional manual debugging as you can modify a value (add/remove bytes, characters, array items, or key/value pairs) without having to recalculate nested length fields all the way up to the root of the message.

--
James Manger