[Cbor] draft-bormann-cbor-sequence-00 and Big Data

[Burt Harris <[e-mail]>]

draft-borman-cbor-sequence-00 has me thinking, it's a great first draft.

It is unquestionably good to logically distinguish between a single CBOR data item and a (potentially large) stream of CBOR data items.   MIME type makes very good sense, but we might also back it up with by allocating a unique tag (similar to the self-describing CBOR tag number 55799 in RFC7049 section 2.4.5)) as a NO-OP "magic number" at the beginning of a stream.

I think the current name and content of the draft implies the order of the data items in the stream might be logically significant.   I suggest for this media type we should consider overall sequence implementation detail (like the order of keys in a map).   In particular, to facilitate building scalable big-data systems using CBOR, it would be valuable to recognize that if the order of data items is important, a CBOR Array is probably more appropriate.   

Perhaps s/sequence/stream/g might be appropriate in the document because a "sequence" (to me) implies order preservation.

For example, to collect data from a few million IoT devices, I would use a scale-out architecture where several identical front-end machines are load balanced with each one buffering the little cbor messages into a large buffer.   The front-end machines flush their buffers asynchronously, either when the buffer is full or the after expiration of a period.   Due to the asynchronous distributed buffering, the order of the data items may not reflect the actual sequence of events.      This non-sequential logging can be significant even in multicore-processors with sophisticated logging infrastructure.

Similarly, big-data distributed processing architectures, e.g., the map / reduce algorithm; there should be no assumption of global ordering.   But local ordering (within a block) can sometimes be very useful.    For that reason, we might want to consider allocating a single-byte CBOR special (MT 7) value for delimiting portions of a stream where the sequence can be considered meaningful.   For applications not sensitive to the order of the values, it would effectively be a NoOp, but when compression approaches are used in a CBOR stream, it should probably reset the compression state.   A 1-byte NoOp in Major Type 7 can also be useful in padding a CBOR stream into a "splittable" storage format, which aids in distributed analysis processing using frameworks like Hadoop.

Finally, by testing for if a NoOp appears at the last element in a stream, it may be possible to detect if the application writing to the stream paused at that point.    For those familiar with YAML, consider the "..." token.

P.S.  I wanted to thank the members of the WG who participated in moving CDDL forward to become RFC8610, great work!