Re: [Cbor] Record proposal

[Carsten Bormann <[e-mail]>]

Hi Kris,

timely question — the CBOR meeting at IETF111 is tomorrow.

My plan was to look at a corpus of SenML (RFC 8428) data to see whether your proposal would turn out to be useful there.  SenML is a spec that allows building “packs” of records, which in principle can be all different, but in reality fall into a small number of structural categories (e.g., you might have water flow, water temperature, and water pressure coming in from separate sensors, so you cannot readily combine them, but each sensor stream has records that always look the same).

Of course, time ran out before this meeting, so I didn’t manage to do that yet.

The SenML example is different from the CSV (RFC 4180/7111) example that comes to mind immediately in that the records go into different groups, while in a CSV Table all entries have the same structure (usually indicated in the first line in CSV’s text representation).

I would be most happy if we could come up with something that fits SenML and CSV applications, but doesn’t create some of the processing model problems that might come up.  Having to run linear processing means that one cannot extract subsequences from an array, which is a bit like the effect SenML base fields have — most SenML packs have base fields only in the first record.  A hierarchical model (tag applies to its content) would mean at least subtrees are extractable, but there is going to be some redundancy between multiple record arrays — redundancy that cbor-packed could pack away.

I’m not sure we will have much time for discussion tomorrow (1 hour meeting and 10 documents to look at :-), but we should continue on the list and create a proposal for one of the next CBOR interims (I’m actually going to be on vacation on 2021-08-11, but the one on 2021-08-25 would suit me).

Grüße, Carsten


> On 2021-07-29, at 02:39, Kris Zyp <[e-mail]> wrote:
> 
> I just wanted to check on the status of this, and if this registry
> entry is still being considered?
> Thank you,
> Kris
> 
> On Wed, May 5, 2021 at 9:11 AM <[e-mail]> wrote:
>> 
>> To try to clarify a little bit about my proposal, my intent with the record proposal was not primarily about minimizing the size of the encodings. I think that is a nice auxiliary benefit, but is not the primary purpose of the proposed tag. The primary purpose is to assign semantics indicating that a sequence of values should be understood as a structured record. I think this is complementary to tag 259, which indicates the opposite, a set of name-values to be interpreted a map structure. And I think this type of semantic application to a primitive data structure is the intent of tags, if I understand correctly.
>> 
>> And the benefit of being able to use this type of tag/semantics is that encoders can more explicitly align encodings with language structures and class. And this can have much more of a significant impact on performance than space. Serializers can “stream” serialization of array or other sequences, that may be of indefinite length, without a prior knowledge of homogeneity, but still be able to use reuse structural information for any objects/elements that have the same structure, in a way that is easily optimized and can scale to arrays and other structures that may not be scannable ahead of time. For example, an encoder can keep a simple cache of class types that it has serialized and recognize that it has already serialized the record structure in a previous entity, and reference that rather reserializing the structure again. And this type of class/type cache can be significantly faster than doing cache lookups for every primitive value. Likewise, on deserialization, this provides an opportunity for a decoder to use a record reference to find and allocate/initialize exactly the correct class or structure, and then decode values directly into that structure.
>> 
>> And my intent with suggesting this as a tag proposal is that hopefully this can simply be a rather unobtrusive addition the tag registry, keeping aligned with the intent and purpose of tag semantics, without trying to alter the direction CBOR itself at all.
>> 
>> 
>> 
>> Thanks,
>> 
>> Kris
>> 
>> 
>> 
>> Sent from Mail for Windows 10
>> 
>> 
>> 
>> From: Carsten Bormann
>> Sent: May 5, 2021 6:50 AM
>> To: Christian Amsüss
>> Cc: [e-mail]; Kris Zyp
>> Subject: Re: [Cbor] Record proposal
>> 
>> 
>> 
>> On 2021-05-05, at 13:16, Christian Amsüss <[e-mail]> wrote:
>> 
>>> 
>> 
>>> (And as much as I dislike being "the person to whom everything looks
>> 
>>> like a nail", I'll probably ask about whether this fits in the general
>> 
>>> model of packed CBOR, with the first entity setting up a single table
>> 
>>> entry, and then the entries expanding a [] to a {}).
>> 
>> 
>> 
>> There are two potential aspects to the proposed tag:
>> 
>> 
>> 
>> * a more compact representation (which is all that cbor-packed is about)
>> 
>> 
>> 
>> * semantic indication that a specific kind of record is being used
>> 
>> 
>> 
>> Proposed Tag 105 currently does not have a place for further semantic indications, but one could be added.
>> 
>> 
>> 
>> By the way, cbor-packed turns the example I gave in the referenced email into
>> 
>> 
>> 
>> 51([["value", "name"], [], [],
>> 
>>   [{simple(1): "one", simple(0): 1},
>> 
>>    {simple(1): "two", simple(0): 2}, {simple(1): "three", simple(0): 3}]])
>> 
>> 
>> 
>> Encoding-wise, the last array looks like this:
>> 
>> 
>> 
>>      83                  # array(3)
>> 
>>         a2               # map(2)
>> 
>>            e1            # primitive(1)
>> 
>>            63            # text(3)
>> 
>>               6f6e65     # "one"
>> 
>>            e0            # primitive(0)
>> 
>>            01            # unsigned(1)
>> 
>>         a2               # map(2)
>> 
>>            e1            # primitive(1)
>> 
>>            63            # text(3)
>> 
>>               74776f     # "two"
>> 
>>            e0            # primitive(0)
>> 
>>            02            # unsigned(2)
>> 
>>         a2               # map(2)
>> 
>>            e1            # primitive(1)
>> 
>>            65            # text(5)
>> 
>>               7468726565 # "three"
>> 
>>            e0            # primitive(0)
>> 
>>            03            # unsigned(3)
>> 
>> 
>> 
>> So the overhead here is one map head and two simple values per row.
>> 
>> (Of course, that assumes that one-byte simple values are still available in the greater context this is in.)
>> 
>> 
>> 
>> Even with a form of circumfix compression (e.g., mapping tables with parameters [1]), this is hard to beat encoding wise.
>> 
>> The record proposal as is takes four bytes per row (1+2 tag, 1 array).
>> 
>> This can be optimized significantly further only by amortizing the tag over more than one row, as my “CSV style” does, but that requires homogeneity.
>> 
>> 
>> 
>> Grüße, Carsten
>> 
>> 
>> 
>> [1]: https://datatracker.ietf.org/doc/draft-bormann-lpwan-cbor-template/
>> 
>> 
>> 
>> 
> 
> _______________________________________________
> CBOR mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/cbor