Re: [Cbor] Record proposal

[Emile Cormier <[e-mail]>]

I'd like to point out that this proposal as it's currently written will
result in loss of information with CBOR decoders that don't understand the
tags, or when the data elements are transcoded to another protocol such as
JSON. This is due to the tags carrying information (record IDs) that the
application needs in order to make sense of the received payload. The
proposed tags do more than provide "semantic meaning"; they also carry
information.

This practice of bundling numeric information into the tags is making it so
that these CBOR items are not interoperable when transcoded to JSON (think
web browser), or when a CBOR decoder doesn't understand the tags and
doesn't propagate them up to the application.


On Mon, Nov 29, 2021 at 9:35 AM Kris Zyp <[e-mail]> wrote:

> To follow up, I did update the proposal/spec to use the tag id range
> of 57342-57599. Let me know if you would prefer different ids.
> Thanks,
> Kris
>
> On Wed, Nov 17, 2021 at 10:52 PM Kris Zyp <[e-mail]> wrote:
> >
> > That sounds reasonable to me.
> >
> > To clarify a little bit of the rationale for this:
> > The purpose of a global registry, as I understand it, is to define the
> > global tag ids that need to be known by independent encoders and
> > decoders prior to their encoding and decoding interaction. However,
> > ids that can be communicated within a message for the scope of that
> > message, don't require global allocation, if they are only used within
> > a specified scope. This is just like in a programming language: a
> > variable from an inner block may shadow another variable from an outer
> > block or a global scope temporarily within its block without affecting
> > the outer or global scope (this doesn't require any coordination with
> > a registry of globals). And a decoder that really implements this
> > proposal would, by definition, be implementing this behavior of
> > temporary tag reassignment/shadowing, if it really is a true record
> > implementation (i.e. a real record implementation wouldn't see
> > gobbledygook if it is conformant).
> >
> > Also allowing dynamic/temporary allocation permits potential use of
> > shorter/more efficient tag ids based on known use of other tags, and
> > in the case of record references, these may occur much more frequently
> > in a data item/document compared record definitions, and be much more
> > sensitive to size differences. Furthermore, this also sidesteps issues
> > with having to figure out the appropriate size of a range of ids to
> > globally allocate. How many ids should be allocated? In terms of
> > percentage of available space, registering/using 256 ids from the tag
> > 1+2 range seems like it is effectively equivalent as registering 1 id
> > from the tag 1+1 range (and tag 1+4 would definitely be undesirable
> > due to the high frequency and sensitivity to size). And I have had
> > users of my implementations say they need a few hundred structures to
> > be defined and referenceable, so even 256 may be somewhat limiting to
> > some users.
> >
> > But that being said, if you are thinking that it may be too onerous
> > for (record) implementators to support this type of potentially
> > dynamic/temporary tag reassignment with possibly conflicting ids (as
> > opposed to using a more stable table of tags or tag ranges), I can
> > understand that concern. What size of chunk of tag ids (above 32768)
> > do you think would be appropriate to allocate? Would a chunk of 256 be
> > reasonable?
> >
> > Anyway, thank you for your help and feedback!
> > Thanks,
> > Kris
> >
> >
> >
> > On Wed, Nov 17, 2021 at 1:52 PM Carsten Bormann <[e-mail]> wrote:
> > >
> > > Hi Kris,
> > >
> > > thank you for the update.
> > > I still need to take a closer look, but I have one immediate reaction:
> > > You probably shouldn’t induce implementations to “hijack” a tag number.
> > > The handling of a tag number that has actually been allocated for
> something else may be deeply embedded into the CBOR decoder; a record
> implementation then would only see gobbledygook.
> > > So I think we should allocate some space for tag-allocated tags — I
> don’t see a big problem with allocating a chunk above 32768 just for the
> record proposal.
> > >
> > > Grüße, Carsten
> > >
> > >
> > > > On 2021-11-17, at 16:16, Kris Zyp <[e-mail]> wrote:
> > > >
> > > > I have updated my tag registration proposal/submission at
> > > > https://github.com/kriszyp/cbor-records to use 1+2 tag entries,
> which
> > > > hopefully makes this a much less intrusive registry entry. I have
> also
> > > > updated the proposed tag definitions to also support up-front
> > > > declaration of a set of record structure definitions for a data item
> > > > ("around" that data item, like the packed approach, as you
> suggested),
> > > > in addition to the current inline record definitions (which can be
> > > > scoped with record definitions and should follow a well-specified
> > > > order for when they can be referenced by subsequent references). I
> > > > hope this offers flexibility for encoders that have all structures
> > > > known a-priori and streaming encoders (that do not), while still
> > > > maintaining nearly the same mechanics for decoders. Let me know if
> you
> > > > think this looks reasonable.
> > > > Thank you!
> > > > Kris
> > > >
> > > > On Thu, Nov 11, 2021 at 9:15 PM Kris Zyp <[e-mail]> wrote:
> > > >>
> > > >>> Actually, stats would be very interesting.
> > > >>> I was assuming that the 1+1 setup comes with a number of 1+2
> referencing records, the hit from going to 1+2 there as well would be
> relatively insignificant.
> > > >>> Number are better than assumptions!
> > > >>
> > > >> You are definitely right, using 1+2 tag for defining records is
> pretty
> > > >> insignificant (around a quarter of percent in my tests). Anyway, I
> put
> > > >> together some tests comparing CBOR packed, record structures, and
> > > >> combinations with a couple test data structures, with my library,
> for
> > > >> the sake of further comparisons:
> > > >> https://gist.github.com/kriszyp/b623b85d2dc25ac9e3b07d8f39df9307
> > > >>
> > > >> Anyway, seeing this, I am happy to go ahead and update my proposal
> to
> > > >> use a 1+2 tag for defining records. And thinking about this, I don't
> > > >> think my proposal necessarily even needs to mandate the tag ids used
> > > >> for referencing records since those are dynamically assigned and
> > > >> explicitly specified by the encoder itself (encoder obviously must
> not
> > > >> conflict and use tag ids that will be used for other purposes),
> albeit
> > > >> can encourage a certain range (presumably from the first-come
> > > >> first-serve range).
> > > >>
> > > >> Do you have any preference for a tag id to use? It looks like 279 is
> > > >> the next in the contiguous block, but sounds like choosing aesthetic
> > > >> characters is the new preference (29299/"rs" perhaps).
> > > >>
> > > >> Anyway, thanks again for the helpful feedback, really appreciate it!
> > > >>
> > > >> Thanks,
> > > >> Kris
> > > >
> > > > _______________________________________________
> > > > CBOR mailing list
> > > > [e-mail]
> > > > https://www.ietf.org/mailman/listinfo/cbor
> > >
>
> _______________________________________________
> CBOR mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/cbor
>