Re: [netconf] [core] YANG encoding in CBOR

[Andy Bierman <[e-mail]>]

On Wed, Mar 27, 2019 at 1:40 AM Ladislav Lhotka <[e-mail]> wrote:

> On Wed, 2019-03-27 at 07:16 +0100, Juergen Schoenwaelder wrote:
> > Hi,
> >
> > a union can be formed by using member types that are imported and not
> > under change control of a single author/organization and ideally this
> > should work without complex coordination of name and number spaces.
> > Duplicate enum/bits values are legal in YANG today so an encoding has
> > to deal with this aspect of life.
> >
> > A robust fix to all these problems will be to tag the type members in
> > order to discriminate the values in the encodings. This, however, will
> > take some time to specify and we will need to preserve backwards
> > compatibility with unions without a tag (but compilers can encourage
> > people to add tags whenever modules are updated).
>
> I already opened a new issue for this in yang-next:
>
> https://github.com/netmod-wg/yang-next/issues/72
>
>
We already explored the solution of giving the member types numbers
so they could be used in CBOR but this was rejected because it is so
complex to implement.

Consider when union is within union, and the types are named types from
other modules. Union types can be legally updated in new versions of the
module,
but the position assignments for SID can never change.

Even without this complexity this solution would cause the encoder/decoder
to
be very schema-aware.

BTW, creating SIDs for enums and bits will break if the server uses
'deviate replace type'.
There is no way to number the deviated type since this is server-specific
not part of the original module
and the deviation is not actually a data-def-stmt.


Lada
>
>
Andy


> >
> > /js
> >
> > On Wed, Mar 27, 2019 at 01:12:52AM +0000, Michel Veillette wrote:
> > > Hi Ladislav
> > >
> > > If I summarize this issue of multiple enumerations or bits in a union,
> this
> > > problem can be solve by the following approaches:
> > >
> > > - To not allows these duplicate values or positions to happen in YANG
> > > - To encode enumerations and bits as string (in all unions or only when
> > > multiple enumerations or bits are defined)
> > > - To encode enumerations and bits as SID (in all unions or only when
> > > multiple enumerations or bits are defined)
> > > - To encode enumerations and bits as delta between the value SID and
> the
> > > leaf SID (in all unions or only when multiple enumerations or bits are
> > > defined)
> > >
> > > In this email, I will like to focus on the first option, fixing the
> problem
> > > directly in YANG instead of fixing the consequences.
> > >
> > > Without any changes in YANG, a union with multiple enumeration or bits
> can
> > > be constructed without value or position overlaps.
> > > For example:
> > >
> > >   leaf multiple-enumerations-test-1 {
> > >     type union {
> > >       type enumeration {
> > >         enum "Monday" { value 0; }
> > >         enum "Tuesday" { value 1; }
> > >         enum "Wednesday" { value 2; }
> > >         enum "Thursday" { value 3; }
> > >         enum "Friday" { value 4; }
> > >
> > >       }
> > >       type enumeration {
> > >         enum "Saturday" { value 5; }
> > >         enum "Sunday" { value 6; }
> > >       }
> > >     }
> > >   }
> > >
> > >   leaf multiple-bits-test-1 {
> > >     type union {
> > >       type bits {
> > >         bit  "Monday" { position  0; }
> > >         bit "Tuesday" { position  1; }
> > >         bit "Wednesday" { position  2; }
> > >         bit "Thursday" { position  3; }
> > >         bit "Friday" { position  4; }
> > >
> > >       }
> > >       type bits {
> > >         bit "Saturday" { position 5; }
> > >         bit "Sunday" { position 6; }
> > >       }
> > >     }
> > >   }
> > >
> > > When using already defined typedef, avoiding overlap is less obvious
> without
> > > some help.
> > > To help building unions with already defined typedefs, I propose to
> > > introduce two extensions.
> > >
> > >   extension value-offset {
> > >     argument offset {
> > >       yin-element true;
> > >     }
> > >     description
> > >       "Offset added to each enum value of the associated enumeration.";
> > >   }
> > >
> > >   extension position-offset {
> > >     argument offset {
> > >       yin-element true;
> > >     }
> > >     description
> > >       "Offset value added to each bit position of the associated
> bits.";
> > >   }
> > >
> > > The value-offset extension can be used as follow:
> > >
> > >     type enumeration {
> > >       enum "Monday";
> > >       enum "Tuesday";
> > >       enum "Wednesday";
> > >       enum "Thursday";
> > >       enum "Friday";
> > >     }
> > >   }
> > >
> > >   typedef weekend {
> > >     type enumeration {
> > >       enum "Saturday";
> > >       enum "Sunday";
> > >     }
> > >   }
> > >
> > >   leaf multiple-enumerations-test-3 {
> > >     type union {
> > >       type weekdays;
> > >       type weekend {
> > >         ext:value-offset 5;
> > >       }
> > >     }
> > >   }
> > >
> > > The position-offset extension can be used as follow:
> > >
> > >   typedef weekdays-flags {
> > >     type bits {
> > >       bit "Monday";
> > >       bit "Tuesday";
> > >       bit "Wednesday";
> > >       bit "Thursday";
> > >       bit "Friday";
> > >     }
> > >   }
> > >
> > >   typedef weekend-flags {
> > >     type bits {
> > >       bit "Saturday";
> > >       bit "Sunday";
> > >     }
> > >   }
> > >
> > >   leaf multiple-bits-test-3 {
> > >     type union {
> > >       type weekdays-flags;
> > >       type weekend-flags {
> > >         ext:position-offset 5;
> > >       }
> > >     }
> > >   }
> > >
> > > The yang file in attachment show different examples based on this
> approach.
> > > This module have been validated using
> http://www.yangvalidator.com/validator
> > >
> > > If this approach is accepted, tools like pyang should be updated to
> produce
> > > an error if multiple enumerations or bits are defined with value or
> position
> > > overleaps.
> > >
> > > Please comment,
> > > Michel
> > >
> > > -----Original Message-----
> > > From: Ladislav Lhotka <[e-mail]>
> > > Sent: Monday, March 25, 2019 4:07 AM
> > > To: Juergen Schoenwaelder <[e-mail]>;
> Carsten
> > > Bormann <[e-mail]>
> > > Cc: Michel Veillette <[e-mail]>;
> [e-mail];
> > > [e-mail]
> > > Subject: Re: [netconf] YANG encoding in CBOR
> > >
> > > Juergen Schoenwaelder <[e-mail]> writes:
> > >
> > > > I think we need to look at the whole picture and in which direction
> we
> > > > want to go. In the longer term, I would prefer a solution where the
> > > > values of a union are discriminated. The current XML encoding
> > > > behaviour of 'first match wins' is fragile (for example, if someone
> > > > adds an enum to a type, the interpretation of data can change).
> > > >
> > > > Look at this:
> > > >
> > > > typedef bar {
> > > >   type union {
> > > >     type enumeration { enum "1"; value 2; enum "2"; value 1; }
> > > >     type uint8;
> > > >   }
> > > > }
> > > >
> > > > We have some encodings that send the string representations of the
> > > > values and some encodings that prefer to send numeric
> representations
> > > > where possible. In order to have a robust solution, encodings should
> > > > likely indicate to which type the value belongs.
> > >
> > > Perhaps the easiest way would be to use (optional) annotation that
> > > specifies, using an ordinal number, which of the member types is used
> for
> > > the particular instance. But since there can be unions inside unions,
> a list
> > > of numbers would be needed in general.
> > >
> > > Lada
> > >
> > > > /js
> > > >
> > > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > > Well, if that is a problem, we can go for a longer representation
> within
> > > > > unions (section 6.12).  Theoretically, we could do that only of
> there is
> > > > > more than one enum in the union type (so things stay efficient if
> there
> > > > > is only one), but that might pose difficulties with model
> evolution.
> > > > >
> > > > > Going for a string representation repeats the feature of XML YANG
> (which
> > > > > was ported over to JSON YANG):
> > > > >
> > > > > typedef foo {
> > > > >   type union {
> > > > >     type enumeration {
> > > > >       enum red { value 1; }
> > > > >       enum breen { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >     type enumeration {
> > > > >       enum tacks { value 1; }
> > > > >       enum nails { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >   }
> > > > > }
> > > > >
> > > > > If you use “glue”, you don’t know which of the enumerations are
> being
> > > > > used.
> > > > >
> > > > > Using SIDs, we can do better.
> > > > >
> > > > > So what do we have to do to get the SID tool to allocate SIDs for
> enum
> > > > > values?
> > > > >
> > > > > We could then define the CBOR tag for enums in unions to take the
> usual
> > > > > SID difference (delta relative to the environment, I’d think), not
> an
> > > > > integer value.
> > > > >
> > > > > Several of us are at the hackathon and could make something happen
> today
> > > > > and tomorrow.
> > > > >
> > > > > Grüße, Carsten
> > > > >
> > > > >
> > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <
> [e-mail]>
> > > > > > wrote:
> > > > > >
> > > > > > I guess that the concern is that this introduces more variation
> in how
> > > > > > data is interpreted between the different XML/JSON/CBOR
> encodings.
> > > > > >
> > > > > > E.g. if someone switched from XML to CBOR, suddenly the
> configuration
> > > > > > or state data may have a different meaning.
> > > > > >
> > > > > > Thanks,
> > > > > > Rob
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > > Sent: 22 March 2019 16:08
> > > > > > > To: Michel Veillette <[e-mail]>
> > > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>; [e-mail];
> > > > > > > [e-mail]
> > > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > >
> > > > > > > On Mar 22, 2019, at 16:45, Michel Veillette
> > > > > > > <[e-mail]>
> > > > > > > wrote:
> > > > > > > > The only potential problem I aware is when multiple
> enumerations
> > > > > > > > are part of
> > > > > > > the same union.
> > > > > > > > Value 4 from enumeration A will be encoded the same way as
> Value
> > > > > > > > 4 from
> > > > > > > enumeration B.
> > > > > > >
> > > > > > > … and that is not a problem for the XML version, because the
> > > > > > > string is being used instead of the value.  (But then if two
> > > > > > > enumerations share a string, you have the equivalent problem
> in
> > > > > > > the XML serialization.)
> > > > > > >
> > > > > > > Anyway, I haven’t seen a piece of real-world YANG that
> actually
> > > > > > > has this problem, so I would be a bit reluctant to make
> CBOR-based
> > > > > > > implementations more complex (and less efficient) so solve this
> > > > > > > (non-?)problem.
> > > > > > >
> > > > > > > Grüße, Carsten
> > > > > >
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > netconf mailing list
> > > > > [e-mail]
> > > > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > > .ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > >
> d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > >
> %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > > kvPZgr4o%3D&amp;reserved=0
> > > >
> > > > --
> > > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > > Phone: [phone]         Campus Ring 1 | 28759 Bremen |
> Germany
> > > > Fax:   [phone]         <
> > > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > > >
> > > >
> > > >
> > > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > > Well, if that is a problem, we can go for a longer representation
> within
> > > > > unions (section 6.12).  Theoretically, we could do that only of
> there is
> > > > > more than one enum in the union type (so things stay efficient if
> there
> > > > > is only one), but that might pose difficulties with model
> evolution.
> > > > >
> > > > > Going for a string representation repeats the feature of XML YANG
> (which
> > > > > was ported over to JSON YANG):
> > > > >
> > > > > typedef foo {
> > > > >   type union {
> > > > >     type enumeration {
> > > > >       enum red { value 1; }
> > > > >       enum breen { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >     type enumeration {
> > > > >       enum tacks { value 1; }
> > > > >       enum nails { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >   }
> > > > > }
> > > > >
> > > > > If you use “glue”, you don’t know which of the enumerations are
> being
> > > > > used.
> > > > >
> > > > > Using SIDs, we can do better.
> > > > >
> > > > > So what do we have to do to get the SID tool to allocate SIDs for
> enum
> > > > > values?
> > > > >
> > > > > We could then define the CBOR tag for enums in unions to take the
> usual
> > > > > SID difference (delta relative to the environment, I’d think), not
> an
> > > > > integer value.
> > > > >
> > > > > Several of us are at the hackathon and could make something happen
> today
> > > > > and tomorrow.
> > > > >
> > > > > Grüße, Carsten
> > > > >
> > > > >
> > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <
> [e-mail]>
> > > > > > wrote:
> > > > > >
> > > > > > I guess that the concern is that this introduces more variation
> in how
> > > > > > data is interpreted between the different XML/JSON/CBOR
> encodings.
> > > > > >
> > > > > > E.g. if someone switched from XML to CBOR, suddenly the
> configuration
> > > > > > or state data may have a different meaning.
> > > > > >
> > > > > > Thanks,
> > > > > > Rob
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > > Sent: 22 March 2019 16:08
> > > > > > > To: Michel Veillette <[e-mail]>
> > > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>; [e-mail];
> > > > > > > [e-mail]
> > > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > >
> > > > > > > On Mar 22, 2019, at 16:45, Michel Veillette
> > > > > > > <[e-mail]>
> > > > > > > wrote:
> > > > > > > > The only potential problem I aware is when multiple
> enumerations
> > > > > > > > are part of
> > > > > > > the same union.
> > > > > > > > Value 4 from enumeration A will be encoded the same way as
> Value
> > > > > > > > 4 from
> > > > > > > enumeration B.
> > > > > > >
> > > > > > > … and that is not a problem for the XML version, because the
> > > > > > > string is being used instead of the value.  (But then if two
> > > > > > > enumerations share a string, you have the equivalent problem
> in
> > > > > > > the XML serialization.)
> > > > > > >
> > > > > > > Anyway, I haven’t seen a piece of real-world YANG that
> actually
> > > > > > > has this problem, so I would be a bit reluctant to make
> CBOR-based
> > > > > > > implementations more complex (and less efficient) so solve this
> > > > > > > (non-?)problem.
> > > > > > >
> > > > > > > Grüße, Carsten
> > > > > >
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > netconf mailing list
> > > > > [e-mail]
> > > > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > > .ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > >
> d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > >
> %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > > kvPZgr4o%3D&amp;reserved=0
> > > >
> > > > --
> > > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > > Phone: [phone]         Campus Ring 1 | 28759 Bremen |
> Germany
> > > > Fax:   [phone]         <
> > > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > > >
> > > >
> > > > _______________________________________________
> > > > netconf mailing list
> > > > [e-mail]
> > > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > > > ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> > > >
> cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> > > >
> 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> > > > gr4o%3D&amp;reserved=0
> > >
> > > --
> > > Ladislav Lhotka
> > > Head, CZ.NIC Labs
> > > PGP Key ID: 0xB8F92B08A9F76C67
> >
> >
> --
> Ladislav Lhotka
> Head, CZ.NIC Labs
> PGP Key ID: 0xB8F92B08A9F76C67
>
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
>