Re: [netconf] YANG encoding in CBOR

[Ladislav Lhotka <[e-mail]>]

On Wed, 2019-03-27 at 07:16 +0100, Juergen Schoenwaelder wrote:
> Hi,
> 
> a union can be formed by using member types that are imported and not
> under change control of a single author/organization and ideally this
> should work without complex coordination of name and number spaces.
> Duplicate enum/bits values are legal in YANG today so an encoding has
> to deal with this aspect of life.
> 
> A robust fix to all these problems will be to tag the type members in
> order to discriminate the values in the encodings. This, however, will
> take some time to specify and we will need to preserve backwards
> compatibility with unions without a tag (but compilers can encourage
> people to add tags whenever modules are updated).

I already opened a new issue for this in yang-next:

https://github.com/netmod-wg/yang-next/issues/72

Lada

> 
> /js
> 
> On Wed, Mar 27, 2019 at 01:12:52AM +0000, Michel Veillette wrote:
> > Hi Ladislav
> > 
> > If I summarize this issue of multiple enumerations or bits in a union, this
> > problem can be solve by the following approaches:
> > 
> > - To not allows these duplicate values or positions to happen in YANG
> > - To encode enumerations and bits as string (in all unions or only when
> > multiple enumerations or bits are defined)
> > - To encode enumerations and bits as SID (in all unions or only when
> > multiple enumerations or bits are defined)
> > - To encode enumerations and bits as delta between the value SID and the
> > leaf SID (in all unions or only when multiple enumerations or bits are
> > defined)
> > 
> > In this email, I will like to focus on the first option, fixing the problem
> > directly in YANG instead of fixing the consequences.
> > 
> > Without any changes in YANG, a union with multiple enumeration or bits can
> > be constructed without value or position overlaps.
> > For example:
> > 
> >   leaf multiple-enumerations-test-1 {
> >     type union {
> >       type enumeration {
> >         enum "Monday" { value 0; }
> >         enum "Tuesday" { value 1; }
> >         enum "Wednesday" { value 2; }
> >         enum "Thursday" { value 3; }
> >         enum "Friday" { value 4; }
> > 
> >       }
> >       type enumeration {
> >         enum "Saturday" { value 5; }
> >         enum "Sunday" { value 6; }
> >       }
> >     }
> >   }
> > 
> >   leaf multiple-bits-test-1 {
> >     type union {
> >       type bits {
> >         bit  "Monday" { position  0; }
> >         bit "Tuesday" { position  1; }
> >         bit "Wednesday" { position  2; }
> >         bit "Thursday" { position  3; }
> >         bit "Friday" { position  4; }
> > 
> >       }
> >       type bits {
> >         bit "Saturday" { position 5; }
> >         bit "Sunday" { position 6; }
> >       }
> >     }
> >   }
> > 
> > When using already defined typedef, avoiding overlap is less obvious without
> > some help.
> > To help building unions with already defined typedefs, I propose to
> > introduce two extensions. 
> > 
> >   extension value-offset {
> >     argument offset {
> >       yin-element true;
> >     }
> >     description
> >       "Offset added to each enum value of the associated enumeration.";
> >   }
> >   
> >   extension position-offset {
> >     argument offset {
> >       yin-element true;
> >     }
> >     description
> >       "Offset value added to each bit position of the associated bits.";
> >   }
> > 
> > The value-offset extension can be used as follow:
> > 
> >     type enumeration {
> >       enum "Monday";
> >       enum "Tuesday";
> >       enum "Wednesday";
> >       enum "Thursday";
> >       enum "Friday";
> >     }
> >   }
> > 
> >   typedef weekend {
> >     type enumeration {
> >       enum "Saturday";
> >       enum "Sunday";
> >     }
> >   }
> >   
> >   leaf multiple-enumerations-test-3 {
> >     type union {
> >       type weekdays;
> >       type weekend {
> >         ext:value-offset 5;
> >       }
> >     }
> >   }
> > 
> > The position-offset extension can be used as follow:
> > 
> >   typedef weekdays-flags {
> >     type bits {
> >       bit "Monday";
> >       bit "Tuesday";
> >       bit "Wednesday";
> >       bit "Thursday";
> >       bit "Friday";
> >     }
> >   }
> > 
> >   typedef weekend-flags {
> >     type bits {
> >       bit "Saturday";
> >       bit "Sunday";
> >     }
> >   }
> >   
> >   leaf multiple-bits-test-3 {
> >     type union {
> >       type weekdays-flags;
> >       type weekend-flags {
> >         ext:position-offset 5;
> >       }
> >     }
> >   }
> > 
> > The yang file in attachment show different examples based on this approach.
> > This module have been validated using http://www.yangvalidator.com/validator
> >  
> > If this approach is accepted, tools like pyang should be updated to produce
> > an error if multiple enumerations or bits are defined with value or position
> > overleaps.
> > 
> > Please comment,
> > Michel
> > 
> > -----Original Message-----
> > From: Ladislav Lhotka <[e-mail]> 
> > Sent: Monday, March 25, 2019 4:07 AM
> > To: Juergen Schoenwaelder <[e-mail]>; Carsten
> > Bormann <[e-mail]>
> > Cc: Michel Veillette <[e-mail]>; [e-mail]; 
> > [e-mail]
> > Subject: Re: [netconf] YANG encoding in CBOR
> > 
> > Juergen Schoenwaelder <[e-mail]> writes:
> > 
> > > I think we need to look at the whole picture and in which direction we 
> > > want to go. In the longer term, I would prefer a solution where the 
> > > values of a union are discriminated. The current XML encoding 
> > > behaviour of 'first match wins' is fragile (for example, if someone 
> > > adds an enum to a type, the interpretation of data can change).
> > > 
> > > Look at this:
> > > 
> > > typedef bar {
> > >   type union {
> > >     type enumeration { enum "1"; value 2; enum "2"; value 1; }
> > >     type uint8;
> > >   }
> > > }
> > > 
> > > We have some encodings that send the string representations of the 
> > > values and some encodings that prefer to send numeric representations 
> > > where possible. In order to have a robust solution, encodings should 
> > > likely indicate to which type the value belongs.
> > 
> > Perhaps the easiest way would be to use (optional) annotation that
> > specifies, using an ordinal number, which of the member types is used for
> > the particular instance. But since there can be unions inside unions, a list
> > of numbers would be needed in general.
> > 
> > Lada
> > 
> > > /js
> > > 
> > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > Well, if that is a problem, we can go for a longer representation within
> > > > unions (section 6.12).  Theoretically, we could do that only of there is
> > > > more than one enum in the union type (so things stay efficient if there
> > > > is only one), but that might pose difficulties with model evolution.
> > > > 
> > > > Going for a string representation repeats the feature of XML YANG (which
> > > > was ported over to JSON YANG):
> > > > 
> > > > typedef foo {
> > > >   type union {
> > > >     type enumeration {
> > > >       enum red { value 1; }
> > > >       enum breen { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >     type enumeration {
> > > >       enum tacks { value 1; }
> > > >       enum nails { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >   }
> > > > }
> > > > 
> > > > If you use “glue”, you don’t know which of the enumerations are being
> > > > used.
> > > > 
> > > > Using SIDs, we can do better.
> > > > 
> > > > So what do we have to do to get the SID tool to allocate SIDs for enum
> > > > values?
> > > > 
> > > > We could then define the CBOR tag for enums in unions to take the usual
> > > > SID difference (delta relative to the environment, I’d think), not an
> > > > integer value.
> > > > 
> > > > Several of us are at the hackathon and could make something happen today
> > > > and tomorrow.
> > > > 
> > > > Grüße, Carsten
> > > > 
> > > > 
> > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[e-mail]>
> > > > > wrote:
> > > > > 
> > > > > I guess that the concern is that this introduces more variation in how
> > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > 
> > > > > E.g. if someone switched from XML to CBOR, suddenly the configuration
> > > > > or state data may have a different meaning.
> > > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > Sent: 22 March 2019 16:08
> > > > > > To: Michel Veillette <[e-mail]>
> > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>; [e-mail]; 
> > > > > > [e-mail]
> > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > 
> > > > > > On Mar 22, 2019, at 16:45, Michel Veillette 
> > > > > > <[e-mail]>
> > > > > > wrote:
> > > > > > > The only potential problem I aware is when multiple enumerations 
> > > > > > > are part of
> > > > > > the same union.
> > > > > > > Value 4 from enumeration A will be encoded the same way as Value 
> > > > > > > 4 from
> > > > > > enumeration B.
> > > > > > 
> > > > > > … and that is not a problem for the XML version, because the 
> > > > > > string is being used instead of the value.  (But then if two 
> > > > > > enumerations share a string, you have the equivalent problem in 
> > > > > > the XML serialization.)
> > > > > > 
> > > > > > Anyway, I haven’t seen a piece of real-world YANG that actually 
> > > > > > has this problem, so I would be a bit reluctant to make CBOR-based 
> > > > > > implementations more complex (and less efficient) so solve this
> > > > > > (non-?)problem.
> > > > > > 
> > > > > > Grüße, Carsten
> > > > > 
> > > > > 
> > > > 
> > > > _______________________________________________
> > > > netconf mailing list
> > > > [e-mail]
> > > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > kvPZgr4o%3D&amp;reserved=0
> > > 
> > > -- 
> > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
> > > Fax:   [phone]         <
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > >
> > > 
> > > 
> > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > Well, if that is a problem, we can go for a longer representation within
> > > > unions (section 6.12).  Theoretically, we could do that only of there is
> > > > more than one enum in the union type (so things stay efficient if there
> > > > is only one), but that might pose difficulties with model evolution.
> > > > 
> > > > Going for a string representation repeats the feature of XML YANG (which
> > > > was ported over to JSON YANG):
> > > > 
> > > > typedef foo {
> > > >   type union {
> > > >     type enumeration {
> > > >       enum red { value 1; }
> > > >       enum breen { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >     type enumeration {
> > > >       enum tacks { value 1; }
> > > >       enum nails { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >   }
> > > > }
> > > > 
> > > > If you use “glue”, you don’t know which of the enumerations are being
> > > > used.
> > > > 
> > > > Using SIDs, we can do better.
> > > > 
> > > > So what do we have to do to get the SID tool to allocate SIDs for enum
> > > > values?
> > > > 
> > > > We could then define the CBOR tag for enums in unions to take the usual
> > > > SID difference (delta relative to the environment, I’d think), not an
> > > > integer value.
> > > > 
> > > > Several of us are at the hackathon and could make something happen today
> > > > and tomorrow.
> > > > 
> > > > Grüße, Carsten
> > > > 
> > > > 
> > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[e-mail]>
> > > > > wrote:
> > > > > 
> > > > > I guess that the concern is that this introduces more variation in how
> > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > 
> > > > > E.g. if someone switched from XML to CBOR, suddenly the configuration
> > > > > or state data may have a different meaning.
> > > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > Sent: 22 March 2019 16:08
> > > > > > To: Michel Veillette <[e-mail]>
> > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>; [e-mail]; 
> > > > > > [e-mail]
> > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > 
> > > > > > On Mar 22, 2019, at 16:45, Michel Veillette 
> > > > > > <[e-mail]>
> > > > > > wrote:
> > > > > > > The only potential problem I aware is when multiple enumerations 
> > > > > > > are part of
> > > > > > the same union.
> > > > > > > Value 4 from enumeration A will be encoded the same way as Value 
> > > > > > > 4 from
> > > > > > enumeration B.
> > > > > > 
> > > > > > … and that is not a problem for the XML version, because the 
> > > > > > string is being used instead of the value.  (But then if two 
> > > > > > enumerations share a string, you have the equivalent problem in 
> > > > > > the XML serialization.)
> > > > > > 
> > > > > > Anyway, I haven’t seen a piece of real-world YANG that actually 
> > > > > > has this problem, so I would be a bit reluctant to make CBOR-based 
> > > > > > implementations more complex (and less efficient) so solve this
> > > > > > (non-?)problem.
> > > > > > 
> > > > > > Grüße, Carsten
> > > > > 
> > > > > 
> > > > 
> > > > _______________________________________________
> > > > netconf mailing list
> > > > [e-mail]
> > > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > kvPZgr4o%3D&amp;reserved=0
> > > 
> > > -- 
> > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
> > > Fax:   [phone]         <
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > >
> > > 
> > > _______________________________________________
> > > netconf mailing list
> > > [e-mail]
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > > ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> > > cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> > > 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> > > gr4o%3D&amp;reserved=0
> > 
> > --
> > Ladislav Lhotka
> > Head, CZ.NIC Labs
> > PGP Key ID: 0xB8F92B08A9F76C67
> 
> 
-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67