Re: [netconf] [core] YANG encoding in CBOR

[Ladislav Lhotka <[e-mail]>]

Andy Bierman píše v St 27. 03. 2019 v 09:13 -0700:
> 
> 
> On Wed, Mar 27, 2019 at 1:40 AM Ladislav Lhotka <[e-mail]> wrote:
> > On Wed, 2019-03-27 at 07:16 +0100, Juergen Schoenwaelder wrote:
> > > Hi,
> > > 
> > > a union can be formed by using member types that are imported and not
> > > under change control of a single author/organization and ideally this
> > > should work without complex coordination of name and number spaces.
> > > Duplicate enum/bits values are legal in YANG today so an encoding has
> > > to deal with this aspect of life.
> > > 
> > > A robust fix to all these problems will be to tag the type members in
> > > order to discriminate the values in the encodings. This, however, will
> > > take some time to specify and we will need to preserve backwards
> > > compatibility with unions without a tag (but compilers can encourage
> > > people to add tags whenever modules are updated).
> > 
> > I already opened a new issue for this in yang-next:
> > 
> > https://github.com/netmod-wg/yang-next/issues/72
> > 
> 
> We already explored the solution of giving the member types numbers
> so they could be used in CBOR but this was rejected because it is so complex
> to implement.

I don't think it is too complex. Get the type in the schema, get the member,
that's all. It is much easier and more robust than the current algorithm.

> 
> Consider when union is within union, and the types are named types from

Union within union is no problem: it jut requires a sequence of numbers.

> other modules. Union types can be legally updated in new versions of the
> module,

Again, I don't agree. Sec. 11 in RFC 7950 says:

   o  A "type" statement may be replaced with another "type" statement
      that does not change the syntax or semantics of the type.
      ...

Adding a new member clearly changes the semantics of the type, if not syntax.

> but the position assignments for SID can never change.

The annotation would also help resolve the differences between JSON and XML. SID
is only available in CBOR.

> 
> Even without this complexity this solution would cause the encoder/decoder to
> be very schema-aware.

The current method of resolving unions is totally schema-aware (somewhat less so
in JSON).

Lada

>  
> 
> BTW, creating SIDs for enums and bits will break if the server uses 'deviate
> replace type'.
> There is no way to number the deviated type since this is server-specific not
> part of the original module
> and the deviation is not actually a data-def-stmt.
> 
> 
> > Lada
> > 
> 
> Andy
>  
> > > 
> > > /js
> > > 
> > > On Wed, Mar 27, 2019 at 01:12:52AM +0000, Michel Veillette wrote:
> > > > Hi Ladislav
> > > > 
> > > > If I summarize this issue of multiple enumerations or bits in a union,
> > this
> > > > problem can be solve by the following approaches:
> > > > 
> > > > - To not allows these duplicate values or positions to happen in YANG
> > > > - To encode enumerations and bits as string (in all unions or only when
> > > > multiple enumerations or bits are defined)
> > > > - To encode enumerations and bits as SID (in all unions or only when
> > > > multiple enumerations or bits are defined)
> > > > - To encode enumerations and bits as delta between the value SID and the
> > > > leaf SID (in all unions or only when multiple enumerations or bits are
> > > > defined)
> > > > 
> > > > In this email, I will like to focus on the first option, fixing the
> > problem
> > > > directly in YANG instead of fixing the consequences.
> > > > 
> > > > Without any changes in YANG, a union with multiple enumeration or bits
> > can
> > > > be constructed without value or position overlaps.
> > > > For example:
> > > > 
> > > >   leaf multiple-enumerations-test-1 {
> > > >     type union {
> > > >       type enumeration {
> > > >         enum "Monday" { value 0; }
> > > >         enum "Tuesday" { value 1; }
> > > >         enum "Wednesday" { value 2; }
> > > >         enum "Thursday" { value 3; }
> > > >         enum "Friday" { value 4; }
> > > > 
> > > >       }
> > > >       type enumeration {
> > > >         enum "Saturday" { value 5; }
> > > >         enum "Sunday" { value 6; }
> > > >       }
> > > >     }
> > > >   }
> > > > 
> > > >   leaf multiple-bits-test-1 {
> > > >     type union {
> > > >       type bits {
> > > >         bit  "Monday" { position  0; }
> > > >         bit "Tuesday" { position  1; }
> > > >         bit "Wednesday" { position  2; }
> > > >         bit "Thursday" { position  3; }
> > > >         bit "Friday" { position  4; }
> > > > 
> > > >       }
> > > >       type bits {
> > > >         bit "Saturday" { position 5; }
> > > >         bit "Sunday" { position 6; }
> > > >       }
> > > >     }
> > > >   }
> > > > 
> > > > When using already defined typedef, avoiding overlap is less obvious
> > without
> > > > some help.
> > > > To help building unions with already defined typedefs, I propose to
> > > > introduce two extensions. 
> > > > 
> > > >   extension value-offset {
> > > >     argument offset {
> > > >       yin-element true;
> > > >     }
> > > >     description
> > > >       "Offset added to each enum value of the associated enumeration.";
> > > >   }
> > > >   
> > > >   extension position-offset {
> > > >     argument offset {
> > > >       yin-element true;
> > > >     }
> > > >     description
> > > >       "Offset value added to each bit position of the associated bits.";
> > > >   }
> > > > 
> > > > The value-offset extension can be used as follow:
> > > > 
> > > >     type enumeration {
> > > >       enum "Monday";
> > > >       enum "Tuesday";
> > > >       enum "Wednesday";
> > > >       enum "Thursday";
> > > >       enum "Friday";
> > > >     }
> > > >   }
> > > > 
> > > >   typedef weekend {
> > > >     type enumeration {
> > > >       enum "Saturday";
> > > >       enum "Sunday";
> > > >     }
> > > >   }
> > > >   
> > > >   leaf multiple-enumerations-test-3 {
> > > >     type union {
> > > >       type weekdays;
> > > >       type weekend {
> > > >         ext:value-offset 5;
> > > >       }
> > > >     }
> > > >   }
> > > > 
> > > > The position-offset extension can be used as follow:
> > > > 
> > > >   typedef weekdays-flags {
> > > >     type bits {
> > > >       bit "Monday";
> > > >       bit "Tuesday";
> > > >       bit "Wednesday";
> > > >       bit "Thursday";
> > > >       bit "Friday";
> > > >     }
> > > >   }
> > > > 
> > > >   typedef weekend-flags {
> > > >     type bits {
> > > >       bit "Saturday";
> > > >       bit "Sunday";
> > > >     }
> > > >   }
> > > >   
> > > >   leaf multiple-bits-test-3 {
> > > >     type union {
> > > >       type weekdays-flags;
> > > >       type weekend-flags {
> > > >         ext:position-offset 5;
> > > >       }
> > > >     }
> > > >   }
> > > > 
> > > > The yang file in attachment show different examples based on this
> > approach.
> > > > This module have been validated using 
> > http://www.yangvalidator.com/validator
> > > >  
> > > > If this approach is accepted, tools like pyang should be updated to
> > produce
> > > > an error if multiple enumerations or bits are defined with value or
> > position
> > > > overleaps.
> > > > 
> > > > Please comment,
> > > > Michel
> > > > 
> > > > -----Original Message-----
> > > > From: Ladislav Lhotka <[e-mail]> 
> > > > Sent: Monday, March 25, 2019 4:07 AM
> > > > To: Juergen Schoenwaelder <[e-mail]>;
> > Carsten
> > > > Bormann <[e-mail]>
> > > > Cc: Michel Veillette <[e-mail]>; [e-mail];
> >  
> > > > [e-mail]
> > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > 
> > > > Juergen Schoenwaelder <[e-mail]> writes:
> > > > 
> > > > > I think we need to look at the whole picture and in which direction
> > we 
> > > > > want to go. In the longer term, I would prefer a solution where the 
> > > > > values of a union are discriminated. The current XML encoding 
> > > > > behaviour of 'first match wins' is fragile (for example, if someone 
> > > > > adds an enum to a type, the interpretation of data can change).
> > > > > 
> > > > > Look at this:
> > > > > 
> > > > > typedef bar {
> > > > >   type union {
> > > > >     type enumeration { enum "1"; value 2; enum "2"; value 1; }
> > > > >     type uint8;
> > > > >   }
> > > > > }
> > > > > 
> > > > > We have some encodings that send the string representations of the 
> > > > > values and some encodings that prefer to send numeric representations 
> > > > > where possible. In order to have a robust solution, encodings should 
> > > > > likely indicate to which type the value belongs.
> > > > 
> > > > Perhaps the easiest way would be to use (optional) annotation that
> > > > specifies, using an ordinal number, which of the member types is used
> > for
> > > > the particular instance. But since there can be unions inside unions, a
> > list
> > > > of numbers would be needed in general.
> > > > 
> > > > Lada
> > > > 
> > > > > /js
> > > > > 
> > > > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > > > Well, if that is a problem, we can go for a longer representation
> > within
> > > > > > unions (section 6.12).  Theoretically, we could do that only of
> > there is
> > > > > > more than one enum in the union type (so things stay efficient if
> > there
> > > > > > is only one), but that might pose difficulties with model evolution.
> > > > > > 
> > > > > > Going for a string representation repeats the feature of XML YANG
> > (which
> > > > > > was ported over to JSON YANG):
> > > > > > 
> > > > > > typedef foo {
> > > > > >   type union {
> > > > > >     type enumeration {
> > > > > >       enum red { value 1; }
> > > > > >       enum breen { value 2; }
> > > > > >       enum glue { value 3; }
> > > > > >     }
> > > > > >     type enumeration {
> > > > > >       enum tacks { value 1; }
> > > > > >       enum nails { value 2; }
> > > > > >       enum glue { value 3; }
> > > > > >     }
> > > > > >   }
> > > > > > }
> > > > > > 
> > > > > > If you use “glue”, you don’t know which of the enumerations are
> > being
> > > > > > used.
> > > > > > 
> > > > > > Using SIDs, we can do better.
> > > > > > 
> > > > > > So what do we have to do to get the SID tool to allocate SIDs for
> > enum
> > > > > > values?
> > > > > > 
> > > > > > We could then define the CBOR tag for enums in unions to take the
> > usual
> > > > > > SID difference (delta relative to the environment, I’d think), not
> > an
> > > > > > integer value.
> > > > > > 
> > > > > > Several of us are at the hackathon and could make something happen
> > today
> > > > > > and tomorrow.
> > > > > > 
> > > > > > Grüße, Carsten
> > > > > > 
> > > > > > 
> > > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[e-mail]
> > >
> > > > > > > wrote:
> > > > > > > 
> > > > > > > I guess that the concern is that this introduces more variation in
> > how
> > > > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > > > 
> > > > > > > E.g. if someone switched from XML to CBOR, suddenly the
> > configuration
> > > > > > > or state data may have a different meaning.
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Rob
> > > > > > > 
> > > > > > > 
> > > > > > > > -----Original Message-----
> > > > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > > > Sent: 22 March 2019 16:08
> > > > > > > > To: Michel Veillette <[e-mail]>
> > > > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>; [e-mail]; 
> > > > > > > > [e-mail]
> > > > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > > > 
> > > > > > > > On Mar 22, 2019, at 16:45, Michel Veillette 
> > > > > > > > <[e-mail]>
> > > > > > > > wrote:
> > > > > > > > > The only potential problem I aware is when multiple
> > enumerations 
> > > > > > > > > are part of
> > > > > > > > the same union.
> > > > > > > > > Value 4 from enumeration A will be encoded the same way as
> > Value 
> > > > > > > > > 4 from
> > > > > > > > enumeration B.
> > > > > > > > 
> > > > > > > > … and that is not a problem for the XML version, because the 
> > > > > > > > string is being used instead of the value.  (But then if two 
> > > > > > > > enumerations share a string, you have the equivalent problem in 
> > > > > > > > the XML serialization.)
> > > > > > > > 
> > > > > > > > Anyway, I haven’t seen a piece of real-world YANG that actually 
> > > > > > > > has this problem, so I would be a bit reluctant to make CBOR-
> > based 
> > > > > > > > implementations more complex (and less efficient) so solve this
> > > > > > > > (non-?)problem.
> > > > > > > > 
> > > > > > > > Grüße, Carsten
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > _______________________________________________
> > > > > > netconf mailing list
> > > > > > [e-mail]
> > > > > > 
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > > >
> > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > > >
> > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > > >
> > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > > > kvPZgr4o%3D&amp;reserved=0
> > > > > 
> > > > > -- 
> > > > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > > > Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
> > > > > Fax:   [phone]         <
> > > > > 
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > > > >
> > > > > 
> > > > > 
> > > > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > > > Well, if that is a problem, we can go for a longer representation
> > within
> > > > > > unions (section 6.12).  Theoretically, we could do that only of
> > there is
> > > > > > more than one enum in the union type (so things stay efficient if
> > there
> > > > > > is only one), but that might pose difficulties with model evolution.
> > > > > > 
> > > > > > Going for a string representation repeats the feature of XML YANG
> > (which
> > > > > > was ported over to JSON YANG):
> > > > > > 
> > > > > > typedef foo {
> > > > > >   type union {
> > > > > >     type enumeration {
> > > > > >       enum red { value 1; }
> > > > > >       enum breen { value 2; }
> > > > > >       enum glue { value 3; }
> > > > > >     }
> > > > > >     type enumeration {
> > > > > >       enum tacks { value 1; }
> > > > > >       enum nails { value 2; }
> > > > > >       enum glue { value 3; }
> > > > > >     }
> > > > > >   }
> > > > > > }
> > > > > > 
> > > > > > If you use “glue”, you don’t know which of the enumerations are
> > being
> > > > > > used.
> > > > > > 
> > > > > > Using SIDs, we can do better.
> > > > > > 
> > > > > > So what do we have to do to get the SID tool to allocate SIDs for
> > enum
> > > > > > values?
> > > > > > 
> > > > > > We could then define the CBOR tag for enums in unions to take the
> > usual
> > > > > > SID difference (delta relative to the environment, I’d think), not
> > an
> > > > > > integer value.
> > > > > > 
> > > > > > Several of us are at the hackathon and could make something happen
> > today
> > > > > > and tomorrow.
> > > > > > 
> > > > > > Grüße, Carsten
> > > > > > 
> > > > > > 
> > > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[e-mail]
> > >
> > > > > > > wrote:
> > > > > > > 
> > > > > > > I guess that the concern is that this introduces more variation in
> > how
> > > > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > > > 
> > > > > > > E.g. if someone switched from XML to CBOR, suddenly the
> > configuration
> > > > > > > or state data may have a different meaning.
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Rob
> > > > > > > 
> > > > > > > 
> > > > > > > > -----Original Message-----
> > > > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > > > Sent: 22 March 2019 16:08
> > > > > > > > To: Michel Veillette <[e-mail]>
> > > > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>; [e-mail]; 
> > > > > > > > [e-mail]
> > > > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > > > 
> > > > > > > > On Mar 22, 2019, at 16:45, Michel Veillette 
> > > > > > > > <[e-mail]>
> > > > > > > > wrote:
> > > > > > > > > The only potential problem I aware is when multiple
> > enumerations 
> > > > > > > > > are part of
> > > > > > > > the same union.
> > > > > > > > > Value 4 from enumeration A will be encoded the same way as
> > Value 
> > > > > > > > > 4 from
> > > > > > > > enumeration B.
> > > > > > > > 
> > > > > > > > … and that is not a problem for the XML version, because the 
> > > > > > > > string is being used instead of the value.  (But then if two 
> > > > > > > > enumerations share a string, you have the equivalent problem in 
> > > > > > > > the XML serialization.)
> > > > > > > > 
> > > > > > > > Anyway, I haven’t seen a piece of real-world YANG that actually 
> > > > > > > > has this problem, so I would be a bit reluctant to make CBOR-
> > based 
> > > > > > > > implementations more complex (and less efficient) so solve this
> > > > > > > > (non-?)problem.
> > > > > > > > 
> > > > > > > > Grüße, Carsten
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > _______________________________________________
> > > > > > netconf mailing list
> > > > > > [e-mail]
> > > > > > 
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > > >
> > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > > >
> > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > > >
> > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > > > kvPZgr4o%3D&amp;reserved=0
> > > > > 
> > > > > -- 
> > > > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > > > Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
> > > > > Fax:   [phone]         <
> > > > > 
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > > > >
> > > > > 
> > > > > _______________________________________________
> > > > > netconf mailing list
> > > > > [e-mail]
> > > > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > > > > ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> > > > > cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> > > > > 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> > > > > gr4o%3D&amp;reserved=0
> > > > 
> > > > --
> > > > Ladislav Lhotka
> > > > Head, CZ.NIC Labs
> > > > PGP Key ID: 0xB8F92B08A9F76C67
> > > 
> > > 
-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67