Re: [netconf] [core] YANG encoding in CBOR

[Andy Bierman <[e-mail]>]

On Wed, Mar 27, 2019 at 12:54 PM Ladislav Lhotka <[e-mail]> wrote:

> Andy Bierman píše v St 27. 03. 2019 v 09:13 -0700:
> >
> >
> > On Wed, Mar 27, 2019 at 1:40 AM Ladislav Lhotka <[e-mail]> wrote:
> > > On Wed, 2019-03-27 at 07:16 +0100, Juergen Schoenwaelder wrote:
> > > > Hi,
> > > >
> > > > a union can be formed by using member types that are imported and not
> > > > under change control of a single author/organization and ideally this
> > > > should work without complex coordination of name and number spaces.
> > > > Duplicate enum/bits values are legal in YANG today so an encoding has
> > > > to deal with this aspect of life.
> > > >
> > > > A robust fix to all these problems will be to tag the type members in
> > > > order to discriminate the values in the encodings. This, however,
> will
> > > > take some time to specify and we will need to preserve backwards
> > > > compatibility with unions without a tag (but compilers can encourage
> > > > people to add tags whenever modules are updated).
> > >
> > > I already opened a new issue for this in yang-next:
> > >
> > > https://github.com/netmod-wg/yang-next/issues/72
> > >
> >
> > We already explored the solution of giving the member types numbers
> > so they could be used in CBOR but this was rejected because it is so
> complex
> > to implement.
>
> I don't think it is too complex. Get the type in the schema, get the
> member,
> that's all. It is much easier and more robust than the current algorithm.
>
>
no it isn't easier because the algorithm is error-prone.
Renumbering un-named types nested within unions and referenced named types
over many revisions of the module is not easy.
The encoding MUST NOT change EVER -- even if the union type is expanded.


> >
> > Consider when union is within union, and the types are named types from
>
> Union within union is no problem: it jut requires a sequence of numbers.
>
>
do not agree since the union can have a member type added



> > other modules. Union types can be legally updated in new versions of the
> > module,
>
> Again, I don't agree. Sec. 11 in RFC 7950 says:
>
>    o  A "type" statement may be replaced with another "type" statement
>       that does not change the syntax or semantics of the type.
>       ...
>
>

   type uint32 { range "1..8"; }

I can refactor that into 2 types (1..4 and 5..8)
This does not change the semantics of the union type but it adds a member
type
Other refactoring is possible that will also change the number of member
types.

Also, deviations do not follow this rule



> Adding a new member clearly changes the semantics of the type, if not
> syntax.
>
>
no it doesn't always change the semantics



> > but the position assignments for SID can never change.
>
> The annotation would also help resolve the differences between JSON and
> XML. SID
> is only available in CBOR.
>
> >
> > Even without this complexity this solution would cause the
> encoder/decoder to
> > be very schema-aware.
>
> The current method of resolving unions is totally schema-aware (somewhat
> less so
> in JSON).
>
>
No -- it just needs to know string vs. number.
Not schema aware at all.


> Lada
>
>
Andy


> >
> >
> > BTW, creating SIDs for enums and bits will break if the server uses
> 'deviate
> > replace type'.
> > There is no way to number the deviated type since this is
> server-specific not
> > part of the original module
> > and the deviation is not actually a data-def-stmt.
> >
> >
> > > Lada
> > >
> >
> > Andy
> >
> > > >
> > > > /js
> > > >
> > > > On Wed, Mar 27, 2019 at 01:12:52AM +0000, Michel Veillette wrote:
> > > > > Hi Ladislav
> > > > >
> > > > > If I summarize this issue of multiple enumerations or bits in a
> union,
> > > this
> > > > > problem can be solve by the following approaches:
> > > > >
> > > > > - To not allows these duplicate values or positions to happen in
> YANG
> > > > > - To encode enumerations and bits as string (in all unions or only
> when
> > > > > multiple enumerations or bits are defined)
> > > > > - To encode enumerations and bits as SID (in all unions or only
> when
> > > > > multiple enumerations or bits are defined)
> > > > > - To encode enumerations and bits as delta between the value SID
> and the
> > > > > leaf SID (in all unions or only when multiple enumerations or bits
> are
> > > > > defined)
> > > > >
> > > > > In this email, I will like to focus on the first option, fixing the
> > > problem
> > > > > directly in YANG instead of fixing the consequences.
> > > > >
> > > > > Without any changes in YANG, a union with multiple enumeration or
> bits
> > > can
> > > > > be constructed without value or position overlaps.
> > > > > For example:
> > > > >
> > > > >   leaf multiple-enumerations-test-1 {
> > > > >     type union {
> > > > >       type enumeration {
> > > > >         enum "Monday" { value 0; }
> > > > >         enum "Tuesday" { value 1; }
> > > > >         enum "Wednesday" { value 2; }
> > > > >         enum "Thursday" { value 3; }
> > > > >         enum "Friday" { value 4; }
> > > > >
> > > > >       }
> > > > >       type enumeration {
> > > > >         enum "Saturday" { value 5; }
> > > > >         enum "Sunday" { value 6; }
> > > > >       }
> > > > >     }
> > > > >   }
> > > > >
> > > > >   leaf multiple-bits-test-1 {
> > > > >     type union {
> > > > >       type bits {
> > > > >         bit  "Monday" { position  0; }
> > > > >         bit "Tuesday" { position  1; }
> > > > >         bit "Wednesday" { position  2; }
> > > > >         bit "Thursday" { position  3; }
> > > > >         bit "Friday" { position  4; }
> > > > >
> > > > >       }
> > > > >       type bits {
> > > > >         bit "Saturday" { position 5; }
> > > > >         bit "Sunday" { position 6; }
> > > > >       }
> > > > >     }
> > > > >   }
> > > > >
> > > > > When using already defined typedef, avoiding overlap is less
> obvious
> > > without
> > > > > some help.
> > > > > To help building unions with already defined typedefs, I propose to
> > > > > introduce two extensions.
> > > > >
> > > > >   extension value-offset {
> > > > >     argument offset {
> > > > >       yin-element true;
> > > > >     }
> > > > >     description
> > > > >       "Offset added to each enum value of the associated
> enumeration.";
> > > > >   }
> > > > >
> > > > >   extension position-offset {
> > > > >     argument offset {
> > > > >       yin-element true;
> > > > >     }
> > > > >     description
> > > > >       "Offset value added to each bit position of the associated
> bits.";
> > > > >   }
> > > > >
> > > > > The value-offset extension can be used as follow:
> > > > >
> > > > >     type enumeration {
> > > > >       enum "Monday";
> > > > >       enum "Tuesday";
> > > > >       enum "Wednesday";
> > > > >       enum "Thursday";
> > > > >       enum "Friday";
> > > > >     }
> > > > >   }
> > > > >
> > > > >   typedef weekend {
> > > > >     type enumeration {
> > > > >       enum "Saturday";
> > > > >       enum "Sunday";
> > > > >     }
> > > > >   }
> > > > >
> > > > >   leaf multiple-enumerations-test-3 {
> > > > >     type union {
> > > > >       type weekdays;
> > > > >       type weekend {
> > > > >         ext:value-offset 5;
> > > > >       }
> > > > >     }
> > > > >   }
> > > > >
> > > > > The position-offset extension can be used as follow:
> > > > >
> > > > >   typedef weekdays-flags {
> > > > >     type bits {
> > > > >       bit "Monday";
> > > > >       bit "Tuesday";
> > > > >       bit "Wednesday";
> > > > >       bit "Thursday";
> > > > >       bit "Friday";
> > > > >     }
> > > > >   }
> > > > >
> > > > >   typedef weekend-flags {
> > > > >     type bits {
> > > > >       bit "Saturday";
> > > > >       bit "Sunday";
> > > > >     }
> > > > >   }
> > > > >
> > > > >   leaf multiple-bits-test-3 {
> > > > >     type union {
> > > > >       type weekdays-flags;
> > > > >       type weekend-flags {
> > > > >         ext:position-offset 5;
> > > > >       }
> > > > >     }
> > > > >   }
> > > > >
> > > > > The yang file in attachment show different examples based on this
> > > approach.
> > > > > This module have been validated using
> > > http://www.yangvalidator.com/validator
> > > > >
> > > > > If this approach is accepted, tools like pyang should be updated to
> > > produce
> > > > > an error if multiple enumerations or bits are defined with value or
> > > position
> > > > > overleaps.
> > > > >
> > > > > Please comment,
> > > > > Michel
> > > > >
> > > > > -----Original Message-----
> > > > > From: Ladislav Lhotka <[e-mail]>
> > > > > Sent: Monday, March 25, 2019 4:07 AM
> > > > > To: Juergen Schoenwaelder <[e-mail]>;
> > > Carsten
> > > > > Bormann <[e-mail]>
> > > > > Cc: Michel Veillette <[e-mail]>;
> [e-mail];
> > >
> > > > > [e-mail]
> > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > >
> > > > > Juergen Schoenwaelder <[e-mail]>
> writes:
> > > > >
> > > > > > I think we need to look at the whole picture and in which
> direction
> > > we
> > > > > > want to go. In the longer term, I would prefer a solution where
> the
> > > > > > values of a union are discriminated. The current XML encoding
> > > > > > behaviour of 'first match wins' is fragile (for example, if
> someone
> > > > > > adds an enum to a type, the interpretation of data can change).
> > > > > >
> > > > > > Look at this:
> > > > > >
> > > > > > typedef bar {
> > > > > >   type union {
> > > > > >     type enumeration { enum "1"; value 2; enum "2"; value 1; }
> > > > > >     type uint8;
> > > > > >   }
> > > > > > }
> > > > > >
> > > > > > We have some encodings that send the string representations of
> the
> > > > > > values and some encodings that prefer to send numeric
> representations
> > > > > > where possible. In order to have a robust solution, encodings
> should
> > > > > > likely indicate to which type the value belongs.
> > > > >
> > > > > Perhaps the easiest way would be to use (optional) annotation that
> > > > > specifies, using an ordinal number, which of the member types is
> used
> > > for
> > > > > the particular instance. But since there can be unions inside
> unions, a
> > > list
> > > > > of numbers would be needed in general.
> > > > >
> > > > > Lada
> > > > >
> > > > > > /js
> > > > > >
> > > > > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > > > > Well, if that is a problem, we can go for a longer
> representation
> > > within
> > > > > > > unions (section 6.12).  Theoretically, we could do that only of
> > > there is
> > > > > > > more than one enum in the union type (so things stay efficient
> if
> > > there
> > > > > > > is only one), but that might pose difficulties with model
> evolution.
> > > > > > >
> > > > > > > Going for a string representation repeats the feature of XML
> YANG
> > > (which
> > > > > > > was ported over to JSON YANG):
> > > > > > >
> > > > > > > typedef foo {
> > > > > > >   type union {
> > > > > > >     type enumeration {
> > > > > > >       enum red { value 1; }
> > > > > > >       enum breen { value 2; }
> > > > > > >       enum glue { value 3; }
> > > > > > >     }
> > > > > > >     type enumeration {
> > > > > > >       enum tacks { value 1; }
> > > > > > >       enum nails { value 2; }
> > > > > > >       enum glue { value 3; }
> > > > > > >     }
> > > > > > >   }
> > > > > > > }
> > > > > > >
> > > > > > > If you use “glue”, you don’t know which of the enumerations are
> > > being
> > > > > > > used.
> > > > > > >
> > > > > > > Using SIDs, we can do better.
> > > > > > >
> > > > > > > So what do we have to do to get the SID tool to allocate SIDs
> for
> > > enum
> > > > > > > values?
> > > > > > >
> > > > > > > We could then define the CBOR tag for enums in unions to take
> the
> > > usual
> > > > > > > SID difference (delta relative to the environment, I’d think),
> not
> > > an
> > > > > > > integer value.
> > > > > > >
> > > > > > > Several of us are at the hackathon and could make something
> happen
> > > today
> > > > > > > and tomorrow.
> > > > > > >
> > > > > > > Grüße, Carsten
> > > > > > >
> > > > > > >
> > > > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <
> [e-mail]
> > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > I guess that the concern is that this introduces more
> variation in
> > > how
> > > > > > > > data is interpreted between the different XML/JSON/CBOR
> encodings.
> > > > > > > >
> > > > > > > > E.g. if someone switched from XML to CBOR, suddenly the
> > > configuration
> > > > > > > > or state data may have a different meaning.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rob
> > > > > > > >
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > > > > Sent: 22 March 2019 16:08
> > > > > > > > > To: Michel Veillette <[e-mail]>
> > > > > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>;
> [e-mail];
> > > > > > > > > [e-mail]
> > > > > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > > > >
> > > > > > > > > On Mar 22, 2019, at 16:45, Michel Veillette
> > > > > > > > > <[e-mail]>
> > > > > > > > > wrote:
> > > > > > > > > > The only potential problem I aware is when multiple
> > > enumerations
> > > > > > > > > > are part of
> > > > > > > > > the same union.
> > > > > > > > > > Value 4 from enumeration A will be encoded the same way
> as
> > > Value
> > > > > > > > > > 4 from
> > > > > > > > > enumeration B.
> > > > > > > > >
> > > > > > > > > … and that is not a problem for the XML version, because
> the
> > > > > > > > > string is being used instead of the value.  (But then if
> two
> > > > > > > > > enumerations share a string, you have the equivalent
> problem in
> > > > > > > > > the XML serialization.)
> > > > > > > > >
> > > > > > > > > Anyway, I haven’t seen a piece of real-world YANG that
> actually
> > > > > > > > > has this problem, so I would be a bit reluctant to make
> CBOR-
> > > based
> > > > > > > > > implementations more complex (and less efficient) so solve
> this
> > > > > > > > > (non-?)problem.
> > > > > > > > >
> > > > > > > > > Grüße, Carsten
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > netconf mailing list
> > > > > > > [e-mail]
> > > > > > >
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > > > >
> > > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > > > >
> > > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > > > >
> > > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > > > > kvPZgr4o%3D&amp;reserved=0
> > > > > >
> > > > > > --
> > > > > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > > > > Phone: [phone]         Campus Ring 1 | 28759 Bremen |
> Germany
> > > > > > Fax:   [phone]         <
> > > > > >
> > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > > > > Well, if that is a problem, we can go for a longer
> representation
> > > within
> > > > > > > unions (section 6.12).  Theoretically, we could do that only of
> > > there is
> > > > > > > more than one enum in the union type (so things stay efficient
> if
> > > there
> > > > > > > is only one), but that might pose difficulties with model
> evolution.
> > > > > > >
> > > > > > > Going for a string representation repeats the feature of XML
> YANG
> > > (which
> > > > > > > was ported over to JSON YANG):
> > > > > > >
> > > > > > > typedef foo {
> > > > > > >   type union {
> > > > > > >     type enumeration {
> > > > > > >       enum red { value 1; }
> > > > > > >       enum breen { value 2; }
> > > > > > >       enum glue { value 3; }
> > > > > > >     }
> > > > > > >     type enumeration {
> > > > > > >       enum tacks { value 1; }
> > > > > > >       enum nails { value 2; }
> > > > > > >       enum glue { value 3; }
> > > > > > >     }
> > > > > > >   }
> > > > > > > }
> > > > > > >
> > > > > > > If you use “glue”, you don’t know which of the enumerations are
> > > being
> > > > > > > used.
> > > > > > >
> > > > > > > Using SIDs, we can do better.
> > > > > > >
> > > > > > > So what do we have to do to get the SID tool to allocate SIDs
> for
> > > enum
> > > > > > > values?
> > > > > > >
> > > > > > > We could then define the CBOR tag for enums in unions to take
> the
> > > usual
> > > > > > > SID difference (delta relative to the environment, I’d think),
> not
> > > an
> > > > > > > integer value.
> > > > > > >
> > > > > > > Several of us are at the hackathon and could make something
> happen
> > > today
> > > > > > > and tomorrow.
> > > > > > >
> > > > > > > Grüße, Carsten
> > > > > > >
> > > > > > >
> > > > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <
> [e-mail]
> > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > I guess that the concern is that this introduces more
> variation in
> > > how
> > > > > > > > data is interpreted between the different XML/JSON/CBOR
> encodings.
> > > > > > > >
> > > > > > > > E.g. if someone switched from XML to CBOR, suddenly the
> > > configuration
> > > > > > > > or state data may have a different meaning.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rob
> > > > > > > >
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Carsten Bormann <[e-mail]>
> > > > > > > > > Sent: 22 March 2019 16:08
> > > > > > > > > To: Michel Veillette <[e-mail]>
> > > > > > > > > Cc: Rob Wilton (rwilton) <[e-mail]>;
> [e-mail];
> > > > > > > > > [e-mail]
> > > > > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > > > >
> > > > > > > > > On Mar 22, 2019, at 16:45, Michel Veillette
> > > > > > > > > <[e-mail]>
> > > > > > > > > wrote:
> > > > > > > > > > The only potential problem I aware is when multiple
> > > enumerations
> > > > > > > > > > are part of
> > > > > > > > > the same union.
> > > > > > > > > > Value 4 from enumeration A will be encoded the same way
> as
> > > Value
> > > > > > > > > > 4 from
> > > > > > > > > enumeration B.
> > > > > > > > >
> > > > > > > > > … and that is not a problem for the XML version, because
> the
> > > > > > > > > string is being used instead of the value.  (But then if
> two
> > > > > > > > > enumerations share a string, you have the equivalent
> problem in
> > > > > > > > > the XML serialization.)
> > > > > > > > >
> > > > > > > > > Anyway, I haven’t seen a piece of real-world YANG that
> actually
> > > > > > > > > has this problem, so I would be a bit reluctant to make
> CBOR-
> > > based
> > > > > > > > > implementations more complex (and less efficient) so solve
> this
> > > > > > > > > (non-?)problem.
> > > > > > > > >
> > > > > > > > > Grüße, Carsten
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > netconf mailing list
> > > > > > > [e-mail]
> > > > > > >
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > > > >
> > > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > > > >
> > > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > > > >
> > > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > > > > kvPZgr4o%3D&amp;reserved=0
> > > > > >
> > > > > > --
> > > > > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > > > > Phone: [phone]         Campus Ring 1 | 28759 Bremen |
> Germany
> > > > > > Fax:   [phone]         <
> > > > > >
> > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > netconf mailing list
> > > > > > [e-mail]
> > > > > >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > > > > > ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> > > > > >
> cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> > > > > >
> 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> > > > > > gr4o%3D&amp;reserved=0
> > > > >
> > > > > --
> > > > > Ladislav Lhotka
> > > > > Head, CZ.NIC Labs
> > > > > PGP Key ID: 0xB8F92B08A9F76C67
> > > >
> > > >
> --
> Ladislav Lhotka
> Head, CZ.NIC Labs
> PGP Key ID: 0xB8F92B08A9F76C67
>
>