Re: [netconf] YANG encoding in CBOR

Ladislav Lhotka <lhotka@nic.cz> Wed, 27 March 2019 08:40 UTC

Return-Path: <lhotka@nic.cz>
X-Original-To: netconf@ietfa.amsl.com
Delivered-To: netconf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 632FF120277; Wed, 27 Mar 2019 01:40:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.999
X-Spam-Level:
X-Spam-Status: No, score=-6.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=nic.cz
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DiSctlJq6m8R; Wed, 27 Mar 2019 01:40:11 -0700 (PDT)
Received: from mail.nic.cz (mail.nic.cz [217.31.204.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6FEEA120276; Wed, 27 Mar 2019 01:40:10 -0700 (PDT)
Received: from birdie (unknown [IPv6:2001:718:1a02:1::380]) by mail.nic.cz (Postfix) with ESMTPSA id 4AD6D605FC; Wed, 27 Mar 2019 09:40:08 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=nic.cz; s=default; t=1553676008; bh=BxpabmPoA5pcWTCBlPBgkrOdUdACty8/51V6MLJCgQU=; h=From:To:Date; b=cNvIN8s5FQTYn0gKDxlM1u/LAgooaULY7U1VpCx7F/qTYxGwarspVwsBLHABNhbrG +L2HM1rfbmMBLUjeH4bte/1DxbPsF6tutDv/R0F1A2LlndF0M2zc4bAKh0VBTe3KKY YCp9hJ3VFxVdj6L5LuPD6R/wQTzNOHHLEuMIo6Po=
Message-ID: <1f8b326e0e05b400457b9446d52a7b0f6c90e05b.camel@nic.cz>
From: Ladislav Lhotka <lhotka@nic.cz>
To: Juergen Schoenwaelder <j.schoenwaelder@jacobs-university.de>, Michel Veillette <Michel.Veillette@trilliant.com>
Cc: Carsten Bormann <cabo@tzi.org>, "netconf@ietf.org" <netconf@ietf.org>, "core@ietf.org" <core@ietf.org>
Date: Wed, 27 Mar 2019 09:40:08 +0100
In-Reply-To: <20190327061637.g5a7t7nulk7kyh2v@anna.jacobs.jacobs-university.de>
References: <6235c6683ff14848a661f8b8cec94280@XCH-RCD-007.cisco.com> <BL0PR06MB5042823429DB7CDA0F33408B9A430@BL0PR06MB5042.namprd06.prod.outlook.com> <588401AB-483E-40F5-95BB-20A066E56DAC@tzi.org> <15fbaf84b20343a1b83f40b571149a14@XCH-RCD-007.cisco.com> <1ADF8201-ABB4-44FD-A515-F3F8E0DBF5FC@tzi.org> <20190323101003.gp3zvsvqqwc26jip@anna.jacobs.jacobs-university.de> <871s2vqsxi.fsf@nic.cz> <BL0PR06MB5042C9AA6B4A0CCD913F50D89A580@BL0PR06MB5042.namprd06.prod.outlook.com> <20190327061637.g5a7t7nulk7kyh2v@anna.jacobs.jacobs-university.de>
Organization: CZ.NIC
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.32.0
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Virus-Scanned: clamav-milter 0.99.2 at mail
X-Virus-Status: Clean
Archived-At: <https://mailarchive.ietf.org/arch/msg/netconf/xOrdjOTsI2JuY4-GyEYAzpIuJJg>
Subject: Re: [netconf] YANG encoding in CBOR
X-BeenThere: netconf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <netconf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/netconf>, <mailto:netconf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/netconf/>
List-Post: <mailto:netconf@ietf.org>
List-Help: <mailto:netconf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/netconf>, <mailto:netconf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Mar 2019 08:40:17 -0000

On Wed, 2019-03-27 at 07:16 +0100, Juergen Schoenwaelder wrote:
> Hi,
> 
> a union can be formed by using member types that are imported and not
> under change control of a single author/organization and ideally this
> should work without complex coordination of name and number spaces.
> Duplicate enum/bits values are legal in YANG today so an encoding has
> to deal with this aspect of life.
> 
> A robust fix to all these problems will be to tag the type members in
> order to discriminate the values in the encodings. This, however, will
> take some time to specify and we will need to preserve backwards
> compatibility with unions without a tag (but compilers can encourage
> people to add tags whenever modules are updated).

I already opened a new issue for this in yang-next:

https://github.com/netmod-wg/yang-next/issues/72

Lada

> 
> /js
> 
> On Wed, Mar 27, 2019 at 01:12:52AM +0000, Michel Veillette wrote:
> > Hi Ladislav
> > 
> > If I summarize this issue of multiple enumerations or bits in a union, this
> > problem can be solve by the following approaches:
> > 
> > - To not allows these duplicate values or positions to happen in YANG
> > - To encode enumerations and bits as string (in all unions or only when
> > multiple enumerations or bits are defined)
> > - To encode enumerations and bits as SID (in all unions or only when
> > multiple enumerations or bits are defined)
> > - To encode enumerations and bits as delta between the value SID and the
> > leaf SID (in all unions or only when multiple enumerations or bits are
> > defined)
> > 
> > In this email, I will like to focus on the first option, fixing the problem
> > directly in YANG instead of fixing the consequences.
> > 
> > Without any changes in YANG, a union with multiple enumeration or bits can
> > be constructed without value or position overlaps.
> > For example:
> > 
> >   leaf multiple-enumerations-test-1 {
> >     type union {
> >       type enumeration {
> >         enum "Monday" { value 0; }
> >         enum "Tuesday" { value 1; }
> >         enum "Wednesday" { value 2; }
> >         enum "Thursday" { value 3; }
> >         enum "Friday" { value 4; }
> > 
> >       }
> >       type enumeration {
> >         enum "Saturday" { value 5; }
> >         enum "Sunday" { value 6; }
> >       }
> >     }
> >   }
> > 
> >   leaf multiple-bits-test-1 {
> >     type union {
> >       type bits {
> >         bit  "Monday" { position  0; }
> >         bit "Tuesday" { position  1; }
> >         bit "Wednesday" { position  2; }
> >         bit "Thursday" { position  3; }
> >         bit "Friday" { position  4; }
> > 
> >       }
> >       type bits {
> >         bit "Saturday" { position 5; }
> >         bit "Sunday" { position 6; }
> >       }
> >     }
> >   }
> > 
> > When using already defined typedef, avoiding overlap is less obvious without
> > some help.
> > To help building unions with already defined typedefs, I propose to
> > introduce two extensions. 
> > 
> >   extension value-offset {
> >     argument offset {
> >       yin-element true;
> >     }
> >     description
> >       "Offset added to each enum value of the associated enumeration.";
> >   }
> >   
> >   extension position-offset {
> >     argument offset {
> >       yin-element true;
> >     }
> >     description
> >       "Offset value added to each bit position of the associated bits.";
> >   }
> > 
> > The value-offset extension can be used as follow:
> > 
> >     type enumeration {
> >       enum "Monday";
> >       enum "Tuesday";
> >       enum "Wednesday";
> >       enum "Thursday";
> >       enum "Friday";
> >     }
> >   }
> > 
> >   typedef weekend {
> >     type enumeration {
> >       enum "Saturday";
> >       enum "Sunday";
> >     }
> >   }
> >   
> >   leaf multiple-enumerations-test-3 {
> >     type union {
> >       type weekdays;
> >       type weekend {
> >         ext:value-offset 5;
> >       }
> >     }
> >   }
> > 
> > The position-offset extension can be used as follow:
> > 
> >   typedef weekdays-flags {
> >     type bits {
> >       bit "Monday";
> >       bit "Tuesday";
> >       bit "Wednesday";
> >       bit "Thursday";
> >       bit "Friday";
> >     }
> >   }
> > 
> >   typedef weekend-flags {
> >     type bits {
> >       bit "Saturday";
> >       bit "Sunday";
> >     }
> >   }
> >   
> >   leaf multiple-bits-test-3 {
> >     type union {
> >       type weekdays-flags;
> >       type weekend-flags {
> >         ext:position-offset 5;
> >       }
> >     }
> >   }
> > 
> > The yang file in attachment show different examples based on this approach.
> > This module have been validated using http://www.yangvalidator.com/validator
> >  
> > If this approach is accepted, tools like pyang should be updated to produce
> > an error if multiple enumerations or bits are defined with value or position
> > overleaps.
> > 
> > Please comment,
> > Michel
> > 
> > -----Original Message-----
> > From: Ladislav Lhotka <lhotka@nic.cz>; 
> > Sent: Monday, March 25, 2019 4:07 AM
> > To: Juergen Schoenwaelder <j.schoenwaelder@jacobs-university.de>;; Carsten
> > Bormann <cabo@tzi.org>;
> > Cc: Michel Veillette <Michel.Veillette@trilliant.com>;; netconf@ietf.org; 
> > core@ietf.org
> > Subject: Re: [netconf] YANG encoding in CBOR
> > 
> > Juergen Schoenwaelder <j.schoenwaelder@jacobs-university.de>; writes:
> > 
> > > I think we need to look at the whole picture and in which direction we 
> > > want to go. In the longer term, I would prefer a solution where the 
> > > values of a union are discriminated. The current XML encoding 
> > > behaviour of 'first match wins' is fragile (for example, if someone 
> > > adds an enum to a type, the interpretation of data can change).
> > > 
> > > Look at this:
> > > 
> > > typedef bar {
> > >   type union {
> > >     type enumeration { enum "1"; value 2; enum "2"; value 1; }
> > >     type uint8;
> > >   }
> > > }
> > > 
> > > We have some encodings that send the string representations of the 
> > > values and some encodings that prefer to send numeric representations 
> > > where possible. In order to have a robust solution, encodings should 
> > > likely indicate to which type the value belongs.
> > 
> > Perhaps the easiest way would be to use (optional) annotation that
> > specifies, using an ordinal number, which of the member types is used for
> > the particular instance. But since there can be unions inside unions, a list
> > of numbers would be needed in general.
> > 
> > Lada
> > 
> > > /js
> > > 
> > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > Well, if that is a problem, we can go for a longer representation within
> > > > unions (section 6.12).  Theoretically, we could do that only of there is
> > > > more than one enum in the union type (so things stay efficient if there
> > > > is only one), but that might pose difficulties with model evolution.
> > > > 
> > > > Going for a string representation repeats the feature of XML YANG (which
> > > > was ported over to JSON YANG):
> > > > 
> > > > typedef foo {
> > > >   type union {
> > > >     type enumeration {
> > > >       enum red { value 1; }
> > > >       enum breen { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >     type enumeration {
> > > >       enum tacks { value 1; }
> > > >       enum nails { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >   }
> > > > }
> > > > 
> > > > If you use “glue”, you don’t know which of the enumerations are being
> > > > used.
> > > > 
> > > > Using SIDs, we can do better.
> > > > 
> > > > So what do we have to do to get the SID tool to allocate SIDs for enum
> > > > values?
> > > > 
> > > > We could then define the CBOR tag for enums in unions to take the usual
> > > > SID difference (delta relative to the environment, I’d think), not an
> > > > integer value.
> > > > 
> > > > Several of us are at the hackathon and could make something happen today
> > > > and tomorrow.
> > > > 
> > > > Grüße, Carsten
> > > > 
> > > > 
> > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <rwilton@cisco.com>;
> > > > > wrote:
> > > > > 
> > > > > I guess that the concern is that this introduces more variation in how
> > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > 
> > > > > E.g. if someone switched from XML to CBOR, suddenly the configuration
> > > > > or state data may have a different meaning.
> > > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Carsten Bormann <cabo@tzi.org>;
> > > > > > Sent: 22 March 2019 16:08
> > > > > > To: Michel Veillette <Michel.Veillette@trilliant.com>;
> > > > > > Cc: Rob Wilton (rwilton) <rwilton@cisco.com>;; core@ietf.org; 
> > > > > > netconf@ietf.org
> > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > 
> > > > > > On Mar 22, 2019, at 16:45, Michel Veillette 
> > > > > > <Michel.Veillette@trilliant.com>;
> > > > > > wrote:
> > > > > > > The only potential problem I aware is when multiple enumerations 
> > > > > > > are part of
> > > > > > the same union.
> > > > > > > Value 4 from enumeration A will be encoded the same way as Value 
> > > > > > > 4 from
> > > > > > enumeration B.
> > > > > > 
> > > > > > … and that is not a problem for the XML version, because the 
> > > > > > string is being used instead of the value.  (But then if two 
> > > > > > enumerations share a string, you have the equivalent problem in 
> > > > > > the XML serialization.)
> > > > > > 
> > > > > > Anyway, I haven’t seen a piece of real-world YANG that actually 
> > > > > > has this problem, so I would be a bit reluctant to make CBOR-based 
> > > > > > implementations more complex (and less efficient) so solve this
> > > > > > (non-?)problem.
> > > > > > 
> > > > > > Grüße, Carsten
> > > > > 
> > > > > 
> > > > 
> > > > _______________________________________________
> > > > netconf mailing list
> > > > netconf@ietf.org
> > > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > kvPZgr4o%3D&amp;reserved=0
> > > 
> > > -- 
> > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> > > Fax:   +49 421 200 3103         <
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > >
> > > 
> > > 
> > > On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> > > > Well, if that is a problem, we can go for a longer representation within
> > > > unions (section 6.12).  Theoretically, we could do that only of there is
> > > > more than one enum in the union type (so things stay efficient if there
> > > > is only one), but that might pose difficulties with model evolution.
> > > > 
> > > > Going for a string representation repeats the feature of XML YANG (which
> > > > was ported over to JSON YANG):
> > > > 
> > > > typedef foo {
> > > >   type union {
> > > >     type enumeration {
> > > >       enum red { value 1; }
> > > >       enum breen { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >     type enumeration {
> > > >       enum tacks { value 1; }
> > > >       enum nails { value 2; }
> > > >       enum glue { value 3; }
> > > >     }
> > > >   }
> > > > }
> > > > 
> > > > If you use “glue”, you don’t know which of the enumerations are being
> > > > used.
> > > > 
> > > > Using SIDs, we can do better.
> > > > 
> > > > So what do we have to do to get the SID tool to allocate SIDs for enum
> > > > values?
> > > > 
> > > > We could then define the CBOR tag for enums in unions to take the usual
> > > > SID difference (delta relative to the environment, I’d think), not an
> > > > integer value.
> > > > 
> > > > Several of us are at the hackathon and could make something happen today
> > > > and tomorrow.
> > > > 
> > > > Grüße, Carsten
> > > > 
> > > > 
> > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <rwilton@cisco.com>;
> > > > > wrote:
> > > > > 
> > > > > I guess that the concern is that this introduces more variation in how
> > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > 
> > > > > E.g. if someone switched from XML to CBOR, suddenly the configuration
> > > > > or state data may have a different meaning.
> > > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Carsten Bormann <cabo@tzi.org>;
> > > > > > Sent: 22 March 2019 16:08
> > > > > > To: Michel Veillette <Michel.Veillette@trilliant.com>;
> > > > > > Cc: Rob Wilton (rwilton) <rwilton@cisco.com>;; core@ietf.org; 
> > > > > > netconf@ietf.org
> > > > > > Subject: Re: [netconf] YANG encoding in CBOR
> > > > > > 
> > > > > > On Mar 22, 2019, at 16:45, Michel Veillette 
> > > > > > <Michel.Veillette@trilliant.com>;
> > > > > > wrote:
> > > > > > > The only potential problem I aware is when multiple enumerations 
> > > > > > > are part of
> > > > > > the same union.
> > > > > > > Value 4 from enumeration A will be encoded the same way as Value 
> > > > > > > 4 from
> > > > > > enumeration B.
> > > > > > 
> > > > > > … and that is not a problem for the XML version, because the 
> > > > > > string is being used instead of the value.  (But then if two 
> > > > > > enumerations share a string, you have the equivalent problem in 
> > > > > > the XML serialization.)
> > > > > > 
> > > > > > Anyway, I haven’t seen a piece of real-world YANG that actually 
> > > > > > has this problem, so I would be a bit reluctant to make CBOR-based 
> > > > > > implementations more complex (and less efficient) so solve this
> > > > > > (non-?)problem.
> > > > > > 
> > > > > > Grüße, Carsten
> > > > > 
> > > > > 
> > > > 
> > > > _______________________________________________
> > > > netconf mailing list
> > > > netconf@ietf.org
> > > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> > > > .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
> > > > %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
> > > > kvPZgr4o%3D&amp;reserved=0
> > > 
> > > -- 
> > > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > > Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> > > Fax:   +49 421 200 3103         <
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > >
> > > 
> > > _______________________________________________
> > > netconf mailing list
> > > netconf@ietf.org
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > > ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> > > cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> > > 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> > > gr4o%3D&amp;reserved=0
> > 
> > --
> > Ladislav Lhotka
> > Head, CZ.NIC Labs
> > PGP Key ID: 0xB8F92B08A9F76C67
> 
> 
-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67