Re: [netconf] YANG encoding in CBOR

Michel Veillette <Michel.Veillette@trilliant.com> Wed, 27 March 2019 01:12 UTC

Return-Path: <Michel.Veillette@trilliant.com>
X-Original-To: netconf@ietfa.amsl.com
Delivered-To: netconf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AF65C120165; Tue, 26 Mar 2019 18:12:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=trilliant.onmicrosoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ut0f4SB4HCqG; Tue, 26 Mar 2019 18:12:55 -0700 (PDT)
Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-eopbgr760091.outbound.protection.outlook.com [40.107.76.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 73F0A12010E; Tue, 26 Mar 2019 18:12:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Trilliant.onmicrosoft.com; s=selector1-Trilliant-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9u5WYVgClj6tJnHYMfY9F6TezUFZDpNTJkXqaVkH99Q=; b=h+Hc/VwmXxOchz6tO6LcW9MxRUzM6Bti/e1ksIrgbI8q6XN0FD2zoQvb6YxU9huTV+MqldPImRGyal4e/4LF8G57MY4sdHIx7svxqu+jsPxgNmsdQfiVQ++4tRHej0qeFk7hOFsXHxrE2axt331uYbiWpxwqwT9mwoEkpVAKvik=
Received: from BL0PR06MB5042.namprd06.prod.outlook.com (10.167.240.31) by BL0PR06MB4193.namprd06.prod.outlook.com (10.167.179.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.18; Wed, 27 Mar 2019 01:12:52 +0000
Received: from BL0PR06MB5042.namprd06.prod.outlook.com ([fe80::d5:e11c:db10:639d]) by BL0PR06MB5042.namprd06.prod.outlook.com ([fe80::d5:e11c:db10:639d%3]) with mapi id 15.20.1730.019; Wed, 27 Mar 2019 01:12:52 +0000
From: Michel Veillette <Michel.Veillette@trilliant.com>
To: Ladislav Lhotka <lhotka@nic.cz>, Juergen Schoenwaelder <j.schoenwaelder@jacobs-university.de>, Carsten Bormann <cabo@tzi.org>
CC: "netconf@ietf.org" <netconf@ietf.org>, "core@ietf.org" <core@ietf.org>
Thread-Topic: [netconf] YANG encoding in CBOR
Thread-Index: AdTf3q8hCOPNH5o0Q0SJZRQQHN87VgA3tcGQAAL8XAAAAtt/gAAgmRkAAAJStYAAYEUIgABUqX2Q
Date: Wed, 27 Mar 2019 01:12:52 +0000
Message-ID: <BL0PR06MB5042C9AA6B4A0CCD913F50D89A580@BL0PR06MB5042.namprd06.prod.outlook.com>
References: <6235c6683ff14848a661f8b8cec94280@XCH-RCD-007.cisco.com> <BL0PR06MB5042823429DB7CDA0F33408B9A430@BL0PR06MB5042.namprd06.prod.outlook.com> <588401AB-483E-40F5-95BB-20A066E56DAC@tzi.org> <15fbaf84b20343a1b83f40b571149a14@XCH-RCD-007.cisco.com> <1ADF8201-ABB4-44FD-A515-F3F8E0DBF5FC@tzi.org> <20190323101003.gp3zvsvqqwc26jip@anna.jacobs.jacobs-university.de> <871s2vqsxi.fsf@nic.cz>
In-Reply-To: <871s2vqsxi.fsf@nic.cz>
Accept-Language: fr-CA, en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is ) smtp.mailfrom=Michel.Veillette@trilliant.com;
x-originating-ip: [184.162.192.248]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: ac3ded06-5667-4074-e904-08d6b251562a
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(2017052603328)(7153060)(49563074)(7193020); SRVR:BL0PR06MB4193;
x-ms-traffictypediagnostic: BL0PR06MB4193:
x-ms-exchange-purlcount: 3
x-microsoft-antispam-prvs: <BL0PR06MB41935CAE0FDD6523EFC1F7569A580@BL0PR06MB4193.namprd06.prod.outlook.com>
x-forefront-prvs: 0989A7979C
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(346002)(376002)(396003)(366004)(136003)(39850400004)(199004)(189003)(13464003)(52314003)(6246003)(71200400001)(52536014)(97736004)(7696005)(53546011)(76176011)(106356001)(105586002)(71190400001)(6506007)(30864003)(14454004)(99286004)(966005)(8936002)(86362001)(53936002)(102836004)(2906002)(72206003)(5660300002)(45080400002)(14444005)(3846002)(5024004)(81156014)(186003)(229853002)(8676002)(6116002)(81166006)(305945005)(9686003)(55016002)(68736007)(478600001)(25786009)(316002)(7736002)(114624004)(33656002)(486006)(6436002)(93886005)(54906003)(99936001)(4326008)(110136005)(66066001)(74316002)(476003)(26005)(11346002)(446003)(6306002)(256004); DIR:OUT; SFP:1102; SCL:1; SRVR:BL0PR06MB4193; H:BL0PR06MB5042.namprd06.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None (protection.outlook.com: trilliant.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: m7Pa1E4i5yKeBbcgbOXLdPk1G9NiZmSoRiR2kmnr/L5nBqTbsDw2h/DxouWydU2PGjdCseuatF3LfLdWkTiPa8aSRODpk7rUGTJx/R5idv2+uemOTqkffaf/rHqN0xdZqrw8sHMwO02YfHNBqzSKtDAxPrsGkKytVq9jkoljjcQwq07JN80mBxICiChmt65Q6H7DO1IGYWR7/0MT8CtzBLcCDL0Z9Xl+zdMHWMuvMmVqUeQiEonIRiZEsdp6zqoriqvh0Ee+ZNXSgLeTwW/kZ7OHLNDYY2/BSHcg/bPLGGRK2zsTWMlij7uMtJ2MrlY+Q3qtSfPh6++QQN/zXZjV42i4A++zoDG/dxVsrRqr74zjpxuRvNmsLC9MfpY4SRCIIV9wmYuR1bO5EO0LTLUa1fn6AOrHt8RRLg/SShL4kA4=
Content-Type: multipart/mixed; boundary="_002_BL0PR06MB5042C9AA6B4A0CCD913F50D89A580BL0PR06MB5042namp_"
MIME-Version: 1.0
X-OriginatorOrg: Trilliant.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ac3ded06-5667-4074-e904-08d6b251562a
X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Mar 2019 01:12:52.5250 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4f6fbd13-0dfb-4150-85c3-d43260c04309
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR06MB4193
Archived-At: <https://mailarchive.ietf.org/arch/msg/netconf/zwnZVCiFbU9341IZA9elBMUEL9c>
Subject: Re: [netconf] YANG encoding in CBOR
X-BeenThere: netconf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <netconf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/netconf>, <mailto:netconf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/netconf/>
List-Post: <mailto:netconf@ietf.org>
List-Help: <mailto:netconf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/netconf>, <mailto:netconf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Mar 2019 01:12:59 -0000

Hi Ladislav

If I summarize this issue of multiple enumerations or bits in a union, this problem can be solve by the following approaches:

- To not allows these duplicate values or positions to happen in YANG
- To encode enumerations and bits as string (in all unions or only when multiple enumerations or bits are defined)
- To encode enumerations and bits as SID (in all unions or only when multiple enumerations or bits are defined)
- To encode enumerations and bits as delta between the value SID and the leaf SID (in all unions or only when multiple enumerations or bits are defined)

In this email, I will like to focus on the first option, fixing the problem directly in YANG instead of fixing the consequences.

Without any changes in YANG, a union with multiple enumeration or bits can be constructed without value or position overlaps.
For example:

  leaf multiple-enumerations-test-1 {
    type union {
      type enumeration {
        enum "Monday" { value 0; }
        enum "Tuesday" { value 1; }
        enum "Wednesday" { value 2; }
        enum "Thursday" { value 3; }
        enum "Friday" { value 4; }

      }
      type enumeration {
        enum "Saturday" { value 5; }
        enum "Sunday" { value 6; }
      }
    }
  }

  leaf multiple-bits-test-1 {
    type union {
      type bits {
        bit  "Monday" { position  0; }
        bit "Tuesday" { position  1; }
        bit "Wednesday" { position  2; }
        bit "Thursday" { position  3; }
        bit "Friday" { position  4; }

      }
      type bits {
        bit "Saturday" { position 5; }
        bit "Sunday" { position 6; }
      }
    }
  }

When using already defined typedef, avoiding overlap is less obvious without some help.
To help building unions with already defined typedefs, I propose to introduce two extensions. 

  extension value-offset {
    argument offset {
      yin-element true;
    }
    description
      "Offset added to each enum value of the associated enumeration.";
  }
  
  extension position-offset {
    argument offset {
      yin-element true;
    }
    description
      "Offset value added to each bit position of the associated bits.";
  }

The value-offset extension can be used as follow:

    type enumeration {
      enum "Monday";
      enum "Tuesday";
      enum "Wednesday";
      enum "Thursday";
      enum "Friday";
    }
  }

  typedef weekend {
    type enumeration {
      enum "Saturday";
      enum "Sunday";
    }
  }
  
  leaf multiple-enumerations-test-3 {
    type union {
      type weekdays;
      type weekend {
        ext:value-offset 5;
      }
    }
  }

The position-offset extension can be used as follow:

  typedef weekdays-flags {
    type bits {
      bit "Monday";
      bit "Tuesday";
      bit "Wednesday";
      bit "Thursday";
      bit "Friday";
    }
  }

  typedef weekend-flags {
    type bits {
      bit "Saturday";
      bit "Sunday";
    }
  }
  
  leaf multiple-bits-test-3 {
    type union {
      type weekdays-flags;
      type weekend-flags {
        ext:position-offset 5;
      }
    }
  }

The yang file in attachment show different examples based on this approach.
This module have been validated using http://www.yangvalidator.com/validator 
If this approach is accepted, tools like pyang should be updated to produce an error if multiple enumerations or bits are defined with value or position overleaps.

Please comment,
Michel

-----Original Message-----
From: Ladislav Lhotka <lhotka@nic.cz> 
Sent: Monday, March 25, 2019 4:07 AM
To: Juergen Schoenwaelder <j.schoenwaelder@jacobs-university.de>; Carsten Bormann <cabo@tzi.org>
Cc: Michel Veillette <Michel.Veillette@trilliant.com>; netconf@ietf.org; core@ietf.org
Subject: Re: [netconf] YANG encoding in CBOR

Juergen Schoenwaelder <j.schoenwaelder@jacobs-university.de> writes:

> I think we need to look at the whole picture and in which direction we 
> want to go. In the longer term, I would prefer a solution where the 
> values of a union are discriminated. The current XML encoding 
> behaviour of 'first match wins' is fragile (for example, if someone 
> adds an enum to a type, the interpretation of data can change).
>
> Look at this:
>
> typedef bar {
>   type union {
>     type enumeration { enum "1"; value 2; enum "2"; value 1; }
>     type uint8;
>   }
> }
>
> We have some encodings that send the string representations of the 
> values and some encodings that prefer to send numeric representations 
> where possible. In order to have a robust solution, encodings should 
> likely indicate to which type the value belongs.

Perhaps the easiest way would be to use (optional) annotation that specifies, using an ordinal number, which of the member types is used for the particular instance. But since there can be unions inside unions, a list of numbers would be needed in general.

Lada

>
> /js
>
> On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
>> Well, if that is a problem, we can go for a longer representation within unions (section 6.12).  Theoretically, we could do that only of there is more than one enum in the union type (so things stay efficient if there is only one), but that might pose difficulties with model evolution.
>> 
>> Going for a string representation repeats the feature of XML YANG (which was ported over to JSON YANG):
>> 
>> typedef foo {
>>   type union {
>>     type enumeration {
>>       enum red { value 1; }
>>       enum breen { value 2; }
>>       enum glue { value 3; }
>>     }
>>     type enumeration {
>>       enum tacks { value 1; }
>>       enum nails { value 2; }
>>       enum glue { value 3; }
>>     }
>>   }
>> }
>> 
>> If you use “glue”, you don’t know which of the enumerations are being used.
>> 
>> Using SIDs, we can do better.
>> 
>> So what do we have to do to get the SID tool to allocate SIDs for enum values?
>> 
>> We could then define the CBOR tag for enums in unions to take the usual SID difference (delta relative to the environment, I’d think), not an integer value.
>> 
>> Several of us are at the hackathon and could make something happen today and tomorrow.
>> 
>> Grüße, Carsten
>> 
>> 
>> > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <rwilton@cisco.com> wrote:
>> > 
>> > I guess that the concern is that this introduces more variation in how data is interpreted between the different XML/JSON/CBOR encodings.
>> > 
>> > E.g. if someone switched from XML to CBOR, suddenly the configuration or state data may have a different meaning.
>> > 
>> > Thanks,
>> > Rob
>> > 
>> > 
>> >> -----Original Message-----
>> >> From: Carsten Bormann <cabo@tzi.org>
>> >> Sent: 22 March 2019 16:08
>> >> To: Michel Veillette <Michel.Veillette@trilliant.com>
>> >> Cc: Rob Wilton (rwilton) <rwilton@cisco.com>; core@ietf.org; 
>> >> netconf@ietf.org
>> >> Subject: Re: [netconf] YANG encoding in CBOR
>> >> 
>> >> On Mar 22, 2019, at 16:45, Michel Veillette 
>> >> <Michel.Veillette@trilliant.com>
>> >> wrote:
>> >>> 
>> >>> The only potential problem I aware is when multiple enumerations 
>> >>> are part of
>> >> the same union.
>> >>> Value 4 from enumeration A will be encoded the same way as Value 
>> >>> 4 from
>> >> enumeration B.
>> >> 
>> >> … and that is not a problem for the XML version, because the 
>> >> string is being used instead of the value.  (But then if two 
>> >> enumerations share a string, you have the equivalent problem in 
>> >> the XML serialization.)
>> >> 
>> >> Anyway, I haven’t seen a piece of real-world YANG that actually 
>> >> has this problem, so I would be a bit reluctant to make CBOR-based 
>> >> implementations more complex (and less efficient) so solve this (non-?)problem.
>> >> 
>> >> Grüße, Carsten
>> > 
>> > 
>> > 
>> 
>> _______________________________________________
>> netconf mailing list
>> netconf@ietf.org
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>> .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
>> d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
>> %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
>> kvPZgr4o%3D&amp;reserved=0
>
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0>
>
>
> On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
>> Well, if that is a problem, we can go for a longer representation within unions (section 6.12).  Theoretically, we could do that only of there is more than one enum in the union type (so things stay efficient if there is only one), but that might pose difficulties with model evolution.
>> 
>> Going for a string representation repeats the feature of XML YANG (which was ported over to JSON YANG):
>> 
>> typedef foo {
>>   type union {
>>     type enumeration {
>>       enum red { value 1; }
>>       enum breen { value 2; }
>>       enum glue { value 3; }
>>     }
>>     type enumeration {
>>       enum tacks { value 1; }
>>       enum nails { value 2; }
>>       enum glue { value 3; }
>>     }
>>   }
>> }
>> 
>> If you use “glue”, you don’t know which of the enumerations are being used.
>> 
>> Using SIDs, we can do better.
>> 
>> So what do we have to do to get the SID tool to allocate SIDs for enum values?
>> 
>> We could then define the CBOR tag for enums in unions to take the usual SID difference (delta relative to the environment, I’d think), not an integer value.
>> 
>> Several of us are at the hackathon and could make something happen today and tomorrow.
>> 
>> Grüße, Carsten
>> 
>> 
>> > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <rwilton@cisco.com> wrote:
>> > 
>> > I guess that the concern is that this introduces more variation in how data is interpreted between the different XML/JSON/CBOR encodings.
>> > 
>> > E.g. if someone switched from XML to CBOR, suddenly the configuration or state data may have a different meaning.
>> > 
>> > Thanks,
>> > Rob
>> > 
>> > 
>> >> -----Original Message-----
>> >> From: Carsten Bormann <cabo@tzi.org>
>> >> Sent: 22 March 2019 16:08
>> >> To: Michel Veillette <Michel.Veillette@trilliant.com>
>> >> Cc: Rob Wilton (rwilton) <rwilton@cisco.com>; core@ietf.org; 
>> >> netconf@ietf.org
>> >> Subject: Re: [netconf] YANG encoding in CBOR
>> >> 
>> >> On Mar 22, 2019, at 16:45, Michel Veillette 
>> >> <Michel.Veillette@trilliant.com>
>> >> wrote:
>> >>> 
>> >>> The only potential problem I aware is when multiple enumerations 
>> >>> are part of
>> >> the same union.
>> >>> Value 4 from enumeration A will be encoded the same way as Value 
>> >>> 4 from
>> >> enumeration B.
>> >> 
>> >> … and that is not a problem for the XML version, because the 
>> >> string is being used instead of the value.  (But then if two 
>> >> enumerations share a string, you have the equivalent problem in 
>> >> the XML serialization.)
>> >> 
>> >> Anyway, I haven’t seen a piece of real-world YANG that actually 
>> >> has this problem, so I would be a bit reluctant to make CBOR-based 
>> >> implementations more complex (and less efficient) so solve this (non-?)problem.
>> >> 
>> >> Grüße, Carsten
>> > 
>> > 
>> > 
>> 
>> _______________________________________________
>> netconf mailing list
>> netconf@ietf.org
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>> .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
>> d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
>> %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
>> kvPZgr4o%3D&amp;reserved=0
>
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0>
>
> _______________________________________________
> netconf mailing list
> netconf@ietf.org
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> gr4o%3D&amp;reserved=0

--
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67