Re: [core] YANG Packages for CoMI

Alexander Pelov <a@ackl.io> Wed, 29 July 2015 08:33 UTC

Return-Path: <a@ackl.io>
X-Original-To: core@ietfa.amsl.com
Delivered-To: core@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 156981A879D for <core@ietfa.amsl.com>; Wed, 29 Jul 2015 01:33:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X6jt2rpYKxUX for <core@ietfa.amsl.com>; Wed, 29 Jul 2015 01:33:29 -0700 (PDT)
Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [IPv6:2001:4b98:c:538::197]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 486421A873D for <core@ietf.org>; Wed, 29 Jul 2015 01:33:28 -0700 (PDT)
Received: from mfilter16-d.gandi.net (mfilter16-d.gandi.net [217.70.178.144]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id 7638541C07A; Wed, 29 Jul 2015 10:33:26 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at mfilter16-d.gandi.net
Received: from relay5-d.mail.gandi.net ([IPv6:::ffff:217.70.183.197]) by mfilter16-d.gandi.net (mfilter16-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id J_7Bju_kUf2E; Wed, 29 Jul 2015 10:33:23 +0200 (CEST)
X-Originating-IP: 178.251.23.166
Received: from Zax.local (unknown [178.251.23.166]) (Authenticated sender: alex@ackl.io) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 375DE41C076; Wed, 29 Jul 2015 10:33:14 +0200 (CEST)
To: Andy Bierman <andy@yumaworks.com>
References: <CABCOCHT7J71iiAgcofradE38JepEJLsPkbmwtivY=HMyij1Svg@mail.gmail.com> <4dde82ad1cd4d261b598cb0e851cdf1c@xs4all.nl> <CABCOCHTukKKkP9wgP4BrtAcKwDd7eSLvFR_x2kJptW+Dfo_0Lw@mail.gmail.com> <55B6070A.1020007@ackl.io> <CABCOCHTOtNvPgAV9R7bnMQDZtTUOmtnadZrGn13X7diQUGseXw@mail.gmail.com> <55B72969.4050006@ackl.io> <CABCOCHQhYidrtDipzGeEj7HhscLy+70pneDNTKNh59PfXvCA6A@mail.gmail.com>
From: Alexander Pelov <a@ackl.io>
Message-ID: <55B88FC9.7070107@ackl.io>
Date: Wed, 29 Jul 2015 10:33:13 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.0.1
MIME-Version: 1.0
In-Reply-To: <CABCOCHQhYidrtDipzGeEj7HhscLy+70pneDNTKNh59PfXvCA6A@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------060605030801090703000306"
Archived-At: <http://mailarchive.ietf.org/arch/msg/core/iYrCaPFcloKAtPHAmbA73utCni4>
Cc: Core <core@ietf.org>
Subject: Re: [core] YANG Packages for CoMI
X-BeenThere: core@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Constrained RESTful Environments \(CoRE\) Working Group list" <core.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/core>, <mailto:core-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/core/>
List-Post: <mailto:core@ietf.org>
List-Help: <mailto:core-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/core>, <mailto:core-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Jul 2015 08:33:34 -0000

Hi Andy,

Le 29/07/2015 04:19, Andy Bierman a écrit :
>
>
> On Tue, Jul 28, 2015 at 12:04 AM, Alexander Pelov <a@ackl.io 
> <mailto:a@ackl.io>> wrote:
>
>     Hi Andy,
>
>     Le 27/07/2015 20:01, Andy Bierman a écrit :
>>
>>
>>     On Mon, Jul 27, 2015 at 3:25 AM, Alexander Pelov
>>     <alexander@ackl.io <mailto:alexander@ackl.io>> wrote:
>>
>>         Hi Andy!
>>
>>         Thanks for the draft! Seems a good start to get the things
>>         moving in the direction of having a good, structured
>>         introspection mechanism for module discovery!
>>
>>         I'm not completely sure that it will be a static document.
>>         You still need to think of the possibility of a firmware
>>         upgrade, in which case you have to consider the clients which
>>         have obtained your previous package descriptions.
>>
>>
>>     Each package has a name and a revision-date.
>>     The expected use for CoMI would be a new
>>     package revision for each firmware release.
>     I meant that there should be a mechanism for the client to be
>     notified when the YANG package definition changes. This may be a
>     rare event, but imagine a new client arrives and discovers the
>     server 1 minute before the server gets a new firmware release (or
>     some other event, making it change its package definition).
>
>
>
> I agree.
> There can be a YANG Package Library URI
>
>   /mg/pkg.uri
>
> URI of the YANG Package Library resource to GET or Observe
>
> The package data tree could be much smaller than the module library tree.
>
>   container packages {
>      leaf pkg-set-id { type uint32; }
>      list package {
>         key name;
>         leaf name { type string; }
>         leaf rev { type yang:revision-date; }
>      }
>    }
>
>
>   { "packages": {
>      "pkg-set-id":42,
>      "package": [
>        {
>           "name":"example-sensor314",
>           "rev":"2015-06-14"
>        }
>      ]
>    }
>  }
>

I suppose in a YANG hash world you need to multiply the file locations 
so that you can have this information in a convenient place. In a 
numbering scheme with managed IDs you can allocate predefined module IDs 
for the important modules. This one seems to be of great importance!

Do you think that we could re-use the RFC 6022 - YANG module for NETCONF 
monitoring? It seems to me that it does exactly what the YANG package 
you are proposing.

In both cases (packages or RFC 6022) if we have the introspection 
information exposed as YANG module, we don't need to invent additional 
mechanisms. By performing a select/fields filtering, a client can obtain 
in a lightweight fashion the information that concerns it. If the client 
wants, it could obtain the full schema information, but that is not 
necessary.

>
>
>     I would think that if a client is expected to keep long-term
>     association to the server, it should use Observe or NOTIFY. The
>     latter will work if the package is defined as module.
>
>>
>>         Why don't you want it to be a YANG module? I would suppose
>>         that for a structured numbering scheme (such as CoOL) we
>>         could have a nice fixed entry point for such introspection
>>         resource, e.g. moduleID = 1. Alternatively, I would think
>>         that it should be discoverable through .well-known/core
>>
>>
>>     Not sure what you mean.
>>     This is static data used to identify a set of YANG modules.
>>     Do you mean why not have the contents of a package be YANG objects
>>     to read from the server?  This would be even more data than the
>>     current YANG module library.
>>
>>     The point of a package list is to avoid listing all the modules,
>>     features,
>>     and revision dates.  There is a YANG Package Identifier in sec. 4
>>
>>        urn:ietf:params:xml:ns:yang:pkg?name=example-sensor314&rev=2015-07-01
>>
>>     This would be the entire response needed to discover the entire YANG
>>     implementation by the server. The size remains the same no matter
>>     how many
>>     modules and features are added over time.
>
>     Ah, OK! Now I understand! It was not obvious to me that you were
>     aiming at having the package definition be detached from the
>     server. I like that! This, however makes me think about something
>     else - you should have a way to obtain the package from the
>     sensor. However, this goes a little bit in the sense of having
>     YANG packages being mandatory for CoMI. I like the "uses-package"
>     statement, as this could be used to have full definition of the
>     modules on a server.
>
>     I would suppose that you could represent the information of a YANG
>     package as a YANG module. Then, a client could query the server
>     with a select/fields statement to obtain only the package
>     identifier. IF the client wants, it could obtain the whole package
>     content from the server:
>
>     E.g. for a YANG module definition such as:
>     module yang-package {
>       ...
>       leaf yang-package-uri {
>         type string;
>       }
>
>       container package {
>         leaf organization {type string;}
>         leaf contact {type string;}
>         leaf description {type string;}
>         leaf revision {type string;}
>         container imports-module {
>            key module-name;
>            leaf module-name {type string;}
>            leaf module-cool-id {type int32;}
>         }
>         container uses-module {
>            ...
>         }
>         ...
>       }
>     }
>
>
>
> The point of the YANG package is that it serves as an offline handle
> for lots of YANG details related to a set of modules.
>
> NETCONF and RESTCONF just advertise the conformance details
> for each module (name, revision, features, deviations). There is
> a <get-schema> operation in both protocols to allow the client to retrieve
> the actual YANG files.  There is no YANG module that modules
> YANG module contents.
>
> The same approach <get-pkg-schema> should be used in CoMI,
> except I would expect a centralized server to have that data,
> not any constrained servers.

I don't understand the need of modeling the YANG module contents. Once 
you have the conformance details (name, revision, features, deviations), 
you can try and fetch them from an remote store (or have them locally). 
If you don't have access to this information, you need to fetch their 
schema. Do you see an intermediate level of details?

For me, the introspection mechanism should be implemented as a YANG 
module. There is no need to invent new mechanisms, when we have an 
existing one. I would imagine that you could declare the YANG hash of 
this module in the .well-known/core if you want to be flexible.

Best,
Alexander

>
>
>
>
>     module ID for this module would be fixed in CoOL to 1, so
>     yang-package-uri will have CoOL ID 2049.
>
>     In CoOL this would be as getting :
>     GET /cdat?fields(2049)
>
>     Response:
>     {
>     2049:"urn:ietf:params:xml:ns:yang:pkg?name=example-sensor314&rev=2015-07-01"
>     }
>
>
>     Maybe this could be even more optimized.
>
>
>
> I don't know if package numbers are really needed.
> The entire package library above would fit in 128 bytes.
>
>
>
> Andy
>
>
>     Now, if the client has no idea how to find this URN, it would
>     simply  GET /cdat?fields(2048) and retrieve the YANG package
>     information from the server.
>
>>
>>
>>
>>
>>         A point to make is that we can have alias mappings through
>>         this introspection mechanism. Defining such an introspection
>>         mechanism for the aliases was actually one of the work I was
>>         considering for CoOL.
>>
>>
>>     RESTCONF and NETCONF provide the ability for the client to retrieve
>>     the schema, and get all the meta-data, for every module.
>>     IMO this should be done by a centralized server in CoMI.
>>
>>     But I agree the package data could include alias mappings.
>>     This might be a way to convert YANG Hash to COOL for example,
>>     to optimize frequently used identifiers..
>
>     Maybe this should also be the way to handle alias attribution,
>     e.g. add a "alias" statement, which indicates the mapping alias-id
>     -> CoOL id / CoMI hash / full identifier.
>
>     module yang-package {
>     ....
>       container alias {
>         key cool-id;
>         leaf cool-id {type int32};
>         leaf alias-id {type int32};
>       }
>     }
>
>     So you can GET /cdat?fields(2058) and have
>     {
>       2058: [
>         {11: 29834394, 12:1},
>         {11: 99999999, 12:2}
>         {11: 88888384, 12:3}
>       ]
>     }
>
>     assuming the following CoOL id automatic attribution:
>     alias -> node ID=10
>     cool-id -> node Id=11
>     alias-id -> node ID=12
>     and some random IDs to be mapped (29834394, 99999999, 88888384).
>
>     Ideally, of course, you would be fetching this information from a
>     server somewhere on the Internet, so that you don't fetch from
>     your constrained server.
>
>     Best,
>     Alexander
>
>>
>>
>>
>>
>>         Best,
>>         Alexander
>>
>>
>>     Andy
>>
>>
>>         Le 27/07/2015 08:29, Andy Bierman a écrit :
>>>
>>>
>>>         On Sun, Jul 26, 2015 at 11:26 PM, peter van der Stok
>>>         <stokcons@xs4all.nl <mailto:stokcons@xs4all.nl>> wrote:
>>>
>>>             HI Andy,
>>>
>>>             Is the proposal to make YANG packages compulsory over
>>>             yang-library in Comi servers and clients?
>>>
>>>
>>>         No, it would be optional.
>>>         Another solution approach would be to make the module-set-id
>>>         values
>>>         global instead of per-server. That might be less work perhaps.
>>>
>>>
>>>             Peter
>>>
>>>
>>>         Andy
>>>
>>>
>>>             Andy Bierman schreef op 2015-07-26 19:44:
>>>
>>>                 Hi,
>>>
>>>                 Michel raised some issues with the YANG library draft
>>>                 http://datatracker.ietf.org/doc/draft-ietf-netconf-yang-library/
>>>
>>>                 Maybe CoMI needs its own optimized module library.
>>>                 I am concerned that data sizes of the YANG library
>>>                 will be
>>>                 too big in some really constrained environments.
>>>
>>>                 I wrote a draft that defines a way to specify the
>>>                 contents of the
>>>                 YANG library in a single static document, called a
>>>                 YANG package.
>>>                 http://datatracker.ietf.org/doc/draft-bierman-netmod-yang-package/
>>>
>>>                 If there was a "shorthand" resource that listed YANG
>>>                 packages, instead
>>>                 of YANG modules, then the CoMI client that supported
>>>                 YANG packages
>>>                 could read that instead of the module library.
>>>
>>>                 The YANG package message response can identify a
>>>                 single package
>>>                 (e.g. match the firmware) so the size of the
>>>                 response will remain very
>>>                 small, and not depend on the entire list of modules,
>>>                 features,
>>>                 and deviations. The data savings could 1000X for 100
>>>                 modules.
>>>
>>>                 The client needs to retrieve the library first-time
>>>                 and anytime it
>>>                 changes.
>>>                 The module-set-id values are per-server, not global.
>>>                 This could be a
>>>                 significant bottleneck when a CoMI client starts or
>>>                 restarts, and
>>>                 manages lots of servers.
>>>
>>>                 Andy
>>>
>>>
>>>                 _______________________________________________
>>>                 core mailing list
>>>                 core@ietf.org <mailto:core@ietf.org>
>>>                 https://www.ietf.org/mailman/listinfo/core
>>>
>>>
>>>
>>>
>>>         _______________________________________________
>>>         core mailing list
>>>         core@ietf.org <mailto:core@ietf.org>
>>>         https://www.ietf.org/mailman/listinfo/core
>>
>>
>
>