Re: [core] progressing ietf-core-yang-cbor and ietf-core-sid

[Ivaylo Petrov <[e-mail]>]

Hello,

In this email I am presenting the change that I am proposing regarding
".sid" file versioning
that I believe we all agreed needs to be supported (the new version is
available in attachment).

I am planning on uploading the new version early next week. Please let
me know if you have any questions
comments or remarks or if you would need more time to review it and
you would want me to wait for a little bit.

Add ".sid" file version to the YANG file code snippet

  typedef sid-file-version-identifier {
    type uint64;
    description
      "Optional attribute that gives information about the \".sid\" file
       version.";
  }

  leaf sid-file-version {
    type sid-file-version-identifier;
    description
      "Version number of the \".sid\" file. Useful to distinguish \".sid\"
       files generated using different YANG module dependencies that might not
       be reflected in the YANG module revision.";
  }
  list dependencies-revisions {
    key "module-name";
    description
      "Information about the revision of each YANG module that the module in
       'module-name' includes used during the \".sid\" file generation.";
    leaf module-name {
      type yang:yang-identifier;
      mandatory true;
      description
        "Name of the YANG module, dependency of 'module-name', for which
         revision information is provided.";
    }
    leaf module-revision {
      type revision-identifier;
      mandatory true;
      description
        "Revision of the YANG module, dependency of 'module-name', for which
         revision information is provided.";
    }
  }

Modifying sec 3.

 Each time a YANG module or one of its imported module(s) or included
-sub-module(s) is updated, the ".sid" file MAY need to be updated. This update
-SHOULD be performed using an automated tool.
+sub-module(s) is updated, a new ".sid" file MAY need to be created. All the
+SIDs present in the previous version of the ".sid" file MUST be present in the
+new version as well. The creation of this new version of the ".sid" file SHOULD
+be performed using an automated tool.

In sec 6.4.2 add

* If there is an older version of the ".sid" file, all allocated SIDs from that
  version are still present in the current version of the ".sid" file.

In Appendix B add

5. The "dependencies-revisions" should reflect the revision numbers of each
   YANG module that the YANG module imports at the moment of the generation.

and again in Appendix B modify

-Changes of SID files can also be automated using the same method
described above, only unassigned YÀNG items are processed at step #3.
Already existing items in the SID file should not be given new SIDs.
+In case of an update to an existing ".sid" file, an additional step is needed
+that increments the ".sid" file version number. If there was no version number
+in the previous version of the ".sid" file, 0 is assumed as the version number
+of the old version of the ".sid" file and the version number is 1 for the new
+".sid" file. Apart from that, changes of SID files can also be automated using
+the same method described above, only unassigned YÀNG items are processed at
+step #3. Already existing items in the SID file should not be given new SIDs.

Best regards,
Ivaylo


On Thu, Jan 23, 2020 at 11:14 AM Ivaylo Petrov <[e-mail]> wrote:
>
> Find my answers below
>
> On Wed, Jan 22, 2020 at 09:52:43AM -0800, Andy Bierman wrote:
> > On Wed, Jan 22, 2020 at 3:44 AM Ivaylo Petrov <[e-mail]> wrote:
> >
> > > Thank you for your clarification, I think we are getting closer to
> > > understanding the difference of viewpoints. Find my comments below.
> > >
> > > On Tue, Jan 21, 2020 at 8:29 PM Andy Bierman <[e-mail]> wrote:
> > > >
> > > >
> > > >
> > > > On Tue, Jan 21, 2020 at 2:55 AM Ivaylo Petrov <[e-mail]> wrote:
> > > >>
> > > >> Hello Andy,
> > > >>
> > > >> Find my answers below
> > > >>
> > > >> On Sun, Jan 19, 2020 at 6:37 AM Andy Bierman <[e-mail]>
> > > wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Jan 16, 2020 at 9:47 AM Carsten Bormann <[e-mail]> wrote:
> > > >> >>
> > > >> >> On 2020-01-16, at 13:50, Andy Bierman <[e-mail]> wrote:
> > > >> >> >
> > > >> >> > There is a significant amount of metadata needed to generate a SID
> > > file,
> > > >> >> > starting with all the previous YANG revisions or SID files, and
> > > when/if
> > > >> >> > the SID files have been reset vs. updated with preserved numbers.
> > > >> >>
> > > >> >> Well, specifically, the current YANG module and the previous SID
> > > file; not the entire history of the universe.
> > > >> >>
> > > >> >> The need for the previous SID file means that there needs to be a
> > > single lineage; SID files cannot be independently progressed.  This calls
> > > for central entity (or a blockchain :-).
> > > >> >> What we need to define unambiguously is who that entity is.
> > > >> >> One we are at the IANA, the Designated Expert (DE) can play this
> > > role.
> > > >> >> But we need to define the process leading up to this.
> > > >> >> (Yes, that process will evolve, and we probably also need to allow
> > > exceptions.
> > > >> >> But there must be strong guide rails that work for the vast majority
> > > of cases.)
> > > >> >>
> > > >> >> Adding a version number to the SID file could indeed help with
> > > accidents/disasters.
> > > >> >> But it could also lead to people thinking they can get away with
> > > concurrent lineages; thinking that version numbers will fix any fallout.
> > > >> >> That would be a disaster on its own, most likely leading to
> > > permanent universe splits.
> > > >> >> So I’m not so sure we want to open that Pandora’s box.
> > > >> >>
> > > >> >
> > > >> >
> > > >> > We already opened Pandora's box by attempting to create a
> > > 1-flat-number globally unique, multi-module,
> > > >> > multi-owner object identifier. I think YANG is quite different than
> > > the MAC address example of success
> > > >> > in this endeavour because YANG modules need the numbers assigned in
> > > the development phase,
> > > >> > not the manufacturing phase.
> > > >> >
> > > >> > YANG has many features that make it easier on the writer and harder
> > > on the tool maker,
> > > >> > such as no order dependencies, but is has some guardrails. No matter
> > > how the writer may
> > > >> > possibly alter or refactor a YANG module, nothing unintentional can
> > > change the YANG identifier for
> > > >> > any defined data node.  SID has no such guardrails.
> > > >>
> > > >> [ivaylo]:
> > > >> It seems that you are implying that refactoring a YANG without changing
> > > the
> > > >> YANG identifiers and the paths will cause changes to the .sid file.
> > > Could you
> > > >> provide an example as I am not sure I see why this needs to be the case.
> > > >> Note that I added the restriction of not changing paths as otherwise I
> > > >> believe the
> > > >> NETCONF representation will also change, so CORECONF will be no
> > > different.
> > > >>
> > > >
> > > >
> > > > YANG modules under development do not follow any of the "module update"
> > > rules
> > > > so there are no restrictions that can be relied on in real YANG
> > > development.
> > > > This requires the tools and developers to be more careful, especially
> > > > since YANG tools have no existing reason to be careful in these areas.
> > > >
> > >
> > > [ivaylo]:
> > > If there are no restrictions to what a YANG module change can bring, I
> > > am afraid we can not provide much help for .sid files. The suggestion
> > > that any implementer of pre-WGLC version of the draft might need to
> > > adjust her implementation due to SIDs change seems acceptable to me and
> > > I don't see how we can provide anything better.
> > >
> > >
> >
> > I think that WGs will want to regenerate the files during development, even
> > though
> > it often takes the IETF 2 - 5 years to complete a YANG module.  Nodes get
> > renamed a lot
> > and every time that is done it creates new SIDs and leaves behind
> > irrelevant SIDs.
> > It might takes 5x - 10x the required range by the time the WG is done.
> >
> > It is not even clear that I-Ds (or RFCs) can include SID files, since they
> > rely on
> > the imports used to generate the file.  I agree that there is not much that
> > can (or should)
> > be done to pretend that work-in-progress has any stability.
> >
>
> [ivaylo]:
> Maybe I failed to provide enough details and I might have been
> misunderstood here, sorry if this is the case, but my idea was that
> before WG adoption all bets are off for stability of a .sid file. Once a
> draft is adopted, the authors should make sure they don't regenerate the
> .sid file from scratch for every little change. While renaming could be
> done by changing the .sid file (automatically or not) making the old SID
> value point to the new name, that might cause other problems. Unless
> someone believes this is something we should explicitly allow, I would
> prefer to let the authors decide if this would be safe in they case or
> not. Once WGLC is done, the .sid file should not reallocate SID values
> unless this is discussed on the mailing list. I will add text about this
> in the draft this week and publish a new version.
>
> >
> >
> >
> > > >
> > > >>
> > > >> >
> > > >> > I think there is a practical solution for deployment, which does not
> > > require the use of the
> > > >> > centrally maintained SID files, which is not realistic for 4 reasons:
> > > >> >   - no way to apply changes to an incorrect SID file (without a
> > > sid-file-identifier)
> > > >>
> > > >> [ivaylo]:
> > > >> I believe currently the way to achieve this is to have a new version
> > > >> of the YANG module,
> > > >> so that a new version of the SID file can be generated. Having a
> > > >> global version of .sid
> > > >> file that is an integer that progresses lineary seems as a good
> > > >> solution to me. It should
> > > >> also be able to accommodate your concern here I believe. As Michael
> > > >> said, the latest
> > > >> version of the .sid file should be able to handle both all clients of
> > > >> previous versions of
> > > >> the YANG module and the latest one. As there could be multiple versions
> > > for one
> > > >> version of a YANG module, errors could be corrected. Does this sound
> > > >> good to you?
> > > >
> > > >
> > > >
> > > > This will not work at all.
> > > > There is no reason to update the YANG module.
> > > > What would change? Just the revision-date?
> > > >
> > > > Even if we ignore all possibility that a SID file can have a flaw in it,
> > > > there is still the problem caused when imported modules change.
> > > > Reusable groupings in a common module are all the rage.
> > > > It is very possible that the target module may never even change after
> > > release 1
> > > >
> > > >    module foo {
> > > >         ...
> > > >         import A { prefix a; }
> > > >         revision 2019-01-01;
> > > >
> > > >         container top {
> > > >            uses a:grouping1;
> > > >        }
> > > >   }
> > > >
> > > > The SID file contents for /top are not determined by module foo.
> > > > Over time, module A will keep changing and module foo will never change.
> > > > All of the unspecified revisions imported by module foo determine the
> > > expansion of 'grouping1.
> > > > This approach only works within a server implementation because all the
> > > module revisions
> > > > are known to the client via the YANG library.
> > > >
> > >
> > > [ivaylo]
> > > I think the difference of viewpoints here came from my assumption that
> > > all yang modules import specific versions of other modules. You point
> > > out in your email that in practice this is not necessarily the case. If
> > > we want to be able to handle this as I believe we are, I agree that we
> > > might need to add information in the SID file about what version of each
> > > of the dependent modules had been used to allocate any given SID as
> > > otherwise we might have serious problems figuring this out at runtime,
> > > if this is at all possible.
> > >
> > > Storing this information can be solved by having many independent
> > > versions of .sid files - one for a given set of YANG modules and
> > > revisions or we can have each SID number allocation contain more
> > > information than just the path to that node - namely the versions of all
> > > the YANG modules and the revision information.  In both cases it appears
> > > that this will complicate how SIDs allocation.
> > >
> > > Andy, do you agree that I have captured the essence of your concern
> > > correctly? Are there any thoughts whether we want to solve this problem
> > > and which is the best approach (even if it is not any of the ones
> > > already mentioned)?
> > >
> > > I might be missing some historical background here, but my personal
> > > preference is to go with the second approach as it would allow for
> > > linear versioning of the .sid files, which I believe is lighter on the
> > > constrained device. It seems that your preference, Andy, is to go with
> > > the first one and have SIDs be more contextually dependent.
> > >
> > >
> >
> > I don't see how the 2nd approach can scale to even 20 YANG modules, let
> > alone 100s.
> > If there are multiple variants of every SID file, then there is no point in
> > an "official" SID file
> > at all. Only the mappings for a specific server implementation can be
> > used.  Even products
> > from the same vendor will use different revisions of the same imported
> > module.
> >
> > It is quite possible that common SID files will be practical after
> > development is done
> > for a module and it is quite stable. There will probably be a mix of
> > deployment strategies
> > depending on module maturity and implementation details.
>
> [ivaylo]:
> After some more consideration, I agree that it adds a lot of complexity
> for something that I am not sure we want to support as a first class
> citizen. I agree that we should have support for handling multiple
> versions of the dependencies, where I am not that sure is that we should
> make this as effortless as possible if that would add that much
> complexity. A simpler approach that would fit that bill for me is that
> each .sid file version N has all the SIDs present in version N-1 without
> changes for any N>0 (assuming that 0 is the initial version). The .sid file
> shall also have a field with the list of versions of all the modules that
> were used to generate the current version of that .sid file. That way
> only traversing the .sid file versions one can discover what versions of
> each YANG module were used for the generation of each SID. At the same
> time we don't add too much more information to each .sid file. As
> Carsten suggested, the latest version always wins.
>
> The drawback I see is that we might carry around a number of SIDs
> that are no longer needed. While in some situations that might be
> unnecessary, I believe there is no way to avoid it without risking
> ambiguity or requesting a client to do even more work.
>
> >
> >
> > > >
> > > >
> > > >>
> > > >>
> > > >> >   - no way for vendors to apply deviations on any YANG modules, e,.g
> > > deviation { not-supported; }
> > > >>
> > > >> [ivaylo]:
> > > >> I am not sure I see why that is the case. In sec 3 of
> > > >> draft-ietf-core-yang-library we have
> > > >>
> > > >> * 'feature' and 'deviation' are implemented using a SID (i.e. an
> > > >> unsigned integer).
> > > >>
> > > >> While I agree that more details can be added here, I don't see any
> > > >> reason deviations
> > > >> not to be supported.
> > > >>
> > > >> >   - the SID file generation depends on the specific revisions of all
> > > modules imported
> > > >> >     and submodules included by the target YANG module
> > > >>
> > > >> [ivaylo]:
> > > >> I agree, but I believe this is a good thing.  As is said sec 5.1.1 of
> > > RFC6020
> > > >>
> > > >> ` modules need to be imported using specific revisions`.
> > > >
> > >
> >
> >
> >
> > The YANG import-by-revision mechanism is fatally flawed and has been mostly
> > worthless
> > since Day One.  Perhaps this will be fixed someday but until then, most
> > designers do
> > not specify the exact revision that must be used.
> >
>
> [ivaylo]:
> I believe my latest proposal will handle appropriately that problem.
>
> >
> >
> > > >
> > > >
> > > > This is not how YANG is used 95% of the time.
> > > > Even if I import-by-revision, it is very common for that module to
> > > > have its own imports, and I have no control over the revisions used
> > > there.
> > > >
> > > >
> > > >>
> > > >>
> > > >> Could you provide more details of the issue you see?
> > > >>
> > > >> >   - vendors are already failing to deploy YANG modules from a central
> > > source model.
> > > >> >     The NETCONF and RESTCONF protocols have robust mechanisms to
> > > discover and
> > > >> >     retrieve the actual YANG files used by the server, and vendors
> > > rely on this feature.
> > > >> >     Some vendors need model branching and other complexity that make
> > > SID assignment even harder
> > > >> >     (see
> > > https://tools.ietf.org/html/draft-ietf-netmod-yang-versioning-reqs-02)
> > > >> >
> > > >>
> > > >> [ivaylo]:
> > > >> I believe that the envisioned .sid file retrieval mechanism is:
> > > >> 1. You find the sid that you are interested in
> > > >> 2. You check in which mega range it is and who is the operator of that
> > > >> mega range in
> > > >> the IANA registry.
> > > >> 3. You contact that mega range operator through a given protocol to
> > > >> obtain further details.
> > > >>
> > > >
> > > >
> > > > No. I am interested in loading the SID data from each individual server,
> > > as needed.
> > > >
> > > > I understand some people want to hard-wire SIDs into their client or
> > > server
> > > > and they will not have the YANG identifiers.  I am not interested in this
> > > > deployment option for YANG-Push so the SID values will get added at
> > > runtime.
> > > > The client will consume or use cached the SID files at
> > > session-startup-time, just like the YANG modules.
> > >
> > > [ivaylo]:
> > > You are correct that I was looking much more from the perspective of
> > > hard-wiring the SIDs on constrained devices and assume that
> > > introspection will be done as needed using the central IANA registry or
> > > other server with such information. If you want to know what .sid file
> > > is present for each server, wouldn't the mechanism from
> > > draft-ietf-core-yang-library solve that? Or maybe you would like to have
> > > a more compact reply that gives only the YANG module name, version and
> > > version the SID file that can be found on a server - be it internal or
> > > external one. If that is the need you have, we can see which is the best
> > > approach to resolve it.
> > >
> >
> >
> >
> > You can hardwire SIDs based on proprietary vendor information or IANA
> > information.
> > I don't see how IANA can maintain any SID files though, since the entire
> > set of imported modules
> > depends on the SID file generation date and modules available to the tools
> > used by by
> > authors submitting files to IANA
> >
>
> [ivaylo]:
> The proposal to keep track of the YANG module versions used to generate
> any given .sid file handles this concern I believe.
>
> >
> >
> > > >
> > > >
> > > >>
> > > >> Presently we believed as there are not going to be too many mega range
> > > >> registries and
> > > >> we have no experience operating them it might be unwise to specify the
> > > >> protocol that
> > > >> they need to implement in order for clients to be able to query them.
> > > >> Specifying such a
> > > >> protocol might be something we do if this is a really big concern for
> > > people.
> > > >> Would that retrieval mechanism cover the cases that you were thinking
> > > >> of? Are there other
> > > >> cases that you believe it should cover?
> > > >
> > > >
> > > >
> > > > I think vendors will continue to fill in the gaps with proprietary
> > > mechanisms.
> > > > The SID file format will get augmented, etc. to support real deployment
> > > scenarios.
> > > > This could be considered a protocol issue and can be ignored for SID
> > > files in general.
> > > >
> > > > How the client gets the YANG and SID files could be considered an
> > > implementation detail.
> > > > The IANA repo is there but there is no requirement to use it.
> > > >
> > >
> > > [ivaylo]:
> > > I assume that one of the participants is a constrained device, which
> > > might not be exactly what you are considering, but in that case if the
> > > constrained device is a client, it is very unlikely for it to try to
> > > discover at runtime what are the SIDs that a server will support. It
> > > will generally send CORECONF requests that contain some SIDs and assume
> > > that the server or a proxy in between will be able to understand their
> > > meaning. That is where the IANA repository and a protocol for
> > > inspection how a SID maps can be useful.
> > >
> > > Then assuming that the constrained device is the server, a client can
> > > discover the supported SIDs either through introspection protocol, which
> > > will be heavy, but might be required in some cases or more likely
> > > through provisioning/bootstrap.
> > >
> > > It seems to me that you are considering also the question if none of the
> > > endpoints is really constrained. In that case I believe the client will
> > > still be able to use the same introspection protocol as before and find
> > > the YANG modules looking at the SID file.
> > >
> > > >
> > > >>
> > > >>
> > > >> > CORECONF (and SID in general) needs a SID file retrieval mechanism.
> > > >> > Perhaps based on YANG Data Instance Files
> > > https://tools.ietf.org/html/draft-ietf-netmod-yang-instance-file-format-06
> > > >> > A server will make the URI of this resource known somehow, which can
> > > be used if needed,
> > > >> > before a client starts using SIDs.
> > > >> >
> > > >> > In addition, or instead of listing YANG modules, the data is SID
> > > files, maybe 1 super-SID-file JSON object.
> > > >> > It is not realistic that a client will be able to use all global SID
> > > files based on the YANG module name
> > > >> > and revision date, especially in the development phase. The IANA SID
> > > file may be useful after the YANG
> > > >> > module is very stable.
> > > >> >
> > > >> > More solution details:
> > > >> >   - The WG or vendor can maintain an informal public archive of SID
> > > files.
> > > >> >   - The YangModels repo update process should include SID files.
> > > https://github.com/YangModels/yang
> > > >> >   - Vendors still need the public SID file for the starting point, or
> > > maybe use it unchanged
> > > >> >   - The SID files need <CODE BEGINS> support and probably some tool
> > > changes in the ID-submission process
> > > >> >   - IANA will only archive the RFC version, just like YANG modules.
> > > >> >
> > > >> >
> > > >> >
> > > >> >> Grüße, Carsten
> > > >> >
> > > >> >
> > > >> >
> > > >> > Andy
> > > >> >
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Ivaylo
> > > >
> > > >
> > > >
> > > > Andy
> > > >
> > >
> > > --
> > > Best regards,
> > > Ivaylo
> > >
> >
> >
> > Andy
>
> --
> Best regards,
> Ivaylo