Re: [core] progressing ietf-core-yang-cbor and ietf-core-sid

[Andy Bierman <[e-mail]>]

On Wed, Jan 22, 2020 at 3:44 AM Ivaylo Petrov <[e-mail]> wrote:

> Thank you for your clarification, I think we are getting closer to
> understanding the difference of viewpoints. Find my comments below.
>
> On Tue, Jan 21, 2020 at 8:29 PM Andy Bierman <[e-mail]> wrote:
> >
> >
> >
> > On Tue, Jan 21, 2020 at 2:55 AM Ivaylo Petrov <[e-mail]> wrote:
> >>
> >> Hello Andy,
> >>
> >> Find my answers below
> >>
> >> On Sun, Jan 19, 2020 at 6:37 AM Andy Bierman <[e-mail]>
> wrote:
> >> >
> >> >
> >> >
> >> > On Thu, Jan 16, 2020 at 9:47 AM Carsten Bormann <[e-mail]> wrote:
> >> >>
> >> >> On 2020-01-16, at 13:50, Andy Bierman <[e-mail]> wrote:
> >> >> >
> >> >> > There is a significant amount of metadata needed to generate a SID
> file,
> >> >> > starting with all the previous YANG revisions or SID files, and
> when/if
> >> >> > the SID files have been reset vs. updated with preserved numbers.
> >> >>
> >> >> Well, specifically, the current YANG module and the previous SID
> file; not the entire history of the universe.
> >> >>
> >> >> The need for the previous SID file means that there needs to be a
> single lineage; SID files cannot be independently progressed.  This calls
> for central entity (or a blockchain :-).
> >> >> What we need to define unambiguously is who that entity is.
> >> >> One we are at the IANA, the Designated Expert (DE) can play this
> role.
> >> >> But we need to define the process leading up to this.
> >> >> (Yes, that process will evolve, and we probably also need to allow
> exceptions.
> >> >> But there must be strong guide rails that work for the vast majority
> of cases.)
> >> >>
> >> >> Adding a version number to the SID file could indeed help with
> accidents/disasters.
> >> >> But it could also lead to people thinking they can get away with
> concurrent lineages; thinking that version numbers will fix any fallout.
> >> >> That would be a disaster on its own, most likely leading to
> permanent universe splits.
> >> >> So I’m not so sure we want to open that Pandora’s box.
> >> >>
> >> >
> >> >
> >> > We already opened Pandora's box by attempting to create a
> 1-flat-number globally unique, multi-module,
> >> > multi-owner object identifier. I think YANG is quite different than
> the MAC address example of success
> >> > in this endeavour because YANG modules need the numbers assigned in
> the development phase,
> >> > not the manufacturing phase.
> >> >
> >> > YANG has many features that make it easier on the writer and harder
> on the tool maker,
> >> > such as no order dependencies, but is has some guardrails. No matter
> how the writer may
> >> > possibly alter or refactor a YANG module, nothing unintentional can
> change the YANG identifier for
> >> > any defined data node.  SID has no such guardrails.
> >>
> >> [ivaylo]:
> >> It seems that you are implying that refactoring a YANG without changing
> the
> >> YANG identifiers and the paths will cause changes to the .sid file.
> Could you
> >> provide an example as I am not sure I see why this needs to be the case.
> >> Note that I added the restriction of not changing paths as otherwise I
> >> believe the
> >> NETCONF representation will also change, so CORECONF will be no
> different.
> >>
> >
> >
> > YANG modules under development do not follow any of the "module update"
> rules
> > so there are no restrictions that can be relied on in real YANG
> development.
> > This requires the tools and developers to be more careful, especially
> > since YANG tools have no existing reason to be careful in these areas.
> >
>
> [ivaylo]:
> If there are no restrictions to what a YANG module change can bring, I
> am afraid we can not provide much help for .sid files. The suggestion
> that any implementer of pre-WGLC version of the draft might need to
> adjust her implementation due to SIDs change seems acceptable to me and
> I don't see how we can provide anything better.
>
>

I think that WGs will want to regenerate the files during development, even
though
it often takes the IETF 2 - 5 years to complete a YANG module.  Nodes get
renamed a lot
and every time that is done it creates new SIDs and leaves behind
irrelevant SIDs.
It might takes 5x - 10x the required range by the time the WG is done.

It is not even clear that I-Ds (or RFCs) can include SID files, since they
rely on
the imports used to generate the file.  I agree that there is not much that
can (or should)
be done to pretend that work-in-progress has any stability.




> >
> >>
> >> >
> >> > I think there is a practical solution for deployment, which does not
> require the use of the
> >> > centrally maintained SID files, which is not realistic for 4 reasons:
> >> >   - no way to apply changes to an incorrect SID file (without a
> sid-file-identifier)
> >>
> >> [ivaylo]:
> >> I believe currently the way to achieve this is to have a new version
> >> of the YANG module,
> >> so that a new version of the SID file can be generated. Having a
> >> global version of .sid
> >> file that is an integer that progresses lineary seems as a good
> >> solution to me. It should
> >> also be able to accommodate your concern here I believe. As Michael
> >> said, the latest
> >> version of the .sid file should be able to handle both all clients of
> >> previous versions of
> >> the YANG module and the latest one. As there could be multiple versions
> for one
> >> version of a YANG module, errors could be corrected. Does this sound
> >> good to you?
> >
> >
> >
> > This will not work at all.
> > There is no reason to update the YANG module.
> > What would change? Just the revision-date?
> >
> > Even if we ignore all possibility that a SID file can have a flaw in it,
> > there is still the problem caused when imported modules change.
> > Reusable groupings in a common module are all the rage.
> > It is very possible that the target module may never even change after
> release 1
> >
> >    module foo {
> >         ...
> >         import A { prefix a; }
> >         revision 2019-01-01;
> >
> >         container top {
> >            uses a:grouping1;
> >        }
> >   }
> >
> > The SID file contents for /top are not determined by module foo.
> > Over time, module A will keep changing and module foo will never change.
> > All of the unspecified revisions imported by module foo determine the
> expansion of 'grouping1.
> > This approach only works within a server implementation because all the
> module revisions
> > are known to the client via the YANG library.
> >
>
> [ivaylo]
> I think the difference of viewpoints here came from my assumption that
> all yang modules import specific versions of other modules. You point
> out in your email that in practice this is not necessarily the case. If
> we want to be able to handle this as I believe we are, I agree that we
> might need to add information in the SID file about what version of each
> of the dependent modules had been used to allocate any given SID as
> otherwise we might have serious problems figuring this out at runtime,
> if this is at all possible.
>
> Storing this information can be solved by having many independent
> versions of .sid files - one for a given set of YANG modules and
> revisions or we can have each SID number allocation contain more
> information than just the path to that node - namely the versions of all
> the YANG modules and the revision information.  In both cases it appears
> that this will complicate how SIDs allocation.
>
> Andy, do you agree that I have captured the essence of your concern
> correctly? Are there any thoughts whether we want to solve this problem
> and which is the best approach (even if it is not any of the ones
> already mentioned)?
>
> I might be missing some historical background here, but my personal
> preference is to go with the second approach as it would allow for
> linear versioning of the .sid files, which I believe is lighter on the
> constrained device. It seems that your preference, Andy, is to go with
> the first one and have SIDs be more contextually dependent.
>
>

I don't see how the 2nd approach can scale to even 20 YANG modules, let
alone 100s.
If there are multiple variants of every SID file, then there is no point in
an "official" SID file
at all. Only the mappings for a specific server implementation can be
used.  Even products
from the same vendor will use different revisions of the same imported
module.

It is quite possible that common SID files will be practical after
development is done
for a module and it is quite stable. There will probably be a mix of
deployment strategies
depending on module maturity and implementation details.





> >
> >
> >>
> >>
> >> >   - no way for vendors to apply deviations on any YANG modules, e,.g
> deviation { not-supported; }
> >>
> >> [ivaylo]:
> >> I am not sure I see why that is the case. In sec 3 of
> >> draft-ietf-core-yang-library we have
> >>
> >> * 'feature' and 'deviation' are implemented using a SID (i.e. an
> >> unsigned integer).
> >>
> >> While I agree that more details can be added here, I don't see any
> >> reason deviations
> >> not to be supported.
> >>
> >> >   - the SID file generation depends on the specific revisions of all
> modules imported
> >> >     and submodules included by the target YANG module
> >>
> >> [ivaylo]:
> >> I agree, but I believe this is a good thing.  As is said sec 5.1.1 of
> RFC6020
> >>
> >> ` modules need to be imported using specific revisions`.
> >
>



The YANG import-by-revision mechanism is fatally flawed and has been mostly
worthless
since Day One.  Perhaps this will be fixed someday but until then, most
designers do
not specify the exact revision that must be used.



> >
> >
> > This is not how YANG is used 95% of the time.
> > Even if I import-by-revision, it is very common for that module to
> > have its own imports, and I have no control over the revisions used
> there.
> >
> >
> >>
> >>
> >> Could you provide more details of the issue you see?
> >>
> >> >   - vendors are already failing to deploy YANG modules from a central
> source model.
> >> >     The NETCONF and RESTCONF protocols have robust mechanisms to
> discover and
> >> >     retrieve the actual YANG files used by the server, and vendors
> rely on this feature.
> >> >     Some vendors need model branching and other complexity that make
> SID assignment even harder
> >> >     (see
> https://tools.ietf.org/html/draft-ietf-netmod-yang-versioning-reqs-02)
> >> >
> >>
> >> [ivaylo]:
> >> I believe that the envisioned .sid file retrieval mechanism is:
> >> 1. You find the sid that you are interested in
> >> 2. You check in which mega range it is and who is the operator of that
> >> mega range in
> >> the IANA registry.
> >> 3. You contact that mega range operator through a given protocol to
> >> obtain further details.
> >>
> >
> >
> > No. I am interested in loading the SID data from each individual server,
> as needed.
> >
> > I understand some people want to hard-wire SIDs into their client or
> server
> > and they will not have the YANG identifiers.  I am not interested in this
> > deployment option for YANG-Push so the SID values will get added at
> runtime.
> > The client will consume or use cached the SID files at
> session-startup-time, just like the YANG modules.
>
> [ivaylo]:
> You are correct that I was looking much more from the perspective of
> hard-wiring the SIDs on constrained devices and assume that
> introspection will be done as needed using the central IANA registry or
> other server with such information. If you want to know what .sid file
> is present for each server, wouldn't the mechanism from
> draft-ietf-core-yang-library solve that? Or maybe you would like to have
> a more compact reply that gives only the YANG module name, version and
> version the SID file that can be found on a server - be it internal or
> external one. If that is the need you have, we can see which is the best
> approach to resolve it.
>



You can hardwire SIDs based on proprietary vendor information or IANA
information.
I don't see how IANA can maintain any SID files though, since the entire
set of imported modules
depends on the SID file generation date and modules available to the tools
used by by
authors submitting files to IANA



> >
> >
> >>
> >> Presently we believed as there are not going to be too many mega range
> >> registries and
> >> we have no experience operating them it might be unwise to specify the
> >> protocol that
> >> they need to implement in order for clients to be able to query them.
> >> Specifying such a
> >> protocol might be something we do if this is a really big concern for
> people.
> >> Would that retrieval mechanism cover the cases that you were thinking
> >> of? Are there other
> >> cases that you believe it should cover?
> >
> >
> >
> > I think vendors will continue to fill in the gaps with proprietary
> mechanisms.
> > The SID file format will get augmented, etc. to support real deployment
> scenarios.
> > This could be considered a protocol issue and can be ignored for SID
> files in general.
> >
> > How the client gets the YANG and SID files could be considered an
> implementation detail.
> > The IANA repo is there but there is no requirement to use it.
> >
>
> [ivaylo]:
> I assume that one of the participants is a constrained device, which
> might not be exactly what you are considering, but in that case if the
> constrained device is a client, it is very unlikely for it to try to
> discover at runtime what are the SIDs that a server will support. It
> will generally send CORECONF requests that contain some SIDs and assume
> that the server or a proxy in between will be able to understand their
> meaning. That is where the IANA repository and a protocol for
> inspection how a SID maps can be useful.
>
> Then assuming that the constrained device is the server, a client can
> discover the supported SIDs either through introspection protocol, which
> will be heavy, but might be required in some cases or more likely
> through provisioning/bootstrap.
>
> It seems to me that you are considering also the question if none of the
> endpoints is really constrained. In that case I believe the client will
> still be able to use the same introspection protocol as before and find
> the YANG modules looking at the SID file.
>
> >
> >>
> >>
> >> > CORECONF (and SID in general) needs a SID file retrieval mechanism.
> >> > Perhaps based on YANG Data Instance Files
> https://tools.ietf.org/html/draft-ietf-netmod-yang-instance-file-format-06
> >> > A server will make the URI of this resource known somehow, which can
> be used if needed,
> >> > before a client starts using SIDs.
> >> >
> >> > In addition, or instead of listing YANG modules, the data is SID
> files, maybe 1 super-SID-file JSON object.
> >> > It is not realistic that a client will be able to use all global SID
> files based on the YANG module name
> >> > and revision date, especially in the development phase. The IANA SID
> file may be useful after the YANG
> >> > module is very stable.
> >> >
> >> > More solution details:
> >> >   - The WG or vendor can maintain an informal public archive of SID
> files.
> >> >   - The YangModels repo update process should include SID files.
> https://github.com/YangModels/yang
> >> >   - Vendors still need the public SID file for the starting point, or
> maybe use it unchanged
> >> >   - The SID files need <CODE BEGINS> support and probably some tool
> changes in the ID-submission process
> >> >   - IANA will only archive the RFC version, just like YANG modules.
> >> >
> >> >
> >> >
> >> >> Grüße, Carsten
> >> >
> >> >
> >> >
> >> > Andy
> >> >
> >>
> >> --
> >> Best regards,
> >> Ivaylo
> >
> >
> >
> > Andy
> >
>
> --
> Best regards,
> Ivaylo
>


Andy