Re: [netmod] yang versioning solution complexity and alternative approaches

[Jürgen Schönwälder <[e-mail]>]

Rob,

discussing details is likely distracting from the main difference in
our viewpoints so I will only give a top posting response.

NBC changes happen to "data definitions" and I believe this is where
they deserve to be documented. Once you have NBC changes properly
marked and documented at this granularity, you can generate any
desirable information easily (perhaps even module version numbers).
But most importantly, you can algorithmically decide whether a module
B importing from module A is affected by any NBC changes in A. Module
version numbers are too course grained to do that, hence they force
consumers of YANG modules (the readers) to reverse engineer the NBC
changes to decide if and how they might be affected.

A NETMOD design principle has been that readers come first, writers
second, and tool makers last. The versioning solution presented by the
design team seems to put writers first, leaving readers with the
exercise to reverse engineer where NBC changes took place. I consider
this backwards.

/js

PS: I believe it is generally not decidable whether two YANG module
    revisions are BC or NBC. This does not mean that smart diffing
    tools are not useful but we should not leave the impression that
    they are a solution for readers; they are a useful tool for writers.

On Tue, Jun 07, 2022 at 12:54:14PM +0000, Rob Wilton (rwilton) wrote:
> Hi Jürgen,
> 
> Thank you for your feedback and apologies for the very delayed reply.  The versions authors have been discussing some of the points that you raised in a lot of detail and hence I was waiting for this discussion to converge before replying to you.
> 
> 
> > -----Original Message-----
> > From: netmod <[e-mail]> On Behalf Of Jürgen
> > Schönwälder
> > Sent: 09 March 2022 10:16
> > To: [e-mail]
> > Subject: [netmod] yang versioning solution complexity and alternative
> > approaches
> > 
> > Hi,
> > 
> > the YANG versioning solution appears to be complex.
> 
> I think some aspects of the solution are complex, but I'm not sure whether they are really avoidable, or whether the complexity comes from wanting to solve a complex problem.
> 
> One aspect of additional complexity is because of a WG desire stated during adoption and previous discussions to support different version schemes (E.g., YANG Semver, Open Config's use of vanilla Semver, vendor versioning schemes).
> 
> I think that keeping this flexible is a reasonable choice. I also think that having some version indicator at the module level (or in the case of YANG packages, YANG schema level) is beneficial.  For example, if my client is designed to work with module x@2.0.0, then if the server has actually implemented x@2.5.2 then it is compatible, assuming that module x has been updated according to the rules.   x@3.0.0 might also be compatible depending on what has changed - but as a client I had better check to see what the changes are to determine what the incompatibility is (using tooling).  I.e., I still think that version numbers provide more useful guidance (and partial relationship) than say revision dates, particularly given that YANG modules are not always strictly evolved in chronological order.
> 
> It was also clear to me when considering versioning schemes there is no perfect one-size fits all solution.  The YANG Semver approach is meant to be a pragmatic compromise.  I.e., encourage linear development, encourage backwards-compatible changes, but allow for non-backwards-compatible changes to occur and allow bugfixes to older revisions if required.
> 
> Finally, the wider versioning solution is also aiming to solve multiple different problems (e.g., package/schema level versioning, differentiation between backwards-compatible and non-backwards-compatible changes, supporting some level of branching for bugfixes, improving import dependencies, etc) which I think naturally somewhat increases its complexity.  We have somewhat tried to put these into separate documents so that each document can focus on specific aspects.
> 
> 
> > 
> > - We introduce version numbers that smell a bit like SemVers but then
> >   are not really SemVer.
> 
> Yes, and no.  I'm not sure that there is really a canonical industry agreed definition of what Semver is, and hence we wanted quite a tight specification of what it means in the context of YANG, whilst still aiming to be mostly compatible with the Semver 2.0.0 definition.
> 
> We created YANG Semver for the sole reason that regular Semver or revision-dates are not sufficient to support one of the key requirements.  Specifically, the need to allow bug fixes to older shipped modules without the luxury of forcing both client and server to always migrate to the latest version of the module (e.g., as an Open Source software project might be able to do).
> 
> It is also worth noting that module authors have the choice of never branching older versions, and only ever making updates to the latest version, in which case YANG Semver is identical to Semver 2.0.0.
> 
> 
> > 
> > - We have no solution to do meaningful things with these version
> >   numbers, it does not even seem to be possible to decide whether
> >   X.Y.Z derives from x.y.z or not.
> 
> The derivation is based solely on the revision history in the YANG module file.  If the previous revision exists in the module's revision history, then the revision is derived from it.
> 
> The intention is that the version numbers represent a partial ordering.  E.g., moving from version X.a.b to X.c.d, where c > a, or c == a and d > b is a backwards compatible change.  This relationship breaks down when the _compatible and _non_compatible modifiers are used, but I regard these as a practical compromise to allow a limited form of branching.
> 
> Even with regular Semver it is not possible to know if version X.Y.Z derives from x.y.z.  Even with semver 2.0.0, it seems (the spec is somewhat ambiguous) that you are allowed to chronologically create 1.2.1 then 2.0.0, then 1.3.1.  And in this case 2.0.0 will not necessarily contain the content that was added in 1.3.1.
> 
> 
>  We still seem to believe that
> >   having compatibility constraints embedded in module imports are a
> >   workable solution even though we acknowledge future breaking
> >   changes.
> 
> This is a point of confusion, which probably means that we need to improve the text, since this isn't our goal for the "import by revision-or-derived".   Note, we did consider a more complete, and hence complex, solution, but this seemed to be the right pragmatic compromise of improving what we have today vs embedding too much information.
> 
> The only goal is to specify a minimum import dependency that the author knows is required.  E.g., if you are using a new type defined in RFC 6991bis then it is useful to indicate the minimum module revision that is likely to be compatible.  Later versions are likely to be okay, but that obviously depends on whether any non-backwards compatible changes happen and what the nature of those changes are.
> 
> YANG packages allow compatibility to be indicated for exact module versions outside the module.  We did consider whether YANG packages could be extended to cover ranges of versions, but that was deemed to increase complexity and best left to future work.  We had an issue tracking this, but after discussion closed the issue.
> 
> 
> > 
> > - We push for a reasonably complex algorithm to calculate deltas of
> >   YANG module revisions to let users of modules to determine the real
> >   impact of module updates on their work.
> 
> Please can you clarify.  Are you saying that the algorithm to determine whether a change is a non-backwards-compatible change is complex?  Or the encoding of it into a semantic version number is complex?
> 
> For the former, most of these rules were also defined in RFC 7950 but extended to fix some corner cases.  For the latter, this is basically the standard Semver which is widely understood but with a minor extension.
> 
> In both cases, I think that tooling can flag most of these warnings and calculate (or check) what the new semantic version number should be.  I'm not sure that this algorithm really ends up being that complex in practice.
> 
> 
> > 
> > I wonder why we not consider the opposite approach, namely to have
> > author making NBC changes to document them right where the changes
> > are. If authors would write something like
> > 
> >       leaf foo {
> > 	nbc-change "2022-03-01" {
> >           description "changed type from int32 to string";
> > 	};
> > 	// ...
> >       }
> 
> In the discussions with the other authors the consensus was that these sorts of annotations are useful in some places.  I.e., the original intention is that this would be covered by the 5th draft of the versioning set, i.e., https://www.ietf.org/archive/id/draft-ietf-netmod-yang-schema-comparison-01.txt, but on further discussion we now believe that it may be more appropriate to define this annotations are part of the module versioning draft instead (which we would be keen to hear wider WG thoughts on).
> 
> The consensus of the discussions from other authors was:
> 
> - Providing a module level indicator of the type of changes is required, even if we have per data-node annotations.
> - Consumers will ultimately need to rely on tooling to do module and package schema comparisons to determine what has changed and the potential impact of those changes to catch the case where the changes have not been correctly labelled by the authors.
> - Using annotations doesn't really work in the case that a definition has been removed from a module.  Even after a deprecation/obsoletion phase this could still break clients.
> - We were concerned that these annotations would end up cluttering the module definition with a lot of extra history, the desire is that the modules can evolve and each module version provides a snapshot of the API at that point in time.
> 
> We believe that the greatest value of the annotations is to use them when a change cannot be robustly detected by tooling alone, e.g., for changes to pattern, must, when, regex statements or descriptions.  There was also a question as to what the tooling should default to, if it cannot calculate what type of change is being made. 
> 
> 
> Hence, we discussed three proposals on how these tags could be used:
> 
> Proposal A:
> We provide both nbc-change and bc-change marker extensions but they are only expected to be used when a tool cannot reasonably automatically determine if the change is bc or nbc.
> Or, in more detail, I would expect the following steps:
> (i) A tool is used to compare a new revision of a module to the previous revision in its revision-history to calculate what type of changes the module contains and validate (or calculate) the module's Semver version label.
> (ii) bc-change and nbc-change markers would be used where the tool cannot automatically determine what the changes are for a given statement (e.g., pattern, must, when, regex, description statements).
> (iii) The module versioning (or tooling) draft would define change defaults for the case that the tool cannot be expected to robustly check.  I.e., the proposal would be to assume that pattern, must, when, regex changes are all nbc-changes unless annotated with a bc-change marker.  Description (and other similar) statements would be assumed to be editorial changes unless marked with an nbc-change or bc-change marker.
> (iv) bc-change and nbc-change markers would also be allowed to be added at the authors discretion (i.e., it is okay to explicitly mark changes that the tool would be expected to identify anyway).
>  
> Proposal B:
> The second proposal is the same as Proposal A, except that the nbc-change marker is mandatory to include everywhere where there is an nbc-change, i.e., even if tooling would be expected to detect it as an NBC change.
> - This approach probably prevents an author from ever deleting a node (because you can't annotate what isn't there), instead a "tombstone" obsolete entry would need to be retained.
> - I'm also concerned that this approach assumes that every change is implicitly compatible unless marked otherwise, and hence I think that we would still need a bc-change marker extension to be used to suppress possibly false nbc warnings by a module/schema comparison tool.  E.g., a tool might not be able to see that a change to a pattern statement is backwards-compatible and hence would probably need a bc-change annotation to avoid it incorrectly flagging it as an non-backwards-compatible change.
> 
> Proposal C:
> The third proposal is the observation that we don't have to take the same approach for all sources of YANG models.  E.g., we could allow YANG modules generally to follow proposal A, but then mandate that IETF modules follow the stricter approach of Proposal B.  I.e., all nbc-changes must be explicitly annotated, and data nodes cannot be removed from published modules.
> 
> 
> Between the authors I think that the discussion was leaning was towards proposal A or C.  But we would be interested in hearing other views from the WG.
> 
> 
> > 
> > then tool and humans can easily figure out in which revision NBC
> > changes occured and if they affect a certain usage of the module.
> 
> Without a module (or schema) level indication then I think that this is harder for humans, giving a module summary indication is I believe also helpful.  I was also assuming that module revision description should highlight any NBC changes - maybe we should add text for this.
> 
> E.g., if over time most updates to modules are backwards compatible then being able to see that the device package version number changed in a backwards-compatible way is much simpler (but both humans and tools) than having to check all changes in all modules trying to spot non-backwards-compatible differences. 
> 
> 
> > 
> > Instead of simply properly documenting the changes, we invent fuzzy
> > version numbers and complex algorithms to reverse engineer the facts
> > that could have easily been documented by whoever makes the change.
> > 
> > If the reason is that developers do not document their changes, then
> > go and develop tools to force them to document their changes. I do not
> > think it is fair to simply push the pain to the users of YANG modules.
> 
> Note, the aim here is not to push pain on the users of YANG modules, but actually to act as a warning to the user.  I.e., if it is a minor version change then they should be safe to use the updated module version, if not then they need to check the changes more closely, and I agree that annotations inline within the module may be helpful for this.
> 
> But in all cases, whether we are talking about a version number at the module level, or annotations on particular data nodes, it is still plausible that publishers will sometimes get this wrong and we need good tooling (alongside bc/nbc annotations) to check when a module is published that it is following the rules, and which can also be used by clients to check that any breaking changes have been properly described.
> 
> Regards,
> Rob
> 
> // All as a contributor and with no hats.
> 
> 
> >   
> > /js
> > 
> > --
> > Jürgen Schönwälder              Jacobs University Bremen gGmbH
> > Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
> > Fax:   [phone]         <https://www.jacobs-university.de/>
> > 
> > _______________________________________________
> > netmod mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/netmod

-- 
Jürgen Schönwälder              Jacobs University Bremen gGmbH
Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
Fax:   [phone]         <https://www.jacobs-university.de/>