Re: [netmod] New Version Notification for draft-ma-netmod-immutable-flag-07.txt

["maqiufang (A)" <[e-mail]>]

Hi, Rob

Thanks for getting back to me,  please also see inline.
From: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Sent: Saturday, July 1, 2023 1:09 AM
To: maqiufang (A) <[e-mail]<mailto:[e-mail]>>
Cc: Kent Watsen <[e-mail]<mailto:[e-mail]>>; Jan Lindblad (jlindbla) <[e-mail]<mailto:[e-mail]>>; Jürgen Schönwälder <[e-mail]<mailto:[e-mail]>>; Andy Bierman <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: RE: [netmod] New Version Notification for draft-ma-netmod-immutable-flag-07.txt

Hi Qiufang,

It may be the WG understanding has moved on, but the latest version of the document hasn’t quite caught up, or otherwise I don’t find it clear, or seem to take very different interpretation of it.
[Qiufang] I do think that the WG understanding has moved on, at least I assume now there is WG consensus to not support non-transactional APIs, yet I also understand it cannot just be the tacit understanding, but also need to be stated very clearly in the document. Sorry if it has caused any misunderstanding or confusion.
Please see inline … (this is all with reference to -07).


From: maqiufang (A) <[e-mail]<mailto:[e-mail]>>
Sent: 09 June 2023 16:39
To: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Cc: Kent Watsen <[e-mail]<mailto:[e-mail]>>; Jan Lindblad (jlindbla) <[e-mail]<mailto:[e-mail]>>; Jürgen Schönwälder <[e-mail]<mailto:[e-mail]>>; Andy Bierman <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: RE: [netmod] New Version Notification for draft-ma-netmod-immutable-flag-07.txt

Hi, Rob

Thanks for sharing your concerns, but I think there might be some misunderstanding that needs to be clarified, please see my reply inline.

From: Rob Wilton (rwilton) [mailto:rwilton=[e-mail]]
Sent: Friday, June 2, 2023 7:01 PM
To: Kent Watsen <[e-mail]<mailto:[e-mail]>>; Jan Lindblad (jlindbla) <[e-mail]<mailto:[e-mail]>>; Jürgen Schönwälder <[e-mail]<mailto:[e-mail]>>; Andy Bierman <[e-mail]<mailto:[e-mail]>>
Cc: maqiufang (A) <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: RE: [netmod] New Version Notification for draft-ma-netmod-immutable-flag-07.txt

Hi Kent, all,

Writing as a contributor, I still have strong concerns with this draft.

From a YANG architecture perspective, I believe that the contents of the running datastore should be entirely under the client’s control, and servers should accept any valid configuration, and be able to move from any valid configuration to any other valid configuration.  We also already have the server datastore draft that I think should be the mechanism to allow a server to include server-controlled configuration before it is merged with running and validated as intended, that is somewhat outside the client’s control.  I.e., I think that having a clean split of ownership and responsibilities between a running datastore (managed by the client) and other datastores (e.g., intended and system-controlled) managed by the server is a good clean architecture.
I agree with you that the client should have fully control over the contents of the running datasore, but I don't see this draft(-07) conflicting with that goal. Maybe we should make it more explicit in the document, but it is already the case the immutable configuration can only be created by the system (and present in <system>)
[Rob Wilton (rwilton)]
I think that the document is unclear about how it interplays with the system datastore, e.g., I find very few references to the system datastore, so I think that it would be helpful for that to be clarified.
[Qiufang] Sure. That’s true, most of the references to system datastore document only appear in the use case appendix, I agree that this should be more explicit, to point out that only the server can create immutable configuration, and thus immutable data appears in <system>(if exists). A client may create a same value of immutable node as in <system> (e.g., to fulfil leafref constraints), but is not allowed to modify or override (https://www.ietf.org/archive/id/draft-ietf-netmod-system-config-01.html#section-5.4) immutable node with different values. Will clarify in the next version.
But I think we don’t necessarily want immutable flag to be coupled with system datastore, that is to say, even a server doesn’t implement a <system> datastore, it could still be possible to have non-modifiable system configuration somewhere (e.g., in <operational>), and thus an immutable flag could be helpful in this case. Make sense?



E.g., section 1.2, starts with: “ The "immutable" concept defined in this document only documents existing write access restrictions to writable datastores.”, which I read as saying that the immutable flag is purely about <startup>/<candidate>/<running> datastores because <system> and <intended> are not writable datastores (as per RFC 8342, section 5), but from your comments below, I don’t think that is now the intent.
[Qiufang] Could you expand on this? It is obvious that immutability is only about restrictions on read-write datastores, because the client cannot edit read-only datastores (e.g., <system>, <intended>, <operational>) as you said. So immutable flag only refers to write access restrictions to writable datastores. Also, the immutable-flag is supported even when a client retrieves a read-only datastore, but it’s still about the indication of restrictions on read-write datastores.

Further, even just having the server creating immutable configuration in <system>, and allowing the client to theoretically have complete control over what is in <running>, can still cause the server to reject config changes to running if <running> + <system> is merged to <intended> and the “immutable” configuration in <system> means that <intended> fails to validate, hence causing the change to <running> to be rejected.
[Qiufang] if the config change to <running> is about trying to override a value of a particular system node which the server considers immutable internally, it will always be rejected by the server. But I think this avoids the non-transactional behaviour, since the server already rejects the configuration from the very beginning.

If the immutable configuration is always immutable, then that is probably okay, since that configuration will always have been rejected.
[Qiufang] Yes, it is always immutable. Jan has raised a similar comment, I am thinking the immutability must only change by software update or license change.

But it isn’t clear to me what actual configuration is going to be marked at immutable.  E.g., could this include certificates that you don’t want the client to be able to change?  If so, what happens if the client copies the certificate into <running> to fulfil a leafref dependencies, and then the system certificate gets updated by an out of bound process.  The copy of the certificate in system would have changed and now it would be different and conflict with the version of the certificate in running.
[Qiufang] This seems much like an issue scoped in system config spec and has been discussed a little before. I don't really have any expertise in security area, but to my understanding, if the server doesn’t want any client (even a role of super-admin) to be able to modify the certificates, it is immutable configuration. If a client copies the certificate into <running> to fulfil a leafref dependency, it should be sufficient for only the “key” name be copied into <running> to make the configuration valid. And then the system certificate gets updated , I assume the certificate name remains unchanged? That way, the configuration in <running> is still valid and no conflict with the contents in <system>.

I think that it would be useful to have an example list of configurations that we expect to be handled this way.  I.e., to ensure that add
[Qiufang] you mean the example list of immutable configurations?
Basically, I am thinking the following list, which is almost consistent with the appendix UC section(https://www.ietf.org/archive/id/draft-ma-netmod-immutable-flag-07.html#appendix-A, UC5 is considered to be removed in the next version):

-          System config related to hardware physical characteristics or capabilities

-          System predefined users and permissions

-          System predefined policies the server doesn’t want to be changed

-          Key or certificate configuration predefined by the server

, other cases like the client creates a node instance but cannot modify that afterwards is the non-transactional behaviour we’ve discussed before and should be avoided.
[Rob Wilton (rwilton)]
I agree that this should be avoided, but section 2 of the document starts with:

2.  Solution Overview

   Already some servers handle immutable configuration data and will
   reject any attempt to the update of such data.  This document allows
   the existing immutable data node or instance to be formally
   documented by YANG extension or metadata annotation rather than be
   written as plain text in the description statement.

I interpret this as: Some servers already do this (i.e., effectively breaking transactional behaviour), and that we accept that this happens and provide a way to document (and implicitly allow) this behaviour.  So, I think that this text also needs to be updated.
[Qiufang] Even the immutable configuration is narrowed to be only defined by system, the objective is still to formally document existing server behaviors, not to encourage server to  introduce more such restrictions. This document doesn’t apply to servers which don’t have any immutable system configuration. Can we just say that the existing server behaviors must be non-transactional? Maybe some of them are, but that's already something this draft should not support, we will see how to make it clear in the next version.

That said, only the server can create/update/delete immutable configuration, this has no impact to clients and <running> datastore by default.
[Rob Wilton (rwilton)]
Okay.


Only when the client would like to reference the immutable system configuration, will that configuration be copied and thus appears in <running>.
[Rob Wilton (rwilton)]
Okay – I presume this is just part of the proposed system datastore behaviour defined in the system datastore draft.


But by default, immutable configuration is only seen in <system> (if implements) and <operational>. Hopefully this clarifies.
[Rob Wilton (rwilton)]
Okay, but I don’t think that this is particularly clear in -07.
[Qiufang] Sure, will work on a new version to clarify the whole intent, and hopefully could be posted before the submission window closes.



I appreciate that not all servers allow clients to fully control their running configuration, but I think that a better solution (for management clients) so to encourage servers to migrate towards the goal of giving full ownership over running to the clients.  Hence, I’m particularly concerned about standardizing a YANG meta-data annotation that allows, and arguably even encourages, vendors or other SDOs to build immutable properties into their management models that breaks this goal.  I think that we need to be really careful here that we are not creating yet another fork of NETCONF/YANG with a fairly fundamentally different architecture to what we are currently aiming for.

As I mentioned above, I think it’s not our intention to break the goal of giving full ownership over running to the clients. It is still about the system-controlled configuration immutability declaration. This does not necessarily mean that we are encouraging such behaviour, the introduction section has also states that:

” Immutability is an existing model handling practice.  While in some
  cases it is needed, it also has disadvantages, therefore it MUST be
  avoided wherever possible.”
Would this be sufficient from your perspective?
[Rob Wilton (rwilton)]
So, really, what the immutable flag means is that for a data node in system then the server will reject any attempts by the client to configure a different value.
[Qiufang] Yes, a client is not allowed to create/update/delete it on a device. It can only be created/updated/deleted by the system. Note a client may configure an immutable node in <running> with the same value as found in <operational> or <system> or delete it from <running>, which merely mean making immutable node visible/invisible in <running>.

If this is the case, then I’m not sure that I understand the value of the “extension immutable”, can you give examples of where this would be useful please?
[Qiufang] The “extension immutable” is considered to be used if a particular data node is always considered to be immutable independent of any its instance(s).
A YANG extension makes immutability even visible for the client at implementation time (not just runtime), Therefore, the client has the opportunity to avoid error response in its NMS design/development time due to attempts to override immutable configuration.

We do have an open issue regarding should the "immutable" metadata annotation also be returned for nodes described as “extension immutable” so that there is a single source of truth.

Appendix A.1 illustrates an example of immutable configuration where a YANG extension could be useful (https://www.ietf.org/archive/id/draft-ma-netmod-immutable-flag-07.html#appendix-A.1).

I am somewhat more amenable to an annotation that indicates that if a particular leaf is modified it will potentially cause a more impactful change, by effectively causing a delete and re-add of the parent list/container (changing interface type could be an example of this).  But I don’t think that this stop clients from modifying the leaf to a new valid state, instead, the server should perform any necessary orchestration steps to apply the configuration rather than pushing that as extra orchestrations steps onto the client.  There is also part of me that questions how useful such an annotation or metadata would really be given that there are many other data nodes that have wide impact if they are modified.  So, from this perspective, I think that “immutable” is perhaps the wrong name.
After the previous discussion on the list, we were trying to avoid the non-transactional APIs, and now we are only targeting immutable configuration with the lifecycle totally driven by the system.
[Rob Wilton (rwilton)]
Okay, so from your comments, am I right to understand that the intention of the “immutable” flag is to mark a subset of the configuration in <system> (if it exists) as configuration that can never be overwritten by the client?
[Qiufang] Yes, but the immutable flag does not require that the server must have configurations that cannot be overwritten by the client.
Instead, it formally documents the existing behavior of the server to improve visibility when it internally think some system configuration immutable.

So I tend to agree with your proposal to define a flag to indicate a leaf is modified will potentially cause the server to delete it and recreate with its parent node. And I also agree this should not be called “immutable” anymore. But that is not we are trying to achieve in the document now, and is not in the scope of the current document.
[Rob Wilton (rwilton)]
Okay.


Finally, I still question the assertion “Clients believe that "config true" nodes are modifiable even though the server is allowed to reject such a modification at any time.“ and regard it as possibly a bit disingenuous or perhaps being overplayed.  I’m not sure whether this assertion is coming from the YANG language (i.e., does RFC 7950 state this – I couldn’t quickly find it), or from NETCONF?  To me, it makes sense that a NETCONF server can reject a configuration change for various reasons (e.g., invalid yang, out of memory, some bug or flaw in the implementation), but I don’t think that really means that it is okay for a server (from a client’s perspective) to arbitrarily reject configuration.  A slightly strawman, but, e.g., would it be valid for a server to reject a request based on whether a generated random number was odd or even?  Can a server reject a config request because although the YANG schema indicates that it should be a number, the server has decided that sometimes it will only accept the item as string?  Perhaps according to the NETCONF spec these are both valid, but I’m not sure that either of these behaviours are helpful to clients or within the spirit of what is expected.

I think it might be the “at any time” that is overplayed. In NETCONF RFC(https://datatracker.ietf.org/doc/html/rfc6241#section-7) it already mentions that

“A protocol operation can fail for various reasons, including
   "operation not supported".  An initiator SHOULD NOT assume that any
   operation will always succeed.”
Immutable configuration is something the server always rejects, I think it a stable behaviour.
[Rob Wilton (rwilton)]
Okay.



I do think that this is useful and interesting topic to have further discussion, particularly because of the external SDO interest - possibly a dedicated interim may be helpful – if we can get the key parties together?  As to adoption, I’m not necessarily opposed to this because there is definitely interest in this work, but personally I would like to see quite significant changes, and I suspect that more work is required to reach consensus.
Thank you, Rob. I don't object to a dedicated interim meeting personally, but before that I think it might be helpful for us to make sure that we have the same understanding about our scope of immutable configuration, agreed?
[Rob Wilton (rwilton)]
Absolutely agree that we should agree the scope.

Also, with a narrowed scope, I want to ensure that this annotation of behaviour description has enough value to justify the additional complexity.
[Qiufang] I agree that we should evaluate how much value it will bring to us and how difficult and complexity it will be to implement. Just some of my thoughts:

1)      For the server that doesn't have any immutable system configuration, this document doesn't apply to it at all. No complexity introduced.

2)      For the server that already have immutable system configuration (regardless of implementation of the <system>), it always rejects a client’s operation request to modify it with a different value. This is already the case.

All the server need to do now is to return the data it considered immutable with annotated immutable flag when asked by the client.

The main benefit of this draft is documentation, so that a client has a way to understand how it could potentially affect its program.

Possibly, if the WG decides to adopt this work, it might be worth considering whether this would be better documented as part of the system datastore draft?
[Qiufang] Since we have decided to limit the scope of immutable flag to system configuration and not support non-transactional APIs, it makes sense to me for it to be documented in the system datastore draft.
But as I mentioned above, we would like the immutable flag to work even when the system datastore is not implemented.
Even that said, I'm open to this, and okay to document it as part of system datastore draft if the WG thinks it's a good idea.

Regards,
Rob

Best Regards,
Qiufang

Regards,
Rob

Best Regards,
Qiufang

From: Kent Watsen <[e-mail]<mailto:[e-mail]>>
Sent: 01 June 2023 21:55
To: Jan Lindblad (jlindbla) <[e-mail]<mailto:[e-mail]>>; Jürgen Schönwälder <[e-mail]<mailto:[e-mail]>>; Andy Bierman <[e-mail]<mailto:[e-mail]>>; Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Cc: maqiufang (A) <maqiufang1=[e-mail]<mailto:maqiufang1=[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: Re: [netmod] New Version Notification for draft-ma-netmod-immutable-flag-07.txt

Hi Quifang,

The latest update looks very good to me - IMO, ready for adoption.

Jan, Jurgen, Andy, Rob - can you confirm that your concerns have been addressed?

Thanks,
Kent



On May 25, 2023, at 8:16 AM, maqiufang (A) <maqiufang1=[e-mail]<mailto:maqiufang1=[e-mail]>> wrote:

Hi, all
This version reflects the input we've received from the mailing list.

Thank you everyone(Jan, Rob, Kent, Jürgen, Andy, Frank et al.) for your great comments and suggestions!
Please see if the following updates are good for you:
   *  Use a Boolean type for the immutable value in YANG extension and
      metadata annotation
   *  Define a "with-immutable" parameter and state that immutable
      metadata annotation is not included in a response unless a client
      explicitly requests them with a "with-immutable" parameter
   *  reword the abstract and related introduction section to highlight
      immutable flag is descriptive
   *  Add a new section to define immutability of interior nodes, and
      merge with "Inheritance of Immutable configuration" section
   *  Add a new section to define what the immutable flag means for each
      YANG data node
   *  Define the "immutable flag" term.
   *  Add an item in the open issues tracking: Should the "immutable"
      metadata annotation also be returned for nodes described as
      immutable in the YANG schema so that there is a single source of
      truth.

Thanks a lot.

Best Regards,
Qiufang

-----Original Message-----
From: [e-mail]<mailto:[e-mail]> [mailto:[e-mail]]
Sent: Thursday, May 25, 2023 4:52 PM
To: Balazs Lengyel <[e-mail]<mailto:[e-mail]>>; Hongwei Li <[e-mail]<mailto:[e-mail]>>; Qin Wu <[e-mail]<mailto:[e-mail]>>; Qin Wu <[e-mail]<mailto:[e-mail]>>; maqiufang (A) <[e-mail]<mailto:[e-mail]>>
Subject: New Version Notification for draft-ma-netmod-immutable-flag-07.txt


A new version of I-D, draft-ma-netmod-immutable-flag-07.txt
has been successfully submitted by Qiufang Ma and posted to the IETF repository.

Name:                  draft-ma-netmod-immutable-flag
Revision:              07
Title:                      YANG Extension and Metadata Annotation for Immutable Flag
Document date:               2023-05-25
Group:                  Individual Submission
Pages:                   24
URL:            https://www.ietf.org/archive/id/draft-ma-netmod-immutable-flag-07.txt
Status:         https://datatracker.ietf.org/doc/draft-ma-netmod-immutable-flag/
Htmlized:       https://datatracker.ietf.org/doc/html/draft-ma-netmod-immutable-flag
Diff:           https://author-tools.ietf.org/iddiff?url2=draft-ma-netmod-immutable-flag-07

Abstract:
   This document defines a way to formally document existing behavior,
   implemented by servers in production, on the immutability of some
   configuration nodes, using a YANG "extension" and a YANG metadata
   annotation, both called "immutable", which are collectively used to
   flag which data nodes are immutable.

   Clients may use "immutable" statements in the YANG, and annotations
   provided by the server, to know beforehand when certain otherwise
   valid configuration requests will cause the server to return an
   error.

   The immutable flag is descriptive, documenting existing behavior, not
   proscriptive, dictating server behavior.




The IETF Secretariat



_______________________________________________
netmod mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/netmod