Re: [netconf] New Version Notification for draft-ietf-netconf-transaction-id-01.txt

["Jan Lindblad (jlindbla)" <[e-mail]>]

Hi James, all,

Thank you James for your very thorough review of draft-ietf-netconf-transaction-id-00 several months ago. It was a great contribution!
Only now did I realize that I never managed to send my thanks nor comments to you in reasonable time after your review. Sorry about that; your contribution was nonetheless truly valuable to me.

As you might have seen, I just posted an updated -01 version of the document to the list.
<https://www.ietf.org/archive/id/draft-ietf-netconf-transaction-id-01.html>
Transaction ID Mechanism for NETCONF<https://www.ietf.org/archive/id/draft-ietf-netconf-transaction-id-01.html>
ietf.org<https://www.ietf.org/archive/id/draft-ietf-netconf-transaction-id-01.html>
[favicon.ico]<https://www.ietf.org/archive/id/draft-ietf-netconf-transaction-id-01.html>
I think it addresses all your suggestions and comments, as far as I am able to. Additional comments below:

Sorry for the delay but I know I promised you a proper review of the transaction ID draft.

[jan] Thank you very much for your feedback, most useful to me.

Thanks for the hard work on this one, I think it’s very useful.  The draft is well written and generally clear.

As a general comment I would like to see some more inline XML NETCONF operation requests and responses to illustrate some of the points inline.  I know there are a number of examples at the end and there are some diagram examples but sometimes seeing (short) actual request and response messages is useful to help understand the specific point being discussed.

[jan] The document is divided up into one part that discusses a general mechanism, and another part that discusses two specific protocols and their encoding, that implement the general mechanism. The general mechanism also applies to RESTCONF, so then there are four different txid mechanisms that are out there. Because of this document structure, I didn't want to get any XML encoding details into the first general part. That's why I invented the call-flow notation used in the first part of the document.

It might also be useful to review RFC7951 for the proposed JSON encoding for any tools doing conversion as well (this might be as simple as saying these are treated the same way as metadata annotations).

[jan] Thanks for pointing this out -- it made me realize that the txid mechanism for YANG-Push was dependent on XML attributes. I have changed this now so that the txid mechanisms for YANG-Push uses plain augmented YANG leafs, which makes it completely obvious how to encode for JSON.

With this adjustment, I believe there is now never any need to encode a txid in JSON, except when modeled as a plain leaf. For RESTCONF, RFC8040 already spells out how to use Etag and last-modified HTTP headers. This means txid information is completely free from JSON @ attribute notation.

I don’t particularly like the “!” etag option.  Have you considered the allocation of etags immediately, even in a candidate?  This would allow for txid operations (conditional) to happen on a candidate configuration as well.  These would be allocated on an edit-config/edit-data operation (and returned to the user) and could be overwritten with new ones on commit (and returned to the user) (comments below detail where this is referenced in the document currently).

[jan] Not super proud over this solution, but it's the best I could come up with. There are use cases where the server really can't know what the future txid will be. One immediate example is when a client uses the last-modified txid mechanism. The server has no way of knowing in advance when the candidate will be committed (if ever). To require that server implementors track txids for individual edits in the candidate is certainly possible, but my current feeling is that this would be more of a burden to implementors than a useful client feature.

I believe the NETCONF spec might need to be updated if the OK message is to be extended to allow additional information to be returned on a success message.  This approach might also lend itself quite well to RESTCONF if txid is going to be applied there as well (although I know that already has an etag and I’m not clear on the interactions between these two solutions – but perhaps that’s alright).

[jan] Hmm, yes. That has been up for discussion many times over the years. Right now the -01 version states that the txid shall be added as an attribute to the <ok> element in the RPC. The NETCONF XSD grammar does not allow that. Since it is only added when clients specifically request it, I feel the solution would still be highly interoperable. An alternative solution would be to state that the txid value should be added to the rpc-reply envelope instead. The NETCONF XSD grammar allows additional attributes there.

I would be interested in everyone's opinion on this choice.

Some specific comments/observations below:


Introduction:

The introduction doesn’t explicitly state that this draft will not discuss state (config false) data and it probably should.  I know this is a question that came up when I was discussing it with colleagues.  You can derive this from the spirit and content of the draft (particularly the last paragraph in section 3.2) but an explicit statement up-front might be beneficial.

[jan] Thanks. I have beefed up the introduction noting the above.

Section 3 - NETCONF Txid Extension:

The compare operation (RFC9144) isn’t mentioned here.  Do you have a view on how/if TxID should be used with this?

[jan] Good idea :-) I added a couple of sections about how a server might combine the NMDA Compare operation with txid information. Have a look at what I propose, and see if this matches your expectation. See sec 3.11, 5.8 and 6.3 in the -01 version.

Section 3.2 – General TxID principles

‘Each server MUST maintain a txid metadata value for each configuration datastore’ but it is unclear where/how this is envisaged being stored.  The top-level element might be one place but really this is just another versioned node.  The entire datastore may a txid value added to the NMDA datastore itself but then this would need to be returned outside of any <config> item returned from a NETCONF operation.  Perhaps a section of entire datastore txid might be useful describing where it is stored and how it is interacted with.

[jan] I intentionally left storage implementation details to implementors. I don't think there is anything useful to be said in the draft about that. If you look closely in section 4.3.2 and 5.1.1, you'll see that the top level datastore txid is mapped to the <data> and <config> tags in the get-config and edit-config RPCs (and similarly for all the other RPCs).

Section 3.3 – Initial configuration retrieval

You mention this ‘The exact encoding varies by mechanism, but all txid mechanisms would have a special "txid-request" txid value (e.g. "?") which is guaranteed to never be used as a normal txid value.’ In the second paragraph.  I believe you are intending to detail that sending the get-config/get-data request without a txid would return the configuration with no txid values, sending it with a txid of ‘?’ would return all of them and sending a specific txid would return those nodes with that txid?

[jan] Sending in a "?" would return all txid values below the point where the attribute was found. If a client sends a specific txid, the contents of that node and descendants is only returned by the server if the txids *differ*. No point sending the data if both sides are already in agreement about its content. Basically the "?" value is just a txid value that is guaranteed to always differ from any real txid value.

 Is this is the intention perhaps this section could include some updated wording and some examples of each.  If this is not the intention, perhaps just omitting the txid value in the request would return all txid values on a server that supports txid.  Receiving this data won’t hurt if you choose not to use it (other than the additional characters being streamed).

[jan] Thanks. I have beefed up the text a little, to clarify what you discuss above. When it comes to additional characters being streamed, I have thought long and hard about how to minimize that overhead. If done carelessly, the added burden of the txids could easily be substantial. Also, I don't want to return txid values unless requested by the client in order to not surprise any legacy implementations.

In the note at the end you mention the proposed options for the format of a txid.  This might be useful in its own section, particularly the details about the values not being contiguous increasing numbers.  We should also cover attempts for txid collision avoidance.  You do have something more about this in section 4.1 but perhaps splitting this out and referencing it from both places might be good?

[jan] Ok, interesting. What do you think we should say about the txid format or collision avoidance? Could you suggest some text or sketch?

Section 3.4 – Subsequent configuration retrieval

This section is probably the least clear in the document for me.  Could you provide some examples (XML inline for each bullet point) to explain the behaviour and the expected results.

[jan] Thanks for the feedback. I tried to rephrase slightly, but I am also not convinced that added verbiage really would improve the situation. If you have specific text suggestions, I'm all ears.

The second bullet seems to imply that the match is an inverse match.  i.e. getting with a specific txid would return things that do not match that txid.  Is that the intent?

[jan] Yes, this is the idea. If server and client have the *same* txid, they are already in agreement. Some data can be pruned. If they *differ*, the server should update the client with the latest events.

How does subtree filtering get involved here.  It seems some of this limiting config output based on specific txids may sit in the subtree filtering rather than the main operation.

[jan] The txid mechanism is really orthogonal to filtering. Since a client may use a filter to specify the root of the area of interest, that's also a point where a txid can be attached.

Section 3.5 – Configuration retrieval from the candidate datastore

We should add some wording here about how this might work with private candidates here.  In these there is a high likelihood that the data in the private candidate does not match the running (or global candidate for that matter).  I think this should be somewhat straight forward though, txid requests on a (any type of) candidate datastore would return data from the candidate datastore only.  It is possible that the txid value could change in a candidate through transactions not made by that session (global candidate or continuous rebase mode private candidates) but as long as this work targeting a (any type of) candidate returns data only from that candidate I don’t believe this creates any sort of issues.

[jan] I rephrased a few sentences and titles to allow a reader to think of other candidate datastores than "the candidate" datastore, but I did not introduce any formal dependency. Should we make the link stronger? Can we do that without forcing this document to wait for RFC status of the privcand one?

Section 3.6 – Conditional transactions

Typo in paragraph 3 (confiuguration)
Paragraph 4.  I like the idea to return the new txid with the edit-config response.  I think this should be standard (mandated) behaviour for devices supporting txid.  You did mention a client could request this behaviour (but didn’t detail how) but I think if you just made this the default you wouldn’t need to.  This will also address the use of the “!” etag mentioned later.

[jan] The server returns the txid in the edit-config response iff the client adds the "with-etag" leaf to the request. I don't think we want to return a txid value unless the client requested it, for compatibility (and other) reasons. Not sure what you have in mind for removing the need for "!" as a temporary txid before commit. Would be great if you could elaborate a bit.

Section 3.7 Transactions toward the candidate datastore

I believe this section is equally valid for any type of candidate datastore (incl. privates) – perhaps we just rename the section slightly to “Transactions towards any type of candidate datastore”?
I am not particularly keen on the “!” and its use (see the section top of this email), this is also only briefly covered in this section as a comment to a picture.  Perhaps a reference to where this is discussed in the document would be useful in this comment.

[jan] Right, that was a good suggestion. As mentioned above, I rephrased a bit to cater for this interpretation, without adding any formal dependency on the privcand draft.

The "!" txid value is discussed in sec 3.5, shown in an example in 3.7, explained in conjunction with candidate datastores in 4.3.1, shown in an example in 5.5 and defined in the YANG module in 6.1. We could very well add more explanations, or we can try to find a better mechanism. What would you suggest?

Section 3.9 – Other NETCONF operations

Commit – You mentioned in section 3.6 that the client could request returning the new txid on commit.  Did you have in mind that this behaviour request would be made in the commit operation, if so we should probably mention that here.  I don’t think we should (see my comment in 3.6) but if you think this is the approach, you’d like to take maybe here is the right place in the document.

[jan] The high level example is in sec 3.7 and the low level one in 5.6.

Section 4 – Txid mechanisms

I am not a fan of the current statement in the last paragraph: “If a client uses more than one txid mechanism, such as both etag and last-modified in a particular message to a server, or particular commit, the result is undefined.”.  I think if this is going to be defined as a txid mechanism then we must define what happens when we use both.  Given that the txid etag approach is mandatory and the txid last-modified is optional, this feels a little like the last-modified should be written as an informational annotation rather than an actual txid mechanism.

[jan] I'd be happy to refine the language, but I don't see how the mechanism could work if the client sends more than one txid value for a specific node. Let's say a client issues a get-config request for /acl:acls with the etag-value 1234 and the last-modified "2022-03-15T14:30:21Z". If both the etag-value and last-modified-value match or do not match the server's txid value for that node, I think I know what to do. But what if one matches and the other doesn't. Do we prune the tree or not? For an edit-config, do we go through with the transaction or not?

It's fine for a client to request both in the reply. It's ok for a server to return both. But it's undefined what happens if a client sends more than one txid for a given node. Another alternative could be to tolerate this as long as it's not ambiguous (but what is the use case here?), or to reject such RPC calls outright (rather than saying undefined).

Section 4.3.1 – Candidate datastore

Typo in the second bullet “candidata"
Second bullet – This means that when you submit a number of changes to the candidate they are assigned the etag “!” until <commit> when they are assigned their new values.  I believe the solution at the top of this email might be simpler and more flexible.

[jan] If there always was a way to get the txid value in advance, I think returning it would of course be great. But I believe there are good use cases where the txid cannot be known in advance, so some sort of temporary placeholder value is needed. We don't want to lie, pretending there is no diff. But we also don't know the txid value for the upcoming change, so we don't know but have to say something.

Section 7.2 – Unchanged configuration

Typo in first line: ‘confiuration’


Thanks again for the work.

[jan] Thank you, excellent contribution on all levels! Terribly sorry for the long response time. Your contribution was well received.

Best Regards,
/jan



On 4 Jul 2023, at 15:12, [e-mail] wrote:


A new version of I-D, draft-ietf-netconf-transaction-id-01.txt
has been successfully submitted by Jan Lindblad and posted to the
IETF repository.

Name: draft-ietf-netconf-transaction-id
Revision: 01
Title: Transaction ID Mechanism for NETCONF
Document date: 2023-07-04
Group: netconf
Pages: 70
URL:            https://www.ietf.org/archive/id/draft-ietf-netconf-transaction-id-01.txt
Status:         https://datatracker.ietf.org/doc/draft-ietf-netconf-transaction-id/
Html:           https://www.ietf.org/archive/id/draft-ietf-netconf-transaction-id-01.html
Htmlized:       https://datatracker.ietf.org/doc/html/draft-ietf-netconf-transaction-id
Diff:           https://author-tools.ietf.org/iddiff?url2=draft-ietf-netconf-transaction-id-01

Abstract:
  NETCONF clients and servers often need to have a synchronized view of
  the server's configuration data stores.  The volume of configuration
  data in a server may be very large, while data store changes
  typically are small when observed at typical client resynchronization
  intervals.

  Rereading the entire data store and analyzing the response for
  changes is an inefficient mechanism for synchronization.  This
  document specifies an extension to NETCONF that allows clients and
  servers to keep synchronized with a much smaller data exchange and
  without any need for servers to store information about the clients.




The IETF Secretariat