Re: [netconf] Magnus Westerlund's Discuss on draft-ietf-netconf-subscribed-notifications-25: (with DISCUSS and COMMENT)

["Eric Voit (evoit)" <[e-mail]>]

Hi Magnus,

Thanks again for your thorough review on the overload aspect.   It is appreciated.  Some thoughts/proposals back to you in-line..

From: Magnus Westerlund, May 7, 2019 8:50 AM

Hi Eric,

Thanks for the answers. I think we are making progress here, see below.

What I see is that there need to be some text changes in the QoS parts.

Regarding the priority I get the impression that it is not the WG's intention to clarify this. However, I would wish that the document is a bit more explicit about the lack of any interface to affect the publisher's action when it comes to dropping subscriptions. Yes there is this paragraph in Section 1.3:

   A publisher MAY terminate a dynamic subscription at any time.

   Similarly, it MAY decide to temporarily suspend the sending of

   notification messages for any dynamic subscription, or for one or

   more receivers of a configured subscription.  Such termination or

   suspension is driven by internal considerations of the publisher.



So I guess I have to drop this aspect now when the explicit reference to priority based actions are removed.



<eric>  I agree it would be great to have a structure of subscription handling precedence on the publisher.  However there are not any solutions that I know of being discussed today.   Building a model here would be hard, and would likely take quite a bit of time/effort.  It is primarily the time cost which is the biggest consideration at this point.  Since NETCONF got started on these subscription specifications, Google's GNMI alternative has been introduced.    GNMI has fewer overload controls, yet has gained industry traction.    Based on this, it doesn't seem like standardized handling of subscription precedence on a publisher is a market necessity yet.



On 2019-05-06 23:19, Eric Voit (evoit) wrote:

Hi Magnus,



Thanks very much for your comments.   Thoughts in-line...



From: Magnus Westerlund, May 2, 2019 8:25 AM



Magnus Westerlund has entered the following ballot position for

draft-ietf-netconf-subscribed-notifications-25: Discuss



When responding, please keep the subject line intact and reply to all email

addresses included in the To and CC lines. (Feel free to cut this introductory

paragraph, however.)





Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html

for more information about IESG DISCUSS and COMMENT positions.





The document, along with other ballot positions, can be found here:

https://datatracker.ietf.org/doc/draft-ietf-netconf-subscribed-notifications/







----------------------------------------------------------------------

DISCUSS:

----------------------------------------------------------------------



My focus when reviewing this document was from a perspective of how to

handle overload.



You are absolutely correct to focus on overload.  Mitigating different dimensions of the overload risk has been a design goal since this effort's inception.  And this is the reason a variety of QoS mechanisms were included in the document.



I have a hard time understanding how this will actually work,

especially in a fashion that preservers goodput and ensure what is considered

the most important subscriptions are delivered. Not having good undertanding

into netconf and restconf don't hesitate to call out likely missunderstanding by

me and provide clarification and pointers.



Few of the mechanisms are specific to either RESTCONF or NETCONF.  For completeness reasons, let me summarize the overload mechanisms available...



1. The publisher is allowed to decline a dynamic subscription.  One of these reasons is that the incremental traffic generated by a particular incremental subscription will be too high.  There are errors back to the subscriber indicating this condition exist.

2. A publisher can suspend any subscription for capacity reasons, and a receiver must be able to gracefully accept this suspension.

3. Much like with HTTP2 streams, higher priority subscriptions intended for a particular receiver can be dequeued first from a publisher.

4. Much like with HTTP2 streams, proportional subscription dequeuing intended for a particular receiver can be performed a publisher.

5. DSCP markings can be placed on subscribed data.

6. Mechanisms for detecting and reacting to different types of subscribed data loss have been embedded.

7. Methods have been included to ensure a configured receiver "ok's" information push before subscribed data is sent. (This should reduce one DDoS vector.)

8. Keep alive mechanisms are established for different transport types, so that subscribed data isn't being sent when the is no receiver listening.



Mechanisms (3) & (4) will likely be seen only with HTTP2 based transports.*   This is because (as documented within draft-ietf-netconf-restconf-notif section 4), these capabilities are to integrated directly HTTP2 RFC-7450 sections 5.3.1 & 5.3.2.

To me the big weakness of this mechanism are actually in the API between the receiver and the publisher. The current API defined by this document does not include any information that allow the receiver to express its actual view on priority for a subscription. It has some other tools that has completely different meanings namely:

- Transport level DSCP expressing PHB and routing priority

- Weight: Proportional bandwidth distribution

- Dependency: Deliver this only after this other subscription

None of this expresses anything related to the priority or importance to maintain a particular subscription, nor does the interface carry any information regarding what delivery requirements the receiver have on the subscription. Thus, the behavior in overload situation is going to be totally random.

So if the WG is okay with this being the case I will hold my nose. However, I think the fact that random implementation dependent things will happen in overload should be explicitly stated in the document.

<eric> I think the documents reflect the state-of-the-art across industry players.  Attempting to layer-in something new which hasn't been considered would be non-trivial.  But your intent should not be lost.  So to reflect your request above, how about I add the following statement at the very end of Section 2.3 QoS:

There are additional types over publisher capacity overload which this specification does not address within its scope. For example, the prioritization of which subscriptions have precedence when the publisher CPU is overloaded is not discussed.  As a result, implementation choices will need to be made to address such considerations.

Does this work for you?





* One background/review note...  Earlier versions of draft-ietf-netconf-subscribed-notifications included specific parallels to RFC-7450 when describing (3) & (4).  However WG members wanted to abstract away what they felt were HTTP2 specific references. This is despite the fact that what was being referenced was the desired functional behavior rather than anything HTTP2 specific.   I can understand the WG reviewers' concern.  This is already a long document, and a reader who only cares about NETCONF hopefully won't need to wade through complex issues which they are unlikely to worry about in deployment.

I did a brief look at the transports. They did not answer my questions when I wrote this discuss. Also, to my understanding the HTTP/2 priority mechanism is that is far from easy to use and also intended to distribute resources in an ongoing transmission. That doesn't actually match the higher level need of determining which subscription are more important than another. Doesn't a 5 level priority value cover a lot of the missing ground here?

<eric> Agree that the HTTP2 QoS mechanisms described do not address anything other than handling dequeuing congestion between a publisher and a receiver as part of an ongoing transmission.  This actually is very relevant problem.  Early implementations of telemetry for SDN have many network element publishers pushing data to a few central data collectors.

Beyond this, I don't think the working group has ever considered a multi-level subscription priority mechanism.  I believe the problem to be a very hard one, even if there are just 5 levels (or two levels).

A) The QoS and priority sending mechanism discussed in 2.3 and furhter defined

by the YANG model.



I do want to raise the usage of the DSCP code point to a discuss. As the DSCP

defines different PHB and relative priorities in the router queues a DSCP value

does not provide the publisher any information about priority. I get the feeling

from the text that this may be intended. If the only intention is to have the

transport perform differential treatment in the network between the publisher

and the receiver



Yes, this is the case.



there are still more considerations are needed.

First of all I think these sentence needs a total rewrite:



   If the publisher supports the "dscp" feature, then a subscription

   with a "dscp" leaf MUST result in a corresponding [RFC2474] DSCP

   marking being placed within the IP header of any resulting

   notification messages and subscription state change notifications.

   Where TCP is used, a publisher which supports the "dscp" feature

   SHOULD ensure that a subscription's notification messages are

   returned within a single TCP transport session where all traffic

   shares the subscription's "dscp" leaf value.



I think one need to put a requriement on the transport to use a transport that

utilize the DSCP marking on its traffic.



I believe you are asking for a  the publisher respect the DSCP markings for traffic egressing an interface on a publisher.  Yes this requirement was assumed, and can be explicitly added.

Please do.

<eric> Section 2.3 QoS now says..

A publisher MUST respect the DSCP markings for subscription traffic egressing that publisher.





Which for the current set of connection

oriented transport protocols, TCP, SCTP, and QUIC all currently only support

using a single DSCP per connection. Implying multiple transport protocol

connections using a particular transport to enable this mapping.



Yes fully agree, a connection would be needed per DSCP.     Is your objection with the text above the words "SHOULD ensure" rather than "MUST ensure"?    If yes, I can ask the WG objects to whether this requirement should be a MUST.



If you think you should use RFC 2119 language, yes then I request a MUST.  However, I think reformulating this so state it as a fact that different DSCP code points requires different transport connections, unless a transport supporting multiple DSCP simultanous are used (No standardized option that has reliable object or stream semantics exist to my knowledge).

<eric>  Adding a statement of fact here is not an issue.  And thinking about your comment regarding RFC2119 language, I don't think RFC2119 language is needed here.  So I have turned the "SHOULD" into "must".   This results in the following proposed text:

Different DSCP code points require different transport connections.  As a result where TCP is used, a publisher which supports the "dscp" feature must ensure that a subscription's notification messages are returned within a single TCP transport session where all traffic shares the subscription's "dscp" leaf value.

Does this work for you?

Where TCP is used, a publisher which supports the "dscp" feature must ensure that a subscription's notification messages are returned within a single TCP transport session where all traffic shares the subscription's "dscp" leaf value.



A.2 Queuing model of a publisher.

With the DSCP and the Weight and dependency model I think it is important to

clarify the model of the queueing in the publisher. So is the intention that several

subscriptions with different weights and possibly dependencies have their

individual queues that goes into a scheduler?



As you know, queuing models are non-trivial.   For that reason we wanted to 100% adopt the functional behavior RFC-7540 Section 5.3.1 and 5.3.2 without having to re-document/mirror that content at a higher level of the network stack.   We are a hoping that a reader of network subscriptions can look to well documented, implemented HTTP2 behaviors as applicable.  Providing an intermediate layer of functional description could easily result in some mis-alignments from what is intended.

Hmm, that is fine, accept that you added DSCP to the mix, where RFC 7540 is all within a single TCP connection. Thus, to avoid complexities I think you at least need to be explicit that one can only perform Dependency and Weight operations between subscriptions that share DSCP, and that they become independent queues.

<eric> To cover this, are you ok with adding as a last sentence of Section 2.3:

"Dependency" and "weight" parameters will only be respected and enforced between subscriptions that share the same "dscp" leaf value.

To avoid complex queue

interactions on this level I think there need to be seperate scheduler instances

per DSCP. I would also note that Dependency mechanism can't ensure that a

dependent stream arrive at receiver after the identified dependency if they are

on different DSCP. In fact if one would have HTTP/3 (over QUIC) we may not

even guarantee it within a single connection and same DSCP due to packet loss

impact. To me this model and what relationship one need to consider here need

to be clarified. I think RFC 7540 Section 5.3.1 is an excellent indication of just the

importance of considering what is in the same dependency tree and what it

means to have different weighting. To conclude I think this needs a model

description and clearer definition and possibly requirements towards the

transport. Espceially if you actually need an in-order delivery requirement?



I think we have the same technical objective in mind.   And it is absolutely the desire to have identical behavior to RFC-7540 Section 5.3.1 and 5.3.2.    For the current set of documents before the IESG, we include within draft-ietf-netconf-restconf-notif Section 4, a 1:1 mapping between draft-ietf-netconf-subscribed-notifications and RFC-7540.



We are hoping that the transport documents like draft-ietf-netconf-restconf-notif can be the place where such supporting documentation and mappings occur.

I still think that this document needs to have that sentence in their stating that Dependency and Weight applies only per subscription sharing DSCP value. Because these values are define on this document level, not on the mapping level that the individual transport exists on. And this is the interface the WG have defined for this. And as it currently stands you appear to have something that lacks a clear meaning.

<eric> Hopefully the last suggested text covers this.



B. The unpredictability of the circuit breaker overload mechanism.



My description of the overload handling in this document is that it is a circuit

breaker based mechanism that can blow a fuse on subscriptions that it fails to

honor in overload situations. What worries me deply is the total unpredictability

of this mechanism.



At the beginning of this email, there are eight methods of overflow handling listed.  We optimized these eight mechanisms for different failure scenarios and congestion conditions.   Our goal was to depend on existing mechanisms/technologies wherever possible.  Reuse of existing mechanisms should reduce at least some of the unpredictability.



First, is it the intention to derive what subscriptions are least important from the

DSCP, weighting and Dependency parameters? If it is, I think that may be a

misstake as priority on what subscriptions are most important to retain are not

necessarily reflected in their QoS parameters.



At this point the document does not attempt to define "important".  All that is done is to provide guidance relating to some elements of network transport.   If there is a dimension of "importance" which an implementer would like to layer onto this solution, it could be done.  For example, "importance" could applied in areas such as what subscriptions should be suspended  in case of CPU exhaust.   However guidance on such enhancements are not included in this document.

Still it is both implied and explicitly stated in one place.

<eric> Hopefully the changes suggested above cover this concern.



Secondly, what are the values when a subscription are considered to be to heavy

or not be handled accordingly. Are there any parameter sets that actually

describe what SLA the subscriber expect that can be converted into timeout

timers or buffer depth thresholds to indicate that publisher side isn't delivering

these in time?



There is no guidance on this provided in the document.   As equipment types, subscription volumes, available memory, will vary between solutions, this will take a while to dimension properly.  I can imagine someday that we might have something like "Erlang for subscriptions" much like we used Erlangs in the old telephony network to dimension call handling capabilities of phone switches.  But we are a long way from that.

To be clear what I mean with SLA here is to express some boundaries on when the subscribed to information will be received by the receiver compared to when it come to existence on the publisher side. However, I do hear you that there is nothing defined and nothing in the pipeline.

<eric> Understood.  We simply don't have anything.





Third, I what I understand there are no any additional back pressure mechanism

between the receiver and the publisher than the transport protocols flow

control? So a receiver that is not keeping up processing the data it process will

not read out the data out of the flow controlled buffers in the receiver and thus

prevent the publisher to write to the transport conncetion, thus causing the

publisher to eventually trigger a suspension due to its queue build up?



There is nothing beyond transport flow control.  We thought about it initially, but we were not ready to pick up even more complexity than we already had.

Ok, which just contribute to what I call random things happen at overload.

<eric> agree





To my understanding the current mechanism places all subscriptions on the

same importance and with the same SLA. Thus likely causing short term overload

situations to cause subscription suspensions in random subscriptions. Is the WG

fine with this type of randomness occuring and expecting that normally there

will be such amount of overprovisioning that being able to indicate priority and

SLA are overkill?



Yes.   We needed a starting point.  And there are technical solutions which can be layered on top.   But what we have now took many years to finalize, and should be a big enough technical jump considering our current knowledge.



At a minimal a change of this sentence in Section 2.5.1 is needed:



  This could

   be for reasons of an unexpected but sustained increase in an event

   stream's event records, degraded CPU capacity, a more complex

   referenced filter, or other higher priority subscriptions which have

   usurped resources.



I have removed "higher priority".

Thanks.





As it states that subscriptions has priorities without reference to a mechanism

that provides that priority.



C. 2.4.5.  Killing a Dynamic Subscription



   The "kill-subscription" operation permits an operator to end a

   dynamic subscription which is not associated with the transport

   session used for the RPC.  A publisher MUST terminate any dynamic

   subscription identified by the "id" parameter in the RPC request, if

   such a subscription exists.



Can someone please clarify the use case for this functionality. To my

understanding it allows another receiver given authority to kill the subscription

being delivered to another receiver. Based on the otherwise rather strict that all

manipulations of dynamic subscriptions happens from the transport session

context that established it I want to better understand the use case it appears

out place.



A network operator needs a very secure mechanism to end a dynamic subscription in progress which it sees as harmful.   The operator cannot do this via configuration operations, as the dynamic subscription is not configured (and therefore cannot be deleted in the configuration datastore).

Thanks, that clarification resolves my question mark. So lets close this issue without any actions.





D. The Requirements on a transport supporting Configured Subscriptions

Reading through Section 2.5 I arrived at a number of questions. I tried to

understand what the requirements are for the transport that supports

Configured Subscriptions. I did note that neither draft-ietf-netconf-restconf-

notif-13 nor

draft-ietf-netconf-netconf-event-notifications-17 supports configured

subscriptions. Thus, there appear no template for a solution either, or does

there exist another document that is being worked on defining such a transport?



This is the case.   Originally both of those documents *did* include configured subscriptions.  However within the WG there was a decision not to progress configured subscriptions yet.  One reason is that other YANG model drafts moving in the NETCONF WG were seen as pre-requisites for securely creating transport sessions via call home for the configured subscriptions.  As a result, support for configured subscriptions was extracted from those transport documents.   It is expected that that updated versions of just those transport documents will be driven when the YANG models complete.

Look at the responses below I see that we are in one of these situation where things are deliberately left open. I will simply assume that this is done in a way that supports those future definitions and clarifications. So lets close this issue without any actions.



Questions that arose for me related to Configured Susbription Transport where

the following: 1. Can Transport Session be initiated in both direction. RFC

8071 provides a solution for Publisher to Receiver initiation. It is unclear if all

parts are in place to have a receiver to publisher initiated transport to be bound

to the subscription.



This will be up to a specific transport draft to make this determination.



To see what might be viable for NETCONF, check out the earlier version of the document at:

https://datatracker.ietf.org/doc/draft-ietf-netconf-netconf-event-notifications/10/



This was seen as a complete version of such a solution.  However the WG wanted a YANG model for session parameters discussed in Section 5.2.



2. What is "name" really? It appears to be defined as an

empty container. Despite that it appears to have requirements on being both a

security identity as well as network address.



You are correct.  In previous versions of this draft, a receiver was identified by the combination of address + port.  However due to the YANG doctor reviews, and call home discussions referenced above, the WG wasn't ready to finalize this YANG structure.  The compromise was the current structure, plus the example YANG model of Appendix A to show how this might be built out.



3. In Figure 9, which is stated to be

for the receiver. What information does the receiver use to determine the

transition (d)? And what does it do in this step. Related to Discuss part B).



This determination is implementation specific.



4. RFC

8071 appears to lack any considerations for how often and how many times it

attempts to connect to the receiver. So applying that mechanism appears to

require some usage guidance to avoid causing overload situations or DoS

potential by misconfiguring network devices with this soltution to dial out

frequently to a target.



As the transport solution requirements are not detail it is actually hard to

determine if there are short comings in the description in Section 2.5 and the

corresponding YANG model. Is it an reasonable request to document these

requirements?



The requirements were documented for both NETCONF and RESTCONF.   However these requirements were removed when the WG decided to wait until there was a YANG model for RFC-8071 ready to go to the IESG.   For a preview on what these requirements might look like, I refer you  to Section 5.2 of

https://datatracker.ietf.org/doc/draft-ietf-netconf-netconf-event-notifications/10/

Thanks, so this is clearly something to have in mind when the WG do get to specify a solution for this. So I don't see a need for any actions here.





----------------------------------------------------------------------

COMMENT:

----------------------------------------------------------------------



1. Section 2.3: DSCP domains

I think the text could benefit from pointing out that the subscriber actually needs

to know what the DSCP values represents in the diffserv domain of the publisher.

As these could be different, they also create an interesting problems for

transports of how they establish a transport connection that uses a particular

DSCP, as the receiver if initiating need to know the local DSCP value that

corresponds to the behavior in the publisher's domain.



This makes sense.



I have added in Section 5.3 Transport Requirements a new entry which states:



"A subscriber which includes a "dscp" leaf within an "establish-subscription" request will need to understand and consider what the corresponding DSCP value represents within the domain of the publisher."



Let me know if this is not sufficient.



2. In general I think there are more than one description that are fuzzy about if it

describes a publisher or receiver action/requirement.



It would be great if you have some specifics.   The authors and previous reviewers likely have looked at this often enough where things look obvious which perhaps are not.

Sorry, didn't take note about those confusion.

<eric> Magnus, once again your comments are thoughtful and appreciated.  Thanks for taking such a good look at this document.

Eric



Cheers



Magnus Westerlund



----------------------------------------------------------------------

Network Architecture & Protocols, Ericsson Research

----------------------------------------------------------------------

Ericsson AB                 | Phone  [phone]

Torshamnsgatan 23           | Mobile [phone]

SE-164 80 Stockholm, Sweden | mailto: [e-mail]<mailto:[e-mail]>

----------------------------------------------------------------------