Re: [netconf] Magnus Westerlund's Discuss on draft-ietf-netconf-subscribed-notifications-25: (with DISCUSS and COMMENT)

["Eric Voit (evoit)" <[e-mail]>]

Hi Magnus,

All items discussed below have now been uploaded as part of:
draft-ietf-netconf-subscribed-notifications-26

Thanks again,
Eric

> From: Magnus Westerlund, May 8, 2019 5:12 AM
> 
> On 2019-05-07 21:22, Eric Voit (evoit) wrote:
> Hi Magnus,
> 
> Thanks again for your thorough review on the overload aspect.   It is
> appreciated.  Some thoughts/proposals back to you in-line..
> 
> From: Magnus Westerlund, May 7, 2019 8:50 AM Hi Eric,
> 
> Thanks for the answers. I think we are making progress here, see below.
> 
> What I see is that there need to be some text changes in the QoS parts.
> 
> Regarding the priority I get the impression that it is not the WG's intention to
> clarify this. However, I would wish that the document is a bit more explicit about
> the lack of any interface to affect the publisher's action when it comes to
> dropping subscriptions. Yes there is this paragraph in Section 1.3:
>    A publisher MAY terminate a dynamic subscription at any time.
>    Similarly, it MAY decide to temporarily suspend the sending of
>    notification messages for any dynamic subscription, or for one or
>    more receivers of a configured subscription.  Such termination or
>    suspension is driven by internal considerations of the publisher.
> 
> So I guess I have to drop this aspect now when the explicit reference to priority
> based actions are removed.
> 
> > <eric> I agree it would be great to have a structure of subscription > handling
> precedence on the publisher. However there are not any > solutions that I know
> of being discussed today. Building a model > here would be hard, and would
> likely take quite a bit of time/effort. > It is primarily the time cost which is the
> biggest consideration at > this point. Since NETCONF got started on these
> subscription > specifications, Google's GNMI alternative has been introduced. >
> GNMI has fewer overload controls, yet has gained industry traction. > Based on
> this, it doesn't seem like standardized handling of > subscription precedence on
> a publisher is a market necessity yet.
> Understood. And I will most definitely not insist on a solution at this time. I was
> considering if there should be more word on that this is something suitable for
> future extension, but I guess it does not provide any good benefit.
> 
> 
> 
> On 2019-05-06 23:19, Eric Voit (evoit) wrote:
> Hi Magnus,
> 
> Thanks very much for your comments.   Thoughts in-line...
> 
> From: Magnus Westerlund, May 2, 2019 8:25 AM
> 
> Magnus Westerlund has entered the following ballot position for
> draft-ietf-netconf-subscribed-notifications-25: Discuss
> 
> When responding, please keep the subject line intact and reply to all email
> addresses included in the To and CC lines. (Feel free to cut this introductory
> paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-netconf-subscribed-notifications/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> My focus when reviewing this document was from a perspective of how to
> handle overload.
> 
> You are absolutely correct to focus on overload.  Mitigating different
> dimensions of the overload risk has been a design goal since this effort's
> inception.  And this is the reason a variety of QoS mechanisms were included in
> the document.
> 
> I have a hard time understanding how this will actually work, especially in a
> fashion that preservers goodput and ensure what is considered the most
> important subscriptions are delivered. Not having good undertanding into
> netconf and restconf don't hesitate to call out likely missunderstanding by me
> and provide clarification and pointers.
> 
> Few of the mechanisms are specific to either RESTCONF or NETCONF.  For
> completeness reasons, let me summarize the overload mechanisms available...
> 
> 1. The publisher is allowed to decline a dynamic subscription.  One of these
> reasons is that the incremental traffic generated by a particular incremental
> subscription will be too high.  There are errors back to the subscriber indicating
> this condition exist.
> 2. A publisher can suspend any subscription for capacity reasons, and a receiver
> must be able to gracefully accept this suspension.
> 3. Much like with HTTP2 streams, higher priority subscriptions intended for a
> particular receiver can be dequeued first from a publisher.
> 4. Much like with HTTP2 streams, proportional subscription dequeuing intended
> for a particular receiver can be performed a publisher.
> 5. DSCP markings can be placed on subscribed data.
> 6. Mechanisms for detecting and reacting to different types of subscribed data
> loss have been embedded.
> 7. Methods have been included to ensure a configured receiver "ok's"
> information push before subscribed data is sent. (This should reduce one DDoS
> vector.) 8. Keep alive mechanisms are established for different transport types,
> so that subscribed data isn't being sent when the is no receiver listening.
> 
> Mechanisms (3) & (4) will likely be seen only with HTTP2 based transports.*   This
> is because (as documented within draft-ietf-netconf-restconf-notif section 4),
> these capabilities are to integrated directly HTTP2 RFC-7450 sections 5.3.1 &
> 5.3.2. To me the big weakness of this mechanism are actually in the API between
> the receiver and the publisher. The current API defined by this document does
> not include any information that allow the receiver to express its actual view on
> priority for a subscription. It has some other tools that has completely different
> meanings namely:
> - Transport level DSCP expressing PHB and routing priority
> - Weight: Proportional bandwidth distribution
> - Dependency: Deliver this only after this other subscription None of this
> expresses anything related to the priority or importance to maintain a particular
> subscription, nor does the interface carry any information regarding what
> delivery requirements the receiver have on the subscription. Thus, the behavior
> in overload situation is going to be totally random.
> So if the WG is okay with this being the case I will hold my nose. However, I think
> the fact that random implementation dependent things will happen in overload
> should be explicitly stated in the document.
> <eric> I think the documents reflect the state-of-the-art across industry
> players.  Attempting to layer-in something new which hasn't been considered
> would be non-trivial.  But your intent should not be lost.  So to reflect your
> request above, how about I add the following statement at the very end of
> Section 2.3 QoS:
> There are additional types over publisher capacity overload which this
> specification does not address within its scope. For example, the prioritization of
> which subscriptions have precedence when the publisher CPU is overloaded is
> not discussed.  As a result, implementation choices will need to be made to
> address such considerations.
> Does this work for you?
> Yes.
> 
> 
> * One background/review note...  Earlier versions of draft-ietf-netconf-
> subscribed-notifications included specific parallels to RFC-7450 when describing
> (3) & (4).  However WG members wanted to abstract away what they felt were
> HTTP2 specific references. This is despite the fact that what was being
> referenced was the desired functional behavior rather than anything HTTP2
> specific.   I can understand the WG reviewers' concern.  This is already a long
> document, and a reader who only cares about NETCONF hopefully won't need
> to wade through complex issues which they are unlikely to worry about in
> deployment.
> I did a brief look at the transports. They did not answer my questions when I
> wrote this discuss. Also, to my understanding the HTTP/2 priority mechanism is
> that is far from easy to use and also intended to distribute resources in an
> ongoing transmission. That doesn't actually match the higher level need of
> determining which subscription are more important than another. Doesn't a 5
> level priority value cover a lot of the missing ground here?
> <eric> Agree that the HTTP2 QoS mechanisms described do not address anything
> other than handling dequeuing congestion between a publisher and a receiver as
> part of an ongoing transmission.  This actually is very relevant problem.  Early
> implementations of telemetry for SDN have many network element publishers
> pushing data to a few central data collectors. Ok Beyond this, I don't think the
> working group has ever considered a multi-level subscription priority
> mechanism.  I believe the problem to be a very hard one, even if there are just 5
> levels (or two levels).
> Ok. Just a suggestion and I agree that solution to this problem will have to be in
> a future extension, if ever.
> 
> A) The QoS and priority sending mechanism discussed in 2.3 and furhter defined
> by the YANG model.
> 
> I do want to raise the usage of the DSCP code point to a discuss. As the DSCP
> defines different PHB and relative priorities in the router queues a DSCP value
> does not provide the publisher any information about priority. I get the feeling
> from the text that this may be intended. If the only intention is to have the
> transport perform differential treatment in the network between the publisher
> and the receiver
> 
> Yes, this is the case.
> 
> there are still more considerations are needed.
> First of all I think these sentence needs a total rewrite:
> 
>    If the publisher supports the "dscp" feature, then a subscription
>    with a "dscp" leaf MUST result in a corresponding [RFC2474] DSCP
>    marking being placed within the IP header of any resulting
>    notification messages and subscription state change notifications.
>    Where TCP is used, a publisher which supports the "dscp" feature
>    SHOULD ensure that a subscription's notification messages are
>    returned within a single TCP transport session where all traffic
>    shares the subscription's "dscp" leaf value.
> 
> I think one need to put a requriement on the transport to use a transport that
> utilize the DSCP marking on its traffic.
> 
> I believe you are asking for a  the publisher respect the DSCP markings for traffic
> egressing an interface on a publisher.  Yes this requirement was assumed, and
> can be explicitly added.
> Please do.
> <eric> Section 2.3 QoS now says..
> A publisher MUST respect the DSCP markings for subscription traffic egressing
> that publisher.
> I think that work, but I like to see it in its full context.
> 
> 
> Which for the current set of connection
> oriented transport protocols, TCP, SCTP, and QUIC all currently only support
> using a single DSCP per connection. Implying multiple transport protocol
> connections using a particular transport to enable this mapping.
> 
> Yes fully agree, a connection would be needed per DSCP.     Is your objection
> with the text above the words "SHOULD ensure" rather than "MUST ensure"?    If
> yes, I can ask the WG objects to whether this requirement should be a MUST.
> 
> If you think you should use RFC 2119 language, yes then I request a
> MUST.  However, I think reformulating this so state it as a fact that different
> DSCP code points requires different transport connections, unless a transport
> supporting multiple DSCP simultanous are used (No standardized option that has
> reliable object or stream semantics exist to my knowledge).
> <eric>  Adding a statement of fact here is not an issue.  And thinking about your
> comment regarding RFC2119 language, I don't think RFC2119 language is
> needed here.  So I have turned the "SHOULD" into "must".   This results in the
> following proposed text:
> Different DSCP code points require different transport connections.  As a result
> where TCP is used, a publisher which supports the "dscp" feature must ensure
> that a subscription's notification messages are returned within a single TCP
> transport session where all traffic shares the subscription's "dscp" leaf value.
> Does this work for you?
> Yes.
> Where TCP is used, a publisher which supports the "dscp" feature must ensure
> that a subscription's notification messages are returned within a single TCP
> transport session where all traffic shares the subscription's "dscp" leaf value.
> 
> A.2 Queuing model of a publisher.
> With the DSCP and the Weight and dependency model I think it is important to
> clarify the model of the queueing in the publisher. So is the intention that several
> subscriptions with different weights and possibly dependencies have their
> individual queues that goes into a scheduler?
> 
> As you know, queuing models are non-trivial.   For that reason we wanted to
> 100% adopt the functional behavior RFC-7540 Section 5.3.1 and 5.3.2 without
> having to re-document/mirror that content at a higher level of the network
> stack.   We are a hoping that a reader of network subscriptions can look to well
> documented, implemented HTTP2 behaviors as applicable.  Providing an
> intermediate layer of functional description could easily result in some mis-
> alignments from what is intended.
> Hmm, that is fine, accept that you added DSCP to the mix, where RFC 7540 is all
> within a single TCP connection. Thus, to avoid complexities I think you at least
> need to be explicit that one can only perform Dependency and Weight
> operations between subscriptions that share DSCP, and that they become
> independent queues.
> <eric> To cover this, are you ok with adding as a last sentence of Section 2.3:
> "Dependency" and "weight" parameters will only be respected and enforced
> between subscriptions that share the same "dscp" leaf value.
> Yes, I think that clarifies the issue.
> To avoid complex queue
> interactions on this level I think there need to be seperate scheduler instances
> per DSCP. I would also note that Dependency mechanism can't ensure that a
> dependent stream arrive at receiver after the identified dependency if they are
> on different DSCP. In fact if one would have HTTP/3 (over QUIC) we may not
> even guarantee it within a single connection and same DSCP due to packet loss
> impact. To me this model and what relationship one need to consider here need
> to be clarified. I think RFC 7540 Section 5.3.1 is an excellent indication of just the
> importance of considering what is in the same dependency tree and what it
> means to have different weighting. To conclude I think this needs a model
> description and clearer definition and possibly requirements towards the
> transport. Espceially if you actually need an in-order delivery requirement?
> 
> I think we have the same technical objective in mind.   And it is absolutely the
> desire to have identical behavior to RFC-7540 Section 5.3.1 and 5.3.2.    For the
> current set of documents before the IESG, we include within draft-ietf-netconf-
> restconf-notif Section 4, a 1:1 mapping between draft-ietf-netconf-subscribed-
> notifications and RFC-7540.
> 
> We are hoping that the transport documents like draft-ietf-netconf-restconf-
> notif can be the place where such supporting documentation and mappings
> occur. I still think that this document needs to have that sentence in their stating
> that Dependency and Weight applies only per subscription sharing DSCP value.
> Because these values are define on this document level, not on the mapping
> level that the individual transport exists on. And this is the interface the WG have
> defined for this. And as it currently stands you appear to have something that
> lacks a clear meaning.
> <eric> Hopefully the last suggested text covers this.
> Yes, it does.
> 
> 
> B. The unpredictability of the circuit breaker overload mechanism.
> 
> My description of the overload handling in this document is that it is a circuit
> breaker based mechanism that can blow a fuse on subscriptions that it fails to
> honor in overload situations. What worries me deply is the total unpredictability
> of this mechanism.
> 
> At the beginning of this email, there are eight methods of overflow handling
> listed.  We optimized these eight mechanisms for different failure scenarios and
> congestion conditions.   Our goal was to depend on existing
> mechanisms/technologies wherever possible.  Reuse of existing mechanisms
> should reduce at least some of the unpredictability.
> 
> First, is it the intention to derive what subscriptions are least important from the
> DSCP, weighting and Dependency parameters? If it is, I think that may be a
> misstake as priority on what subscriptions are most important to retain are not
> necessarily reflected in their QoS parameters.
> 
> At this point the document does not attempt to define "important".  All that is
> done is to provide guidance relating to some elements of network transport.   If
> there is a dimension of "importance" which an implementer would like to layer
> onto this solution, it could be done.  For example, "importance" could applied in
> areas such as what subscriptions should be suspended  in case of CPU
> exhaust.   However guidance on such enhancements are not included in this
> document.
> Still it is both implied and explicitly stated in one place.
> <eric> Hopefully the changes suggested above cover this concern.
> Yes, I think it is sufficiently well handled.
> 
> 
> Secondly, what are the values when a subscription are considered to be to heavy
> or not be handled accordingly. Are there any parameter sets that actually
> describe what SLA the subscriber expect that can be converted into timeout
> timers or buffer depth thresholds to indicate that publisher side isn't delivering
> these in time?
> 
> There is no guidance on this provided in the document.   As equipment types,
> subscription volumes, available memory, will vary between solutions, this will
> take a while to dimension properly.  I can imagine someday that we might have
> something like "Erlang for subscriptions" much like we used Erlangs in the old
> telephony network to dimension call handling capabilities of phone
> switches.  But we are a long way from that.
> To be clear what I mean with SLA here is to express some boundaries on when
> the subscribed to information will be received by the receiver compared to when
> it come to existence on the publisher side. However, I do hear you that there is
> nothing defined and nothing in the pipeline.
> <eric> Understood.  We simply don't have anything.
> 
> 
> Third, I what I understand there are no any additional back pressure mechanism
> between the receiver and the publisher than the transport protocols flow
> control? So a receiver that is not keeping up processing the data it process will
> not read out the data out of the flow controlled buffers in the receiver and thus
> prevent the publisher to write to the transport conncetion, thus causing the
> publisher to eventually trigger a suspension due to its queue build up?
> 
> There is nothing beyond transport flow control.  We thought about it initially,
> but we were not ready to pick up even more complexity than we already had.
> Ok, which just contribute to what I call random things happen at overload.
> <eric> agree
> 
> 
> To my understanding the current mechanism places all subscriptions on the
> same importance and with the same SLA. Thus likely causing short term overload
> situations to cause subscription suspensions in random subscriptions. Is the WG
> fine with this type of randomness occuring and expecting that normally there
> will be such amount of overprovisioning that being able to indicate priority and
> SLA are overkill?
> 
> Yes.   We needed a starting point.  And there are technical solutions which can
> be layered on top.   But what we have now took many years to finalize, and
> should be a big enough technical jump considering our current knowledge.
> 
> At a minimal a change of this sentence in Section 2.5.1 is needed:
> 
>   This could
>    be for reasons of an unexpected but sustained increase in an event
>    stream's event records, degraded CPU capacity, a more complex
>    referenced filter, or other higher priority subscriptions which have
>    usurped resources.
> 
> I have removed "higher priority".
> Thanks.
> 
> 
> As it states that subscriptions has priorities without reference to a mechanism
> that provides that priority.
> 
> C. 2.4.5.  Killing a Dynamic Subscription
> 
>    The "kill-subscription" operation permits an operator to end a
>    dynamic subscription which is not associated with the transport
>    session used for the RPC.  A publisher MUST terminate any dynamic
>    subscription identified by the "id" parameter in the RPC request, if
>    such a subscription exists.
> 
> Can someone please clarify the use case for this functionality. To my
> understanding it allows another receiver given authority to kill the subscription
> being delivered to another receiver. Based on the otherwise rather strict that all
> manipulations of dynamic subscriptions happens from the transport session
> context that established it I want to better understand the use case it appears
> out place.
> 
> A network operator needs a very secure mechanism to end a dynamic
> subscription in progress which it sees as harmful.   The operator cannot do this
> via configuration operations, as the dynamic subscription is not configured (and
> therefore cannot be deleted in the configuration datastore).
> Thanks, that clarification resolves my question mark. So lets close this issue
> without any actions.
> 
> 
> D. The Requirements on a transport supporting Configured Subscriptions
> Reading through Section 2.5 I arrived at a number of questions. I tried to
> understand what the requirements are for the transport that supports
> Configured Subscriptions. I did note that neither draft-ietf-netconf-restconf-
> notif-13 nor
> draft-ietf-netconf-netconf-event-notifications-17 supports configured
> subscriptions. Thus, there appear no template for a solution either, or does
> there exist another document that is being worked on defining such a transport?
> 
> This is the case.   Originally both of those documents *did* include configured
> subscriptions.  However within the WG there was a decision not to progress
> configured subscriptions yet.  One reason is that other YANG model drafts
> moving in the NETCONF WG were seen as pre-requisites for securely creating
> transport sessions via call home for the configured subscriptions.  As a result,
> support for configured subscriptions was extracted from those transport
> documents.   It is expected that that updated versions of just those transport
> documents will be driven when the YANG models complete.
> Look at the responses below I see that we are in one of these situation where
> things are deliberately left open. I will simply assume that this is done in a way
> that supports those future definitions and clarifications. So lets close this issue
> without any actions.
> 
> Questions that arose for me related to Configured Susbription Transport where
> the following: 1. Can Transport Session be initiated in both direction. RFC
> 8071 provides a solution for Publisher to Receiver initiation. It is unclear if all
> parts are in place to have a receiver to publisher initiated transport to be bound
> to the subscription.
> 
> This will be up to a specific transport draft to make this determination.
> 
> To see what might be viable for NETCONF, check out the earlier version of the
> document at:
> https://datatracker.ietf.org/doc/draft-ietf-netconf-netconf-event-
> notifications/10/
> 
> This was seen as a complete version of such a solution.  However the WG
> wanted a YANG model for session parameters discussed in Section 5.2.
> 
> 2. What is "name" really? It appears to be defined as an
> empty container. Despite that it appears to have requirements on being both a
> security identity as well as network address.
> 
> You are correct.  In previous versions of this draft, a receiver was identified by
> the combination of address + port.  However due to the YANG doctor reviews,
> and call home discussions referenced above, the WG wasn't ready to finalize this
> YANG structure.  The compromise was the current structure, plus the example
> YANG model of Appendix A to show how this might be built out.
> 
> 3. In Figure 9, which is stated to be
> for the receiver. What information does the receiver use to determine the
> transition (d)? And what does it do in this step. Related to Discuss part B).
> 
> This determination is implementation specific.
> 
> 4. RFC
> 8071 appears to lack any considerations for how often and how many times it
> attempts to connect to the receiver. So applying that mechanism appears to
> require some usage guidance to avoid causing overload situations or DoS
> potential by misconfiguring network devices with this soltution to dial out
> frequently to a target.
> 
> As the transport solution requirements are not detail it is actually hard to
> determine if there are short comings in the description in Section 2.5 and the
> corresponding YANG model. Is it an reasonable request to document these
> requirements?
> 
> The requirements were documented for both NETCONF and
> RESTCONF.   However these requirements were removed when the WG decided
> to wait until there was a YANG model for RFC-8071 ready to go to the IESG.   For
> a preview on what these requirements might look like, I refer you  to Section 5.2
> of
> https://datatracker.ietf.org/doc/draft-ietf-netconf-netconf-event-
> notifications/10/
> Thanks, so this is clearly something to have in mind when the WG do get to
> specify a solution for this. So I don't see a need for any actions here.
> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> 1. Section 2.3: DSCP domains
> I think the text could benefit from pointing out that the subscriber actually needs
> to know what the DSCP values represents in the diffserv domain of the publisher.
> As these could be different, they also create an interesting problems for
> transports of how they establish a transport connection that uses a particular
> DSCP, as the receiver if initiating need to know the local DSCP value that
> corresponds to the behavior in the publisher's domain.
> 
> This makes sense.
> 
> I have added in Section 5.3 Transport Requirements a new entry which states:
> 
> "A subscriber which includes a "dscp" leaf within an "establish-subscription"
> request will need to understand and consider what the corresponding DSCP
> value represents within the domain of the publisher."
> 
> Let me know if this is not sufficient.
> 
> 2. In general I think there are more than one description that are fuzzy about if it
> describes a publisher or receiver action/requirement.
> 
> It would be great if you have some specifics.   The authors and previous
> reviewers likely have looked at this often enough where things look obvious
> which perhaps are not.
> Sorry, didn't take note about those confusion.
> <eric> Magnus, once again your comments are thoughtful and
> appreciated.  Thanks for taking such a good look at this document.
> Good, I will await the updated document, but I expect to be able to clear with
> the changes discussed here.
> Cheers
> 
> Magnus Westerlund
> 
> ----------------------------------------------------------------------
> Network Architecture & Protocols, Ericsson Research
> ----------------------------------------------------------------------
> Ericsson AB                 | Phone  [phone]
> Torshamnsgatan 23           | Mobile [phone]
> SE-164 80 Stockholm, Sweden | mailto:
> mailto:[e-mail]
> ----------------------------------------------------------------------