Re: [netconf] Magnus Westerlund's Discuss on draft-ietf-netconf-subscribed-notifications-25: (with DISCUSS and COMMENT)

Magnus Westerlund <> Tue, 07 May 2019 12:50 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 5897412012C; Tue, 7 May 2019 05:50:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.009
X-Spam-Status: No, score=-2.009 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_HIGH=-0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id Z9oSBZ-oZKxH; Tue, 7 May 2019 05:50:10 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id EA790120058; Tue, 7 May 2019 05:50:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zniLpG1kOgiKTAaO2i7nKpLjdFZ25QqvTRHutinvI5c=; b=FGUU5DwjJx+IjK1z1BWtgIxBZ18gGhJwE5kyU0VptVin7GUaLgib20Q4PjFDK10cN0lOn2X7xtug4OpPOuwJayJqMBSgD6Qz/TkBoBexRhJN1dgizHDuPsfp9Vu8Rp5RINcOmMsqMArWDcIvmHaDHARVRIVD7LADsp0O32JBy0g=
Received: from ( by ( with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1878.19; Tue, 7 May 2019 12:50:06 +0000
Received: from ([fe80::b161:fb77:e4ea:4723]) by ([fe80::b161:fb77:e4ea:4723%3]) with mapi id 15.20.1878.019; Tue, 7 May 2019 12:50:06 +0000
From: Magnus Westerlund <>
To: "Eric Voit (evoit)" <>, The IESG <>
CC: "" <>, Kent Watsen <>, "" <>, "" <>
Thread-Topic: Magnus Westerlund's Discuss on draft-ietf-netconf-subscribed-notifications-25: (with DISCUSS and COMMENT)
Thread-Index: AQHVAOIY2D7aivy2PU25mAR0nTlyqA==
Date: Tue, 07 May 2019 12:50:06 +0000
Message-ID: <>
References: <> <>
Accept-Language: sv-SE, en-US
Content-Language: en-US
authentication-results: spf=none (sender IP is );
x-originating-ip: []
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: c549dbd7-0f74-41f0-a0c9-08d6d2ea8815
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(2017052603328)(7193020); SRVR:HE1PR0701MB2523;
x-ms-traffictypediagnostic: HE1PR0701MB2523:
x-ms-exchange-purlcount: 3
x-microsoft-antispam-prvs: <>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-forefront-prvs: 0030839EEE
x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(39860400002)(376002)(346002)(396003)(366004)(51914003)(51444003)(85664002)(189003)(199004)(54906003)(4326008)(110136005)(73956011)(66946007)(66476007)(66556008)(64756008)(76116006)(33656002)(102836004)(99286004)(6506007)(52536014)(76176011)(7696005)(2906002)(6246003)(6116002)(53546011)(74316002)(14454004)(66066001)(966005)(53936002)(3846002)(68736007)(15650500001)(478600001)(9686003)(476003)(54896002)(446003)(236005)(6306002)(55016002)(44832011)(486006)(21615005)(7736002)(6436002)(71190400001)(71200400001)(30864003)(5660300002)(66446008)(256004)(14444005)(66574012)(8676002)(81156014)(81166006)(8936002)(86362001)(316002)(25786009)(229853002)(186003)(26005)(606006); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0701MB2523;; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None ( does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: u2Ofn+MmKhyduvbAf6ZwW+5tZdDHcgwpjvAVMSp/tTar7QuoPhBAs5/9z+QxJ/pJU+y4+S/XPFS9gfoRXQWkBRSA1cCnU71WBLD1wXlFD4t+NS5AQcEOKovwCilBILwQhua+JCYRTnTj6AnzawnqZ2L2se2YvGBFvjRR9usgWbDN3HrSvpMJ1s1XQdEyaCuAPS7MpUe7BPeCzN2mkAK5ZUSbc2wBbgAffp7fcD9GUelCqxYLfn2+hF60/kEm/a42PdCoDBGOY9vnLeOJYc+Brxb2lWPxF7e+dn13xqzdjhGpVVo53a/lW3BlsU9RCjU+7yrW1yfToEMuNxswOaaRw6fjIfKYMFbkcjfTfFBpOS+L6Cmn3/dR/hNyosTmD9H3FV8vfzAJLSdS5fYjK2Z4vjk/yxu3+sN7k9WkJkyviHU=
Content-Type: multipart/alternative; boundary="_000_HE1PR0701MB2522DA9FB3E9C9F0FB25597E95310HE1PR0701MB2522_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: c549dbd7-0f74-41f0-a0c9-08d6d2ea8815
X-MS-Exchange-CrossTenant-originalarrivaltime: 07 May 2019 12:50:06.4921 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0701MB2523
Archived-At: <>
Subject: Re: [netconf] Magnus Westerlund's Discuss on draft-ietf-netconf-subscribed-notifications-25: (with DISCUSS and COMMENT)
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 07 May 2019 12:50:19 -0000

Hi Eric,

Thanks for the answers. I think we are making progress here, see below.

What I see is that there need to be some text changes in the QoS parts.

Regarding the priority I get the impression that it is not the WG's intention to clarify this. However, I would wish that the document is a bit more explicit about the lack of any interface to affect the publisher's action when it comes to dropping subscriptions. Yes there is this paragraph in Section 1.3:

   A publisher MAY terminate a dynamic subscription at any time.
   Similarly, it MAY decide to temporarily suspend the sending of
   notification messages for any dynamic subscription, or for one or
   more receivers of a configured subscription.  Such termination or
   suspension is driven by internal considerations of the publisher.

So I guess I have to drop this aspect now when the explicit reference to priority based actions are removed.

On 2019-05-06 23:19, Eric Voit (evoit) wrote:

Hi Magnus,

Thanks very much for your comments.   Thoughts in-line...

From: Magnus Westerlund, May 2, 2019 8:25 AM

Magnus Westerlund has entered the following ballot position for
draft-ietf-netconf-subscribed-notifications-25: Discuss

When responding, please keep the subject line intact and reply to all email
addresses included in the To and CC lines. (Feel free to cut this introductory
paragraph, however.)

Please refer to
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:


My focus when reviewing this document was from a perspective of how to
handle overload.

You are absolutely correct to focus on overload.  Mitigating different dimensions of the overload risk has been a design goal since this effort’s inception.  And this is the reason a variety of QoS mechanisms were included in the document.

I have a hard time understanding how this will actually work,
especially in a fashion that preservers goodput and ensure what is considered
the most important subscriptions are delivered. Not having good undertanding
into netconf and restconf don't hesitate to call out likely missunderstanding by
me and provide clarification and pointers.

Few of the mechanisms are specific to either RESTCONF or NETCONF.  For completeness reasons, let me summarize the overload mechanisms available...

1. The publisher is allowed to decline a dynamic subscription.  One of these reasons is that the incremental traffic generated by a particular incremental subscription will be too high.  There are errors back to the subscriber indicating this condition exist.
2. A publisher can suspend any subscription for capacity reasons, and a receiver must be able to gracefully accept this suspension.
3. Much like with HTTP2 streams, higher priority subscriptions intended for a particular receiver can be dequeued first from a publisher.
4. Much like with HTTP2 streams, proportional subscription dequeuing intended for a particular receiver can be performed a publisher.
5. DSCP markings can be placed on subscribed data.
6. Mechanisms for detecting and reacting to different types of subscribed data loss have been embedded.
7. Methods have been included to ensure a configured receiver “ok’s” information push before subscribed data is sent. (This should reduce one DDoS vector.)
8. Keep alive mechanisms are established for different transport types, so that subscribed data isn’t being sent when the is no receiver listening.

Mechanisms (3) & (4) will likely be seen only with HTTP2 based transports.*   This is because (as documented within draft-ietf-netconf-restconf-notif section 4), these capabilities are to integrated directly HTTP2 RFC-7450 sections 5.3.1 & 5.3.2.

To me the big weakness of this mechanism are actually in the API between the receiver and the publisher. The current API defined by this document does not include any information that allow the receiver to express its actual view on priority for a subscription. It has some other tools that has completely different meanings namely:

- Transport level DSCP expressing PHB and routing priority

- Weight: Proportional bandwidth distribution

- Dependency: Deliver this only after this other subscription

None of this expresses anything related to the priority or importance to maintain a particular subscription, nor does the interface carry any information regarding what delivery requirements the receiver have on the subscription. Thus, the behavior in overload situation is going to be totally random.

So if the WG is okay with this being the case I will hold my nose. However, I think the fact that random implementation dependent things will happen in overload should be explicitly stated in the document.

* One background/review note...  Earlier versions of draft-ietf-netconf-subscribed-notifications included specific parallels to RFC-7450 when describing (3) & (4).  However WG members wanted to abstract away what they felt were HTTP2 specific references. This is despite the fact that what was being referenced was the desired functional behavior rather than anything HTTP2 specific.   I can understand the WG reviewers' concern.  This is already a long document, and a reader who only cares about NETCONF hopefully won't need to wade through complex issues which they are unlikely to worry about in deployment.

I did a brief look at the transports. They did not answer my questions when I wrote this discuss. Also, to my understanding the HTTP/2 priority mechanism is that is far from easy to use and also intended to distribute resources in an ongoing transmission. That doesn't actually match the higher level need of determining which subscription are more important than another. Doesn't a 5 level priority value cover a lot of the missing ground here?

A) The QoS and priority sending mechanism discussed in 2.3 and furhter defined
by the YANG model.

I do want to raise the usage of the DSCP code point to a discuss. As the DSCP
defines different PHB and relative priorities in the router queues a DSCP value
does not provide the publisher any information about priority. I get the feeling
from the text that this may be intended. If the only intention is to have the
transport perform differential treatment in the network between the publisher
and the receiver

Yes, this is the case.

there are still more considerations are needed.
First of all I think these sentence needs a total rewrite:

   If the publisher supports the "dscp" feature, then a subscription
   with a "dscp" leaf MUST result in a corresponding [RFC2474] DSCP
   marking being placed within the IP header of any resulting
   notification messages and subscription state change notifications.
   Where TCP is used, a publisher which supports the "dscp" feature
   SHOULD ensure that a subscription's notification messages are
   returned within a single TCP transport session where all traffic
   shares the subscription's "dscp" leaf value.

I think one need to put a requriement on the transport to use a transport that
utilize the DSCP marking on its traffic.

I believe you are asking for a  the publisher respect the DSCP markings for traffic egressing an interface on a publisher.  Yes this requirement was assumed, and can be explicitly added.

Please do.

Which for the current set of connection
oriented transport protocols, TCP, SCTP, and QUIC all currently only support
using a single DSCP per connection. Implying multiple transport protocol
connections using a particular transport to enable this mapping.

Yes fully agree, a connection would be needed per DSCP.     Is your objection with the text above the words "SHOULD ensure" rather than "MUST ensure"?    If yes, I can ask the WG objects to whether this requirement should be a MUST.

If you think you should use RFC 2119 language, yes then I reuqest a MUST. However, I think reformulating this so state it as a fact that different DSCP code points requires different transport connections, unless a transport supporting multiple DSCP simultanous are used (No standardized option that has reliable object or stream semantics exist to my knowledge).

A.2 Queuing model of a publisher.
With the DSCP and the Weight and dependency model I think it is important to
clarify the model of the queueing in the publisher. So is the intention that several
subscriptions with different weights and possibly dependencies have their
individual queues that goes into a scheduler?

As you know, queuing models are non-trivial.   For that reason we wanted to 100% adopt the functional behavior RFC-7540 Section 5.3.1 and 5.3.2 without having to re-document/mirror that content at a higher level of the network stack.   We are a hoping that a reader of network subscriptions can look to well documented, implemented HTTP2 behaviors as applicable.  Providing an intermediate layer of functional description could easily result in some mis-alignments from what is intended.

Hmm, that is fine, accept that you added DSCP to the mix, where RFC 7540 is all within a single TCP connection. Thus, to avoid complexities I think you at least need to be explicit that one can only perform Dependency and Weight operations between subscriptions that share DSCP, and that they become independent queues.

To avoid complex queue
interactions on this level I think there need to be seperate scheduler instances
per DSCP. I would also note that Dependency mechanism can't ensure that a
dependent stream arrive at receiver after the identified dependency if they are
on different DSCP. In fact if one would have HTTP/3 (over QUIC) we may not
even guarantee it within a single connection and same DSCP due to packet loss
impact. To me this model and what relationship one need to consider here need
to be clarified. I think RFC 7540 Section 5.3.1 is an excellent indication of just the
importance of considering what is in the same dependency tree and what it
means to have different weighting. To conclude I think this needs a model
description and clearer definition and possibly requirements towards the
transport. Espceially if you actually need an in-order delivery requirement?

I think we have the same technical objective in mind.   And it is absolutely the desire to have identical behavior to RFC-7540 Section 5.3.1 and 5.3.2.    For the current set of documents before the IESG, we include within draft-ietf-netconf-restconf-notif Section 4, a 1:1 mapping between draft-ietf-netconf-subscribed-notifications and RFC-7540.

We are hoping that the transport documents like draft-ietf-netconf-restconf-notif can be the place where such supporting documentation and mappings occur.

I still think that this document needs to have that sentence in their stating that Dependency and Weight applies only per subscription sharing DSCP value. Because these values are define on this document level, not on the mapping level that the individual transport exists on. And this is the interface the WG have defined for this. And as it currently stands you appear to have something that lacks a clear meaning.

B. The unpredictability of the circuit breaker overload mechanism.

My description of the overload handling in this document is that it is a circuit
breaker based mechanism that can blow a fuse on subscriptions that it fails to
honor in overload situations. What worries me deply is the total unpredictability
of this mechanism.

At the beginning of this email, there are eight methods of overflow handling listed.  We optimized these eight mechanisms for different failure scenarios and congestion conditions.   Our goal was to depend on existing mechanisms/technologies wherever possible.  Reuse of existing mechanisms should reduce at least some of the unpredictability.

First, is it the intention to derive what subscriptions are least important from the
DSCP, weighting and Dependency parameters? If it is, I think that may be a
misstake as priority on what subscriptions are most important to retain are not
necessarily reflected in their QoS parameters.

At this point the document does not attempt to define "important".  All that is done is to provide guidance relating to some elements of network transport.   If there is a dimension of "importance" which an implementer would like to layer onto this solution, it could be done.  For example, "importance" could applied in areas such as what subscriptions should be suspended  in case of CPU exhaust.   However guidance on such enhancements are not included in this document.

Still it is both implied and explicitly stated in one place.

Secondly, what are the values when a subscription are considered to be to heavy
or not be handled accordingly. Are there any parameter sets that actually
describe what SLA the subscriber expect that can be converted into timeout
timers or buffer depth thresholds to indicate that publisher side isn't delivering
these in time?

There is no guidance on this provided in the document.   As equipment types, subscription volumes, available memory, will vary between solutions, this will take a while to dimension properly.  I can imagine someday that we might have something like "Erlang for subscriptions" much like we used Erlangs in the old telephony network to dimension call handling capabilities of phone switches.  But we are a long way from that.

To be clear what I mean with SLA here is to express some boundaries on when the subscribed to information will be received by the receiver compared to when it come to existence on the publisher side. However, I do hear you that there is nothing defined and nothing in the pipeline.

Third, I what I understand there are no any additional back pressure mechanism
between the receiver and the publisher than the transport protocols flow
control? So a receiver that is not keeping up processing the data it process will
not read out the data out of the flow controlled buffers in the receiver and thus
prevent the publisher to write to the transport conncetion, thus causing the
publisher to eventually trigger a suspension due to its queue build up?

There is nothing beyond transport flow control.  We thought about it initially, but we were not ready to pick up even more complexity than we already had.

Ok, which just contribute to what I call random things happen at overload.

To my understanding the current mechanism places all subscriptions on the
same importance and with the same SLA. Thus likely causing short term overload
situations to cause subscription suspensions in random subscriptions. Is the WG
fine with this type of randomness occuring and expecting that normally there
will be such amount of overprovisioning that being able to indicate priority and
SLA are overkill?

Yes.   We needed a starting point.  And there are technical solutions which can be layered on top.   But what we have now took many years to finalize, and should be a big enough technical jump considering our current knowledge.

At a minimal a change of this sentence in Section 2.5.1 is needed:

  This could
   be for reasons of an unexpected but sustained increase in an event
   stream's event records, degraded CPU capacity, a more complex
   referenced filter, or other higher priority subscriptions which have
   usurped resources.

I have removed "higher priority".


As it states that subscriptions has priorities without reference to a mechanism
that provides that priority.

C. 2.4.5.  Killing a Dynamic Subscription

   The "kill-subscription" operation permits an operator to end a
   dynamic subscription which is not associated with the transport
   session used for the RPC.  A publisher MUST terminate any dynamic
   subscription identified by the "id" parameter in the RPC request, if
   such a subscription exists.

Can someone please clarify the use case for this functionality. To my
understanding it allows another receiver given authority to kill the subscription
being delivered to another receiver. Based on the otherwise rather strict that all
manipulations of dynamic subscriptions happens from the transport session
context that established it I want to better understand the use case it appears
out place.

A network operator needs a very secure mechanism to end a dynamic subscription in progress which it sees as harmful.   The operator cannot do this via configuration operations, as the dynamic subscription is not configured (and therefore cannot be deleted in the configuration datastore).

Thanks, that clarification resolves my question mark. So lets close this issue without any actions.

D. The Requirements on a transport supporting Configured Subscriptions
Reading through Section 2.5 I arrived at a number of questions. I tried to
understand what the requirements are for the transport that supports
Configured Subscriptions. I did note that neither draft-ietf-netconf-restconf-
notif-13 nor
draft-ietf-netconf-netconf-event-notifications-17 supports configured
subscriptions. Thus, there appear no template for a solution either, or does
there exist another document that is being worked on defining such a transport?

This is the case.   Originally both of those documents *did* include configured subscriptions.  However within the WG there was a decision not to progress configured subscriptions yet.  One reason is that other YANG model drafts moving in the NETCONF WG were seen as pre-requisites for securely creating transport sessions via call home for the configured subscriptions.  As a result, support for configured subscriptions was extracted from those transport documents.   It is expected that that updated versions of just those transport documents will be driven when the YANG models complete.

Look at the responses below I see that we are in one of these situation where things are deliberately left open. I will simply assume that this is done in a way that supports those future definitions and clarifications. So lets close this issue without any actions.

Questions that arose for me related to Configured Susbription Transport where
the following: 1. Can Transport Session be initiated in both direction. RFC
8071 provides a solution for Publisher to Receiver initiation. It is unclear if all
parts are in place to have a receiver to publisher initiated transport to be bound
to the subscription.

This will be up to a specific transport draft to make this determination.

To see what might be viable for NETCONF, check out the earlier version of the document at:

This was seen as a complete version of such a solution.  However the WG wanted a YANG model for session parameters discussed in Section 5.2.

2. What is "name" really? It appears to be defined as an
empty container. Despite that it appears to have requirements on being both a
security identity as well as network address.

You are correct.  In previous versions of this draft, a receiver was identified by the combination of address + port.  However due to the YANG doctor reviews, and call home discussions referenced above, the WG wasn't ready to finalize this YANG structure.  The compromise was the current structure, plus the example YANG model of Appendix A to show how this might be built out.

3. In Figure 9, which is stated to be
for the receiver. What information does the receiver use to determine the
transition (d)? And what does it do in this step. Related to Discuss part B).

This determination is implementation specific.

4. RFC
8071 appears to lack any considerations for how often and how many times it
attempts to connect to the receiver. So applying that mechanism appears to
require some usage guidance to avoid causing overload situations or DoS
potential by misconfiguring network devices with this soltution to dial out
frequently to a target.

As the transport solution requirements are not detail it is actually hard to
determine if there are short comings in the description in Section 2.5 and the
corresponding YANG model. Is it an reasonable request to document these

The requirements were documented for both NETCONF and RESTCONF.   However these requirements were removed when the WG decided to wait until there was a YANG model for RFC-8071 ready to go to the IESG.   For a preview on what these requirements might look like, I refer you  to Section 5.2 of

Thanks, so this is clearly something to have in mind when the WG do get to specify a solution for this. So I don't see a need for any actions here.


1. Section 2.3: DSCP domains
I think the text could benefit from pointing out that the subscriber actually needs
to know what the DSCP values represents in the diffserv domain of the publisher.
As these could be different, they also create an interesting problems for
transports of how they establish a transport connection that uses a particular
DSCP, as the receiver if initiating need to know the local DSCP value that
corresponds to the behavior in the publisher's domain.

This makes sense.

I have added in Section 5.3 Transport Requirements a new entry which states:

"A subscriber which includes a "dscp" leaf within an "establish-subscription" request will need to understand and consider what the corresponding DSCP value represents within the domain of the publisher."

Let me know if this is not sufficient.

2. In general I think there are more than one description that are fuzzy about if it
describes a publisher or receiver action/requirement.

It would be great if you have some specifics.   The authors and previous reviewers likely have looked at this often enough where things look obvious which perhaps are not.

Sorry, didn't take note about those confusion.


Magnus Westerlund

Network Architecture & Protocols, Ericsson Research
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto:<>