Re: [tsvwg] Comments on draft-white-tsvwg-nqb-02

[Greg White <[e-mail]>]

Roland and Kyle,

Thanks for the review and comments, and apologies for the delay in responding.  Yes, the terms “non-queue-building” (NQB) and “queue-building” (QB) are not mathematically precise, but I don’t think they need to be.   This solution was developed with a particular environment and use cases in mind, but we believe that it solves a wider problem, and so should be described in more general terms.  Your questions are good ones to help us articulate what those terms should be.

The concept of enabling the dual-queue at a bottleneck link is to be able to provide significantly better quality of experience for latency sensitive applications without causing degradation of other applications, and without violating principles of network neutrality.  The bottleneck provides two queues, one that is essentially optimized for traditional Reno/Cubic congestion control (i.e. it has a buffer size that is on the order of the expected worst-case BDP, and it perhaps implements a traditional AQM like PIE, ARED, CoDel, etc.), and the other that is optimized for applications that are relatively isochronous or sparse (i.e. it has a much smaller buffer, and does not implement a packet-drop-based AQM).  The precise characteristics of these two queues will likely be dependent on the characteristics of the link that they serve.  In general, the NQB queue isn’t going to provide deterministic latency.  The NQB buffer isn’t a single packet deep, in some cases it may be on the order of 5-10 ms.  So, with a mix of uncoordinated senders, the latency will vary a bit.

In the case of a DOCSIS link, the NQB queue is sized at 10 ms, but it includes Queue Protection, which will aim to prevent the queue from getting to the tail drop limit by shedding load.

The use case that motivated the NQB proposal was one in which L4S ECN marking is provided in the NQB queue as well, so compatibility between NQB traffic and ECT(1) traffic is an important characteristic that we wish to promote.  In other words, capacity-seeking flows that wish to utilize the low latency queue would need to implement L4S-style congestion response (and signal ECT(1) rather than NQB in the TOS byte).  So, I think it is correct to state that capacity-seeking flows (regardless of congestion control algorithm) are not recommended to be tagged as NQB.

In the case of BBR, Neal Cardwell has indicated that BBRv2 will support an L4S-style congestion response, and so would presumably be marked ECT(1) (not NQB).

For the scheduler, an equal weighted round robin would be an example (recommended?) approach in the absence of dualq-coupled-aqm.  When dualq-coupled-aqm is present (and thus the low latency queue is being shared by both NQB and ECT(1) traffic), a weighted round robin as described in the dualq-coupled-aqm draft is appropriate.    This is one reason to say that capacity-seeking flows should not be tagged as NQB.

For high data rate (e.g.) CBR flows that are not capacity-seeking, I do agree that they could technically be candidates to mark as NQB, but I think there is some risk in doing so.  As the draft discusses, some links (e.g. WiFi) have limitations in that they by default only support priority queues, so the only way to get isolation between QB and NQB on these links entails also getting different priority.  If we limit the definition of NQB to traffic that is relatively low data rate (i.e. consistent with the definitions of the existing DSCPs that map to AC_VI in default WiFi equipment), the risk of mapping NQB to the AC_VI queue (for purposes of isolation) is reduced.  WiFi links are an important piece of the solution, so we can’t sweep this aside.

-Greg



From: Kyle Rose <[e-mail]>
Date: Monday, September 16, 2019 at 12:06 PM
To: "Bless, Roland (TM)" <[e-mail]>
Cc: "[e-mail]" <[e-mail]>, Greg White <[e-mail]>
Subject: Re: [tsvwg] Comments on draft-white-tsvwg-nqb-02

On Fri, Sep 13, 2019 at 11:49 AM Bless, Roland (TM) <[e-mail]<mailto:[e-mail]>> wrote:
· I find the whole term "non-queue building flows" a bit confusing.
What the draft actually means are application-limited, mostly low data rate
flows (sparse flows in RFC8290). According to the draft, capacity seeking
flows seem to always be queue building.
I have had the same concern about the terminology. I'm wondering if "NQB" is a term of art that isn't intended to be interpreted literally. As you point out below, a tiny additional flow to a saturated link can cause a queue to start building, which makes me question whether "queue building" is really the right description at all. It seems the draft is trying to classify flows merely according to sender behavior when it's not clear how useful that will be as a tool in preventing queue buildup in bottlenecks of varying size and with varying traffic profiles.

Comparing BBR with Reno, the latter will tend toward full queues while the former will oscillate around a small queue, and yet both are capacity-seeking. The draft classifies BBR flows as QB. Should they be? If so, what actually constitutes queue-building? BBR builds a small queue to probe for capacity but backs off quickly in response to increased latency. Does minuscule transient queue-building like this (with the intent to keep queues small) really qualify as QB? Why should they be stuck in the same queue with Reno flows, when from a latency and queue-building standpoint they are more like the flows described in the draft as NQB (and in some cases better, since they'll actually be able to back off, unlike fixed-bandwidth flows).

· It's not clear to me how the service rates between the NQB and QB queue
are distributed by the scheduler. From the description in the draft, I have
no clue what kind of scheduler would be a good fit for the desired behavior.
...
1.      "NQB traffic is not rate limited or rate policed." what happens if the load in the
NQB queue increases (e.g., not by queue building flows)? Will QB flows be starved?
I think that some scheduler will determine the rate share between NQB/QB queues...
...
1.      "it places unnecessary restrictions on the scheduling between the two queues"
so what is the requirement for your own approach then?
Good questions related to some of my main concerns related to the L4S, SCE, and FQ debate.

When a link is close to saturated, FQ policy allocates an equal share of available capacity to all non-sparse flows/buckets, preventing a subset of flows from starving the rest via aggressive capacity-seeking behavior. This clearly isn't ideal in every circumstance, but what's an application-oblivious alternative that is? I want someone to explain how a bottleneck is supposed to divide capacity on a saturated link between (say) file downloads and videoconferencing without knowing what either of those flows contain. An oracle might give priority to video and treat the file download as best-effort, but that's a policy choice as much as one in which the user is willing to sacrifice video quality by forcing the application to downgrade encoding bitrate so their download completes more quickly.

Whether we like it or not, there is no knob-free QoS even with huge pipes. This draft must attempt to identify and address relevant policy considerations at bottlenecks.

Kyle




From: "Bless, Roland (TM)" <[e-mail]>
Organization: Institute of Telematics, Karlsruhe Institute of Technology
Date: Friday, September 13, 2019 at 9:49 AM
To: "[e-mail]" <[e-mail]>, Greg White <[e-mail]>
Subject: Comments on draft-white-tsvwg-nqb-02

Hi,

I'm quite late to the discussion and eventually took time to have quick read
over the draft, but I think it is rather vague at times and there is quite some
work required to either get rid of implicit assumptions or to write them down
explicitly.

First, some general comments:
·         I find the whole term "non-queue building flows" a bit confusing.
What the draft actually means are application-limited, mostly low data rate
flows (sparse flows in RFC8290). According to the draft, capacity seeking
flows seem to always be queue building.
This is not true, because it depends on the particular congestion control
and how the capacity seeking is done. For instance, TCP LoLa [1] flows are
not queue building, but are capacity seeking nevertheless, i.e., they will
saturate the bottleneck link. Strictly speaking, they are building a _limited_
standing queue.
·         The notion "queue building" is also somewhat fuzzy: do you mean flows that build
increasing and unlimited standing queues (i.e., until buffer is completely filled) or
also flows that are able to build limited standing queues? LoLa and BBR would fall
into the latter class, where BBR's standing queue depends on "the BDP".
·         Moreover, even sparse flows can build a queue: assume we have a sparse flow
arriving at a link that is already saturated by a single capacity seeking flow.
So even if the sparse flow is only sending 0.1% of the link capacity,
it will _cause_ a queue to build up (under the assumption that the other flow
does not back off immediately). The only difference is that the large flow possesses
at lot more data in the queue compared to the sparse flow. Similarly, if
there are a lot of sparse flows saturating a bottleneck, they will also create a growing
queue (until they reduce their sending rate).
·         It's not clear to me how the service rates between the NQB and QB queue
are distributed by the scheduler. From the description in the draft, I have
no clue what kind of scheduler would be a good fit for the desired behavior.
·         Would it be possible to let an NQB flow (or several) saturate the link? I think it would be a perfectly valid scenario to let high-data rate, low delay flows use the NQB queue.
"NQB traffic is not rate limited or rate policed." suggests that.
·         Can a low-latency high data rate flow be in the NQB queue, e.g., a TCP LoLa
flow? An NQB PHB would be nice for them, because they are neither suppressed
nor delay-wise adversely affected by the (loss-based) queue filling flows.
·         I also second Dave Black's comments on looking at the PHB definition
requirements. A PHB definition should define the behavioral characteristics
and more details, see
https://tools.ietf.org/html/rfc2474#section-5 and https://tools.ietf.org/html/rfc2475#section-3
While the PHB definition should be more abstract, I find it useful to provide at least
examples of the mechanisms that can be used for an implementation.

Some more text related comments:
1.          "Active Queue Management (AQM) mechanisms (such as PIE [RFC8033],
   DOCSIS-PIE [RFC8034], or CoDel [RFC8289]) can improve the quality of
   experience for latency sensitive applications, but there are
   practical limits to the amount of improvement that can be achieved
   without impacting the throughput of capacity-seeking applications."
I'm not sure what your are referring to here. My experience tells me that
this is also highly related to the congestion control variant. Let's assume
Cubic TCP for now: If only a few flows are present at the AQM bottleneck, then the
utilization/throughput may be adversely affected, but if already a small number of flows is
present, a good AQM achieves good utilization due to its desynchronization feature.
2.       "These applications do not make use of
   network buffers, but nonetheless can be subjected to packet delay and
   delay variation as a result of sharing a network buffer with those
   that do make use of them."
These application are actually using the buffer too, but their proportion
w.r.t. queued data is probably small compared to "queue building" flows.
3.       "Here the data rate is essentially limited by the Application itself."
That would be good to mention earlier. Is this a requirement for NQB flows?
As I wrote above: there are also elastic NQB flows possible, using all available
capacity. Furthermore, a lot of app-limited NQB flows could also saturate
a link, do you exclude this case?
4.       "but there are also application flows that may be in a gray area in between
(e.g. they are NQB on higher-speed links, but QB on lower-speed links)."
Is this the case I just described: enough sparse flows may saturate lower-speed
links already? or do you think of something else?
5.       "As an answer the last question" => As an answer _to_ the last question
6.       "Thus, a useful property of nodes that support separate queues for NQB and QB
   flows would be..."
It seems that supporting separate queues is a requirement for the PHB. If so,
please state that.
7.       " and for QB flows, the QB queue provides better performance
   (considering latency, loss and throughput) than the NQB queue."
I don't see how they achieve better latency, except in the case that
they experience less packet loss and thus suffer from less retransmissions,
but this may be quite situation dependent...
8.       Should starvation of QB flows be impossible?
9.       "and reclassify those flows/packets to the QB queue."
This should probably be explicit about flow classification, because otherwise
per-microflow reordering may also adversely affect the e2e performance. So
probably there is a requirement to move all packets belonging to a flow
consistently to the QB queue.
10.   "This queue SHOULD support a latency-based queue protection mechanism"
as others already pointed out: this seems to be an important requirement and
therefore, you should either state how that is supposed to work or the corresponding
requirements.
11.   "NQB traffic is not rate limited or rate policed." what happens if the load in the
NQB queue increases (e.g., not by queue building flows)? Will QB flows be starved?
I think that some scheduler will determine the rate share between NQB/QB queues...
12.   "The node supporting the NQB PHB makes no guarantees on latency or
data rate for NQB marked flows"
The EF PHB is also not giving actual guarantees other than "low latency, low jitter,
low loss"...
13.   "The choice of
   the 0x2A codepoint (0b101010) for NQB would conveniently allow a node
   to select these two codepoints using a single mask pattern of
   0b101x10."
Diffserv codepoints are unstructured and should NOT be handled like this.
RFC 2474 requires anyway that the "mapping of codepoints to PHBs
MUST be configurable"
14.   Section 6: a discussion is missing if intermediate domains/nodes do not
support the proposed NQB PHB: the low latency property may get lost
completely if it gets treated as default PHB etc. RFC 2474 states
"Packets received with an unrecognized codepoint SHOULD be forwarded
   as if they were marked for the Default behavior (see Sec. 4), and
   their codepoints should not be changed."
15.   "cable broadband services MUST be configured
   to provide a separate queue for NQB traffic that shares the service
   rate shaping configuration with the queue for QB traffic."
I have no clue what that means...
16.   Section 9 contains some "negative examples" or why existing solutions
are not solving the problem. I think there are probably some hidden/implicit
requirements that should be rather extracted from this section and put
into the definition of this PHB. For example you seem to do not want
"that each non-sparse flow gets an equal fraction of link bandwidth"...
or that there is a difference between the "sparse flow" definition of
FQ and your definition of NQB flows (which are probably sparse, but
app-limited or latency-sensitive at least?). What about latency-sensitive
high data rate flows?
17.   "it places unnecessary restrictions on the scheduling between the two queues"
so what is the requirement for your own approach then?

That's all for now, regards

 Roland

[1] Mario Hock, Felix Neumeister, Martina Zitterbart, Roland Bless: TCP
LoLa: Congestion Control for Low Latencies and High Throughput,
IEEE LCN 2017, http://ieeexplore.ieee.org/document/8109356/