Re: [tsvwg] Fwd: New Version Notification for draft-white-tsvwg-lld-00.txt

[Greg White <[e-mail]>]

Hi Luca,

First off, thank you for your review and probing questions.  I very much appreciate it.

Now on the points where you still have concerns, either I am not understanding them, or you aren’t understanding my answers.  I’ve added some responses inline below.  Hopefully we can get to the point that we agree.

-Greg



From: Luca Muscariello <[e-mail]>
Date: Tuesday, March 19, 2019 at 4:13 PM
To: Greg White <[e-mail]>
Cc: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[e-mail]>, tsvwg IETF list <[e-mail]>
Subject: Re: [tsvwg] Fwd: New Version Notification for draft-white-tsvwg-lld-00.txt

Greg,

You may be right. We may be doing different assumptions.
To me the point where we disagree is that you assume that an application,
to get access to the low latency queue, is not only well behaving
but also sending at a very low rate compared to the link capacity.

[GW]  No, that is not the assumption. The fundamental requirement is that, in order to have 100% of its packets remain in the low latency queue, a flow cannot be contributing appreciably to queuing latency.  Note, we are actually not describing the performance of the low latency queue as being a DetNet guarantee, so the requirement for 100% of packets to have deterministic latency does not exist.  Rather, the focus is on 99th percentile latency.  Nonetheless, in the abstract, a single application flow could send smooth (CBR) non-responsive traffic that fully utilizes the link capacity, and have all of that traffic utilize either queue (low latency or classic, depending on if it marked its packets NQB or not).

The point I'm trying to make is that in general, we do not know that rate.
As long as we have no predictable information about the amount of non classified
traffic that can be admitted to the low latency queue, the coupling factor, that
determines how time is shared between the two queues, cannot be set.
Any value would break the stable equilibrium assumed by the system and therefore
that much traffic has to be kicked out of the queue.

[GW] In practice, as you point out, an application generally cannot know the link rate (nor can it know whether it is the only application).  So, in practice, an application that is interested in low latency has 2 options in order to ensure that it isn’t causing queuing delay.  One option is that it simply does not send traffic that is likely to exceed the available capacity of the link. There are a lot of applications that do just this, and are great candidates for being marked NQB.  (Even without dual-queue, this is a practical constraint for any non-responsive application that cares about latency or loss performance, the only additional factor in the low latency queue is that the constraint applies on a more instantaneous basis than for a deep buffered link.) Given that the application cannot “know” the available capacity, all non-responsive applications take on some risk that they have guessed wrong, and have to accept the consequences (latency or loss) in the case that they have.  Of course, the lower the sending rate, the greater the likelihood that it won’t exceed the link capacity. The other option is to implement a low latency scalable congestion control (e.g. TCP Prague).  TCP Prague can fully utilize its per-flow-fair share of the link capacity across a very wide range of cases without the need to tune any parameters in the dual-queue configuration.  Outside that range of cases, the behavior is not catastrophic.   I’m not certain what you mean by “non-classified traffic” do you mean non-responsive (or non-congestion-controlled)? Also you used the term “coupling factor” which is a specific term used in dual-queue coupled AQM to refer to the multiplicative factor that relates the L4S ECN marking probability to the Classic AQM drop probability.  I don’t think that is what you were referring to.  I think you intended to mean the WRR weight.  In case I am right on both counts, I thought I had explained this in my earlier email, but let me make it clear.  Stability is not in question. What you are arguing is that per-flow-fairness isn’t guaranteed in certain situations.   This is true, but as I pointed out, the range of conditions where fairness is more-or-less achieved is pretty large, and outside that range things aren’t so bad.  I say “more-or-less achieved” because, with the dual-queue, just like a single queue, flows will only tend toward fairness (with the caveats of RTT differences, variations in congestion controllers, etc.).   But, a goal of TCP Prague is to at least reduce this RTT dependence.

This is not a positive incentive but rather policy enforcement.
This is in itself a subject of discussion.

[GW] This seems to be a matter of opinion, but I’m actually not sure why you feel the need to make the distinction.  The low latency queue is only for flows that aren’t causing queuing delay.  If a flow is causing queuing delay then the system (and in some cases the application itself) is better off if the application’s packets take the classic queue.  The job of Queue Protection is to enforce this, and ideally create incentives for “correctly” marking packets.  Furthermore, the applications that are the best behaved are the most likely to be able to send all of their packets via the low latency queue, so this is an incentive. Keep in mind, that this isn’t DetNet.

Another concern I see is more about DoS to the low latency queue and
then to both queues.

It seems a priori definitely possible to create a limit cycle to the control theoretic
equilibrium by injecting background non classified traffic that generates little queuing.
Such packet process could then disturb all other sources or worse starve them enough
to force user to abandon.
In this case even a FIFO queue would be better (providing more predictable performance)
than two coupled queues relying on
a closed loop equilibrium which assumes only compliant traffic is present.

[GW] I think that I have already dispelled this in an earlier email, but let me know if you would like me to try again.

Now, being aware of how difficult it is to compute stability regions of a closed loop system
with delays, which is the one we are discussing here, my overall impression is that
this queueing system is fragile and can provide unpredictable performance.

[GW] Then you would undoubtedly have the overall impression that the current internet is fragile and can provide unpredictable performance (which, I suppose, is true).  In my opinion, dual-queue reduces the unpredictability, and does not increase fragility.  Analyzing stability is even more difficult given that the internet is not a linear time-invariant system.  As a result, I don’t personally give a lot of credence to that sort of analysis for congestion control systems.  Nonetheless, LTI approximations have been used to give confidence in the stability of PIE (at least for Reno CC), and in the cases that you seem to be concerned about, the PIE controller is in use.  Simulations have confirmed stability for DCTCP in these mixed traffic situations as well.

Assuming that you're pretty confident about the cable use case you have in mind,
with all the assumptions you are making about today applications' rates, would you also
be confident about future applications or about other network access technologies?

[GW] Again, we’re not making any assumptions about today’s application rates.  It is the applications that need to make assumptions based on all of the environments that they expect to be deployed in (and this is 100% not new).  If an application wants low latency and low loss, then it needs to fit in the available capacity along the path, whatever that path may be.  If it doesn’t want to use congestion control for some reason, then it needs to consider how it will send its traffic to ensure that it doesn’t overload the bottleneck link.  That is fundamental (and not specific to dual queue).

Luca

On Tue, Mar 19, 2019 at 10:12 PM Greg White <[e-mail]<mailto:[e-mail]>> wrote:
I think that is a mischaracterization.

I don’t believe that (at least on cable broadband links) non-responsive, link-saturating traffic is “currently a big chunk of Internet traffic that requires low latency”.   Perhaps on very low speed links this may happen, but what are the applications that are streaming at (e.g.) 10 Mbps without any congestion control?

The current Internet places expectations on data sources that they implement congestion control, and that they respond in a fairly consistent way to congestion signals (usually packet drops).  Not all of them do, but the majority do, and it has not resulted in a tragedy of the commons.  The dual queue mechanism continues that expectation.

I also don’t agree that the dual queue system lacks incentives.  For an application to get the benefit of sub millisecond queuing delay, it needs to ensure that it is not the cause of the queuing delay.  It can do that in a couple of ways.  One is that it sends at a low and smooth enough rate that it is highly unlikely to cause queuing, the other is that it can respond to fast congestion signals (new ECN marks).  The applications that do this can enjoy sub millisecond queuing delay.  If that isn’t an incentive, that’s fine. Some NQB traffic may not care about latency performance, and so would be equally happy in the classic queue.

-Greg




From: Luca Muscariello <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, March 19, 2019 at 2:53 AM
To: Greg White <[e-mail]>
Cc: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[e-mail]<mailto:[e-mail]>>, tsvwg IETF list <[e-mail]<mailto:[e-mail]>>
Subject: Re: [tsvwg] Fwd: New Version Notification for draft-white-tsvwg-lld-00.txt

Got it, thanks.

Consider that one concern that I have is on what you Greg consider a pathological case.
That case is currently a big chunk of Internet traffic that requires low latency.

The current scheduler makes substantial assumptions about the behaviour and misbehaviour of data sources.
The thing the I find uncomfortable is that this proposal assumes that every source conforms to a given profile,
including explicit marking.

The current queueing system instead of creating incentives to well behave creates a mechanism that punishes
sources that do not conform. Even  sources that well behave can be punished just because they do not conform.

My understanding is that this might be difficult to accept for many applications and, most important, people.

Luca






On Tue, Mar 19, 2019 at 4:31 AM Greg White <[e-mail]<mailto:[e-mail]>> wrote:
Hi Luca, see below [GW].

From: Luca Muscariello <[e-mail]<mailto:[e-mail]>>

On Mon, Mar 18, 2019 at 1:21 PM Tilmans, Olivier (Nokia - BE/Antwerp) <[e-mail]<mailto:[e-mail]>> wrote:
Hi Luca,

Just to make sure, did I answer your question?

maybe. Let me ask a few more clarification questions below.


If not the behavior of putting NQB flows in the L4S queue is dependent on the
AQM used, hence the answer would be to look at
draft-ietf-tsvwg-aqm-dualq-coupled for a discussion of the effects of doing so.
DOCSIS adds on top of the AQM a protection scheme to sanction flows in the
L4S that cause queue build up, and uses a WRR to control how packets get
dequeued.

The final answer would then be: yes, you can put non-ECN controlled flows in
the L4S queue provided those are sparse enough. Otherwise, they would need
to be wrapped in a scalable CC and/or interpret the received ECN feedback (as
 the 2015 demo showing live video streaming zooming/zoomout did).

This is where things become unclear or just unspecified in the documents I read so far.
Sometimes maths or code is simpler to read that a thousand words.

My interpretation of what you say is that the time sharing between the two queues
depends on a closed loop mechanism that relies on classification first and a compliant sender that
is sensitive to ECN according  to the mechanism described in draft-ietf-tsvwg-l4s-arch.

Assuming that the control theoretic equilibrium you get requires a given WRR weight,
anytime non conformant traffic is allowed into the low latency queue this equilibrium can be
perturbed. Which is fine as long as the perturbation is small enough.

[GW] When both queues are in use by responsive traffic, the classic queue AQM drives the congestion signal for both queues, and gives them both a congestion signal that the two sender types interpret in a consistent way, leading toward per-flow fairness.  Just as with a single queue AQM, the consistent (flow unaware) application of drop probability (and presumed consistent interpretation of those congestion signals by senders) leads toward per-flow fairness.  The equilibrium is not strongly related to the WRR weight.  As long as the weight is high enough, the relationship between the congestion signals (drop in the case of the classic queue and ECN mark in the case of the low latency queue) drives the equilibrium.  Just as an example of a simple case, if all of the flows happened to be long-running responsive flows in congestion avoidance, then the congestion signals will drive the equilibrium as long as the WRR weight given to the low latency queue is greater than or equal to the share of the flows that are occupying that queue.  In the DOCSIS spec the default value for the weight is 230/256 (about 90%, see section C.2.2.7.17.6 in the DOCSIS spec), so as long as less than 90% of the simultaneous flows are occupying the low latency queue (for example if there are four L4S flows and one CUBIC flow), per-flow fairness will be maintained.   If greater than 90% of the simultaneous flows are in the low latency queue (e.g. twenty-four L4S flows vs. two CUBIC flows), the ~10%  weight given to the classic queue will result in the classic flows each getting somewhat more than their share of bandwidth (e.g. they will each get 5% of the capacity, while the L4S flows will each get 3.75%).  If we could assume that ALL flows are well-behaved responsive flows, then strict priority could be used, and per-flow fairness would be maintained regardless of the mix.  Since we can’t make that assumption, the non-100% weight protects some bandwidth in the Classic queue.

Non controlled traffic can generate a busy period in the low latency queue that can grow
w/o necessarily generating large standing queues. That should not trigger the queue protection
mechanism Greg mentioned.
From what I understand WRR would  then make the equilibrium drift to another value.
This value should be that the classic queue gets lower service time compared to what
the original configuration was meant to provide.

[GW] Non-responsive traffic in the low latency queue will consume some of that queue’s WRR weight (similarly, non-responsive traffic in the classic queue will consume some of that queue’s weight), with the result being that for the remaining congestion controlled traffic the range of flow mixes over which per-flow fairness is achieved is shifted.  As an example, if non-responsive traffic in the low latency queue consumed 50% of the total capacity, and the rest of the traffic was congestion controlled, the system would behave as if (for the remaining 50% of capacity available to the congestion controlled flows) the WRR weight was 80% (90% minus 50%, divided by 50%).   So per-flow fairness would only be achieved when the low latency queue is carrying less than or equal to 80% of the flows, otherwise the classic flows will get somewhat more than their share.    As another example, if non-responsive traffic in the classic queue were consuming 50% of the total capacity, and the rest of the traffic was congestion controlled, the system would provide per-flow fairness (with the remaining 50% of the capacity) across ALL mixes of L4S/Classic.

[GW] Yes, there are pathological cases where high-bandwidth non-responsive flows are consuming the entire channel capacity.  In those cases, the WRR weight will result in the flows in the low latency queue getting more aggregate bandwidth than the ones in the classic queue.


Do I get this right?

Thanks
Luca



Best,
Olivier


> -----Original Message-----
> From: Luca Muscariello <[e-mail]<mailto:[e-mail]>>
> Sent: Monday, March 18, 2019 12:21 PM
> To: Tilmans, Olivier (Nokia - BE/Antwerp) <olivier.tilmans@nokia-bell-
> labs.com<http://labs.com>>
> Cc: Greg White <[e-mail]<mailto:[e-mail]>>; tsvwg IETF list <[e-mail]<mailto:[e-mail]>>
> Subject: Re: [tsvwg] Fwd: New Version Notification for draft-white-tsvwg-lld-
> 00.txt
>
>
>
> On Mon, Mar 18, 2019 at 9:56 AM Tilmans, Olivier (Nokia - BE/Antwerp)
> <[e-mail]<mailto:[e-mail]> <mailto:olivier.tilmans@nokia-bell-<mailto:olivier.tilmans@nokia-bell->
> labs.com<http://labs.com>> > wrote:
>
>
>       > Is there an assumption that the low latency queue is only used by
> rate
>       > controlled flows
>       > that are sensitive to ECN feedback?
>
>       The L4S definitions in draft-ietf-tsvwg-l4s-arch state that L4S flow
> have a
>       scalable CC, in order to perform bandwidth pooling (fairly) between
> the
>       L4S flows and the classic ones.
>       This can however be relaxed a bit, see below.
>
>       > I would assume that a low latency queue is open to flows that are
> non
>       > necessarily controlled by ECN but just exogenously rate limited by
> say a
>       > codec rate.
>
>       This is what draft draft-white-tsvwg-nqb discusses, namely also
> adding
>       non-queue building flows in the L4S queue (e.g., non-capacity
> seeking
>       flows such as gaming, DNS traffic, pings, sparse CBR flows).
>
>       Note that the presence of such unresponsive flows in the L4S queue
>       implicitly decreases the throughput of the scalable L4S flows (but not
>       their classic counterpart), as they have to back off on behalf of the
>       non-responsive flows (but can do so without fiddling with a magic
> WRR
>       number). This can be accounted for in the coupling factor of the
> queue.
>
>
>
> Right, this is what I was trying to understand.
> I checked draft-white-tsvwg-nqb but the mechanism to share traffic in the
> two queues
> is not described. This document refers back to draft-ietf-tsvwg-l4s-arch so I
> cycled back to
> my starting point as there is no description of how the queueing system
> works when
> non marked ECNed traffic is sent  to the low latency queue.
>
> I thought that that was described in the DOCSIS spec Greg shared, but did
> not find an answer there either.
>
>
>
>
>
>
>
>       Best,
>       Olivier
>
>
>       > But now it seems like this is not the case.
>       >
>       > Thanks
>       > Luca
>       >
>       > On Sun 17 Mar 2019 at 18:32, Tilmans, Olivier (Nokia - BE/Antwerp)
>       > <[e-mail]<mailto:[e-mail]>
> <mailto:[e-mail]<mailto:[e-mail]>>  <mailto:olivier.tilmans@nokia-<mailto:olivier.tilmans@nokia->
> bell- <mailto:olivier.tilmans@nokia-bell-<mailto:olivier.tilmans@nokia-bell->>
>       > labs.com<http://labs.com> <http://labs.com> > > wrote:
>       >
>       >
>       >       Hi Luca,
>       >
>       >       > The protection mechanism assumes that one queue has soft
>       > priority over
>       >       > the other. Strict priority would be stupid, so there must be a
> wfq
>       > weight to
>       >       > schedule the classic queue less frequently. I did not find the
> magic
>       > number
>       >       > that  is set in the DOCSIS specs but whatever number is chosen
> I
>       > wonder
>       >       > which opitimization objective would be the foundation of that.
>       >       > 10%, 20%, 30% or any number would imply that if the priority
>       > queue is used
>       >       > at 100% utilization the other apps would get a small fraction of
> the
>       > link
>       >       > capacity.
>       >       >
>       >       > What number is chosen and based on which calculations?
>       >
>       >       The protection mechanism only affect how competing packets
> within
>       > a RTT gets
>       >       dequeued. In this instance, a WRR protects the classic flows
> from
>       >       unresponsive/misclassified L4S flows, at the expense of the
> other
>       > L4S flows.
>       >
>       >       It does not affect the overall throughput of the flows when
>       > measuring it over a
>       >       timeframe of a few Tupdate interval (e.g., 16ms). Indeed, the
> core of
>       > the
>       >       DualPI2 AQM is a PIE2 controller computing the random
>       > drop/marking probability
>       >       based on the classic queue. This probability is then also used as
> base
>       > to mark L4S
>       >       flows, albeit with a quadratic relationship (i.e., for the same base
>       > probability, the
>       >       L4S flows are marked much more often than the classic ones).
>       >
>       >       Consequently, whenever L4S traffic causes the classic queue to
> build
>       > up (even
>       >       slightly above the PIE2 target), its marking probability drastically
>       > increases as the
>       >       PIE2 controller reacts. As the L4S flows have scalable CCs (i.e.,
>       > understand how to
>       >       leverage multiple CE marks per RTT), they reduce their cwnd
>       > accordingly within
>       >        a RTT (which is much smaller than the classic flows' RTT).
>       >
>       >       In other words, L4S senders end up slowing down themselves
> (to a
>       > point where
>       >       the L4S queue is empty for most of the transmission slots) to
> allow
>       > the classic
>       >       queue to drain, and this reaction is the result of a much faster
>       > control-loop than
>       >       the one used by the classic sender.
>       >
>       >       There are thus no magic number per say for the WRR ratio.
> §4.1.4 of
>       >        draft-ietf-tsvwg-aqm-dualq-coupled explains the guidelines to
> pick
>       > one.
>       >
>       >       Note that the exact throughput ratio between L4S and classic
> flows
>       > can be
>       >       tweaked using a coupling factor, as explained in appendix C. of
>       >        draft-ietf-tsvwg-aqm-dualq-coupled.
>       >
>       >       I hope this was clear enough.
>       >
>       >
>       >       Best,
>       >       Olivier
>       >
>
>