Re: [tsvwg] L4S status: #17 Interaction w/ FQ AQMs

[Greg White <[e-mail]>]

Comments prefaced with "[GREG]" below.


On 8/12/19, 7:30 PM, "Jonathan Morton" <[e-mail]> wrote:

    > On 13 Aug, 2019, at 3:00 am, Greg White <[e-mail]> wrote:
[snip]
    > [Sebastian]     But let's assume I have two flows one L4S one RFC 3168, and my link is already at saturation with each flow getting 50% of the shaper's configured bandwidth…
    
    Okay, this scenario covers a steady-state case.  I'm more concerned with transient behaviour, specifically ensuring that short-term changes in the traffic conditions result in only short-term effects on other traffic.  But there are a couple of fallacies in the following wall of text that I was able to spot, and which need squashing.
    
    > I don't see how the RFC3168 sender would be triggered to reduce its cwnd below 50% of BDP.  (by an FQ-AQM, competing with one other bulk flow)
    
    This is trivially demonstrable as soon as you have an RTT long enough for a given sampling tool to observe the effect reliably, and in fact you can see it in the SCE slide deck (in which there is one chart showing FQ-AQM behaviour).
    
    RFC-3168 senders respond to single CE marks with a Multiplicative Decrease, and FQ-AQMs will provide these marks very shortly after the fair share is reached.  Assuming the standard 50% decrease is employed with NewReno, that means it will be sent down to about 25% link capacity, or more precisely, half the fair share - from where it must linearly grow back again.  CUBIC reduces to 70%, which means it is sent down to about 35% briefly, and recovers more quickly with its polynomial response.
    

[GREG] I don't believe that what I described was a fallacy.   To be clear, when an RFC-3168 sender responds to a CE mark with a multiplicative decrease, what it is multiplicatively decreasing is NOT its sending rate.  It is decreasing its cwnd.  When cwnd is greater than BDP, sending rate is no longer proportional to cwnd.  In an idealized case, the bottleneck link will trigger cwnd reductions such that the reduced cwnd is exactly equal to the BDP (and queuing delay is zero), which means that the cwnd reduction is triggered when cwnd = BDP/0.5 (for Reno) or cwnd = BDP/0.7 (for Cubic).  This preserves continuous 100% link utilization with the minimum queuing delay, and cwnd is always >= BDP.   The CA sender behavior when cwnd is >= BDP is (via ack clocking or pacing) that the sender sends traffic *just slightly* greater than the link bandwidth (B) causing the queue to grow.  In the case of Reno the sender rate is B + (1 MTU / RTT) (where RTT = base RTT + queue delay), and the queue grows by one MTU each RTT.   The situation where the behavior you described would happen is if there is essentially zero buffering, and so the cwnd reduction happens when cwnd = BDP.  This would reduce the cwnd to 0.5*BDP (Reno) or 0.7*BDP (Cubic) which (due to zero queuing delay) would actually equate to a sending rate of 0.5*B or 0.7*B.  If the AQM in the systems you are using causes this behavior, then I would argue that it is not CoDel.  The design of CoDel was such that the *minimum* (not maximum) sojourn time over the last window (100ms) is driven to 5ms or below (not zero).  It does happen that CoDel occasionally drops/marks excessively, such that the queue fully drains and the link is underutilized, but not to the extent you are describing (unless CoDel is implemented incorrectly or misconfigured).    The goal of AQMs has generally been to preserve (in as many scenarios as possible) near 100% link utilization, while minimizing queuing delay.   Is your AQM aiming at some other goal?



    > It seems as if your concern is that, when the RFC3168 flow cuts its cwnd to drain standing queue, that the L4S sender will shut out the RFC3168 sender on the access network link, and thus force the RFC3168 flow's FQ queue to drain completely.
    
    No, the *eventual* convergence to fair shares in steady state is not in question.
    
[GREG] Neither (in my view) is the rapid convergence to fair shares**.  But this apparently needs to be demonstrated.



    FQ enforces it to the best of its ability (which, crucially, is determined by the traffic which makes it through the preceding network elements),
    
[GREG]  FQ enforces it *precisely* as long as neither queue drains completely.



    and Codel (at least in COBALT form, which is what we tested) does eventually ramp up its marking rate to the point where a DCTCP response function is properly controlled.  But it takes four whole seconds to get there when a new L4S flow starts up, with default settings and the short inherent RTTs that the L4S team appears to favour.  Note that I'm not claiming that four-second queues exist, only that elevated queuing latency (peaking around 125ms) persists for four seconds.

    A lot can happen in those four seconds, from the user's perspective.  It's long enough to seriously disrupt online gameplay, causing the player to miss crucial shots because his timing is thrown off by an eighth of a second.  VoIP streams will be forced to switch on their jitter buffers and pause the received audio stream to catch up, and will most likely never recover from that condition to the more realtime behaviour they previously enjoyed.  And so on.
    
[GREG]  Ok, here I think we're on the same page.  The CoDel response is too sluggish to bring latency under control as quickly as we would like.  An L4S sender is aiming to see 2 marks per RTT, and the normal CoDel ramp up of marking rate (in marks/second) over time is approx. 7 + 50*t.  So, for a 10ms RTT, where an L4S sender is wanting to see 200 marks per second, CoDel would take about 3.8 seconds to ramp up to that level.  We will need to conduct the experiments to see what the latency looks like during that period, but if it is up to 125ms of queuing delay for 3.8 seconds, then that is not what an L4S sender would be wanting to see.  (minor quibble: I'd argue that VoIP probably isn't the best example of an L4S flow, but cloud gaming certainly is.)   



    Here is a simple experiment that should verify the existence and extent of the problem:
    
    Control:
    [Sender] -> [Baseline RTT 10ms] -> [FQ_Codel, 100% BW] -> [Receiver]
    
    Subject:
    [Sender] -> [Baseline RTT 10ms] -> [Dumb FIFO, 105% BW] -> [FQ_Codel, 100% BW] -> [Receiver]
    
    Background traffic is a sparse latency-measuring flow, essentially a surrogate for gaming or VoIP.  The instantaneous latency experienced by this flow over time is the primary measurement.
    
    The experiment is simply to start up one L4S flow in parallel with the sparse flow, and let it run in saturation for say 60 seconds.  Repeat with an RFC-3168 flow (NewReno, CUBIC, doesn't matter which) for a further experimental control.  Flent offers a convenient method of doing this.
    
    Correct behaviour would show a brief latency peak caused by the interaction of slow-start with the FIFO in the subject topology, or no peak at all for the control topology; you should see this for whichever RFC-3168 flow is chosen as the control.  Expected results with L4S in the subject topology, however, are a peak extending about 4 seconds before returning to baseline.
    
[GREG] I would not expect to see the FIFO holding a queue in either case, (i.e. I would expect this phenomenon to only affect the latency of the L4S flow) as long as BW > 24 Mbps (see ** below) but we will see.


    Please let me know how you get on with reproducing this result.
    
     - Jonathan Morton
    
    
[GREG] ** In "Sebastian's Topology" the difference between the Dumb FIFO BW and the FQ_Codel BW needs to be greater than or equal to one MTU / RTT (for Reno or DCTCP/Prague) to prevent queuing from occurring in the Dumb FIFO for a single steady state flow.  I suppose it needs to be an even bigger difference to prevent FIFO queuing with Cubic.   What is the rationale for using a multiplicative factor (105%) rather than an additive one scaled based on expected CC dynamics?