Re: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02

David Lang <> Mon, 13 April 2015 21:47 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id 9615E1A897D for <>; Mon, 13 Apr 2015 14:47:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.91
X-Spam-Status: No, score=-1.91 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01] autolearn=ham
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 0f4vN0DjBA5v for <>; Mon, 13 Apr 2015 14:47:11 -0700 (PDT)
Received: from ( []) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id CF8051A8938 for <>; Mon, 13 Apr 2015 14:47:10 -0700 (PDT)
Received: from ( []) by (8.13.4/8.13.4/Debian-3) with ESMTP id t3DLkgkw006770; Mon, 13 Apr 2015 14:46:42 -0700
Date: Mon, 13 Apr 2015 14:46:42 -0700 (PDT)
From: David Lang <>
To: Bob Briscoe <>
In-Reply-To: <>
Message-ID: <>
References: <> <> <> <> <> <> <> <> <20150327183659.GI39886@verdi> <> <> <> <> <> <> <>
User-Agent: Alpine 2.02 (DEB 1266 2009-07-14)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Archived-At: <>
Cc: "Scheffenegger, Richard" <>, Vishal Misra <>, KK <>, John Leslie <>, "" <>
Subject: Re: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 13 Apr 2015 21:47:14 -0000

On Mon, 13 Apr 2015, Bob Briscoe wrote:

> David,
> Returning from a fortnight offlist...
> I think your conception of how ECN works is incorrect. You describe ECN as if 
> the AQM marks one packet when it drops another packet. You say that the 
> ECN-mark speeds up the retransmission of the dropped packet. On the contrary, 
> the idea of classic ECN [RFC3168] is that the ECN marks replace the drops. In 
> all known testing (except pathological cases), classic ECN effectively 
> eliminates drops for all ECN-capable packets.

That's what I thought, and if that was the case, then marking packets as 
ECN-capable would mean that they would have an advantage over non-ECN packets 
(by not getting dropped, so getting a higher share of bandwidth)

that's what the gaming ECN thread was about, and if I understood the responses, 
I was being told that marking packets as ECN-capable, but not slowing down 
(actually responding to ECN) would not let an application get any advantage 
because the packets would just end up getting dropped anyway, since marking and 
dropping happen at the same level, even on ECN-capable flows.

If the packets are just marked, but not dropped, then the ECN-capable flows will 
occupy a disproportinate share of the available buffer space, since they just 
get marked instead of dropped.

David Lang

> Nonetheless, I do agree with your sentiment that the perfect is the enemy of 
> the good. We can remove most of the really bloat-induced latency without ECN. 
> So the message must be clear: Deploy AQM now. No need to wait for ECN. But 
> implementations SHOULD allow ECN packets to be classifed into a separately 
> configurable instance of the AQM algo. {Note 1}
> Similarly, this WG made sure we did not deprecate RED in the AQM 
> recommendations. Because, in existing equipment, even a poorly tuned RED is 
> usually much better than a bloated buffer with no AQM.
> Just as the WG mustn't confuse messages, you mustn't get confused by a 
> discussion about the potential for more ambitious reductions in latency. Koen 
> started the thread with reference to our presentation in the ICCRG in the 
> IRTF, where 'R' in both cases stands for Research.
> And I believe it is valid for the ECN benefits draft (in the IETF AQM WG) to 
> point to the potential of ECN, by using insights from research in progress.
> There are a number of different contributions to unnecessary latency, not 
> just the one popularised as bufferbloat:
> [In the following, I will use the term 'short message(s)' as shorthand for 
> either a short interactive flow or a long flow consisting of short 
> interactive messages (like in a game), or video frames or voice datagrams or 
> anything where the perceived latency depends on the latency of each 
> 'message', especially a string of messages with serial dependency as is 
> typical in Web.]
> #1 The well-known bufferbloat problem, where a long-running flow fills a 
> bloated buffer, delays short messages.
>  - AQM and/or flow queuing can remove this delay, without needing ECN.
> #2 A loss causes head-of-line blocking for short message(s) while waiting for 
> the retransmission.
>  - AQM without ECN cannot remove this delay.
>  - Flow queuing cannot remove this delay, if the losses are self-induced.
>  - FEC can remove this delay without needing ECN, but the increased 
> redundancy is equivalent to poorer utilisation (altho only the short flows 
> need the redundancy).
>  - ECN can remove this delay.
> #3 A loss near the end of a short flow can lead to multi-RTT delay.
>  - AQM without ECN cannot remove this delay.
>  - Techniques like tail-loss probe and RTO-restart can mitigate this delay, 
> without ECN, but not remove it.
>  - ECN can remove this delay.
> #4 The Reno/Cubic sawtooth causes variation in delay between 1 and 2 base 
> RTTs. This can affect short messages.
>  - AQM without ECN cannot remove this delay unless configured to sacrifice 
> utilisation.
>  - Flow queuing removes this delay variation if caused by a separate 
> long-running flow.{Note 2}
>  - A change to the TCP algo (e.g. DCTCP) can remove this delay variation. 
> Smaller sawteeth imply a much higher signalling rate, which in turn requires 
> ECN, otherwise drop probability would be excessive. (This was the main point 
> Koen was making.)
>  - Therefore, ECN can remove this delay.
> #5 Slow-starts{Note 3} cause spikes in delay.
>  - AQM without ECN cannot remove this delay, and typically AQM is designed 
> to allow such bursts of delay in the hope they will disappear of their own 
> accord.
>  - Flow queuing can remove the effect of these delay bursts on other flows, 
> but only if it gives all flows a separate queue from the start.{Note 2}
>  - Delay-based softening of slow-start, such as Hybrid Slow-Start in Linux, 
> can mitigate these variations, but with increased risk of coming out of SS 
> early, causing significantly longer completion time.
>  - ECN with AQM based on the instantaneous queue limits this delay, without 
> the risk of longer completion time.
> #6 Slow-starts{Note 3} can cause runs of losses, which in turn cause delays.
>  - AQM without ECN cannot remove these delays.
>  - Flow queuing cannot remove these losses, if self-induced.
>  - Delay-based SS like HSS can mitigate these losses, with increased risk of 
> longer completion time.
>  - ECN can remove these losses, and the consequent delays.
> Summary:
> * AQM alone solves the main problem
> * Flow queuing solves or mitigates most of the remaining secondary problems.
> * ECN has the potential to solve all the remaining secondary problems 
> (pending further research to prove some of them).
> Whether flow queuing is applicable depends on the scale. The work I'm doing 
> with Koen is to reduce the cost of the queuing mechanisms on our BNGs 
> (broadband network gateways). We're trying to reduce the cost of per-customer 
> queuing at scale, so per-flow queuing is simply out of the question. Whereas 
> ECN requires no more processing than drop.
> ECN has potential cheating problems, but we have per-customer queues anyway.
> Using flow as the unit of allocation also has its own problems, with no 
> proposed solutions.
> Bob
> {Note 1}: Your general point is that the perfect can be the enemy of the 
> good. Here's the presentation I gave in TSVAREA straight after VJ presented 
> CoDel in 2012 entitled "DCTCP & CoDel; the Best is the Friend of the Good."
> <>
> {Note 2}:  A lone flow can cause this delay variation to itself, but that's 
> irrelevant because, if the delay were not in the network it would be at the 
> sender.
> {Note 3}: Delay and loss spikes can equivalently be caused when Cubic's 
> window rises to seek out newly available capacity after another flow finishes 
> or the link rate varies.
> At 05:16 30/03/2015, David Lang wrote:
>> On Sat, 28 Mar 2015, Scheffenegger, Richard wrote:
>>> David,
>>> Perhaps you would care to provide some text to address the misconception 
>>> that you pointed out? (To wait for a 100% fix as a 90% fix appears much 
>>> less appealing, while the current state of art is at 0%)
>> Ok, you put me on the spot :-) Here goes.
>>> If you think that aqm-recommendations is not strogly enough worded. I 
>>> think this particular discussion (to aqm or not) really belongs there. The 
>>> other document (ecn benefits) has a different target in arguing for going 
>>> those last 10%...
>> so here is my "elevator pitch" on the problem. Feel free to take anything I 
>> say here for any purpose, and I'm sure I'll get corrected for anything I am 
>> wong on
>> Problem statement: Transmit buffers are needed to keep the network layer 
>> fully utilized, but excessive buffers result in poor latency for all 
>> traffic. This latency is frequently bad enough to cause some types of 
>> traffic to fail entirely.
>>   <link to more background goes here, including how separate benchmarks for 
>> throughput and latency have mislead people, "packet loss considered evil", 
>> cheaper memory encouraging larger buffers, etc. Include tests like 
>> netperf-wrapper and ping latency while under load, etc. Include examples 
>> where buffers have resulted in latencies so long that packets are 
>> retransmitted before the first copy gets to the destination>
>> Traditionally, transmit buffers have been sized to handle a fixed number of 
>> packets. Due to teh variation in packet sizes, it is impossible to tune 
>> this value to both keep the link fully utilized when small packets dominate 
>> the trafific without having the queue size be large enough to cause latency 
>> problems when large packets dominate the traffic.
>> Shifting to Byte Queue Lengths where queues are allowed to hold a variable 
>> number of packets depending on how large they are makes it possible to 
>> manually tune the transmit buffer size to get good latency under all 
>> traffic conditions at a given speed. However, this step forward revealed 
>> two additional problems.
>> 1. whenever the data rate changes, this value needs to be manually changed 
>> (multi-link paths loose a link, noise degrades max throughput on a link, 
>> etc)
>> 2. high volume flows (i.e. bulk downloads) can starve other flows (DNS 
>> lookups, VoIP, Gaming, etc). this happens because space in tue queue is on 
>> a first-com-first-served basis, so the high-volume traffic fills the queue 
>> (at which point it starts to be dropped), but all other traffic that tries 
>> to arrive is also dropped. It turns out that these light flows tend to have 
>> a larger effect on the user experience than heavier flows, because things 
>> tend to be serialized behind the lighter flows (DNS lookup before doing a 
>> large download, retrieving a small HTML page to find what additional 
>> resources need to be fetched to display a page), or the user experience is 
>> directly effected by light flows (gaming lag, VoIP drops, etc)
>> Active Queue Management addresses these problems by adapting the amount of 
>> data that is buffered to match the data transmission capacity, and prevents 
>> high volume flows from starving low-volume flows without the need to 
>> implement QoS classifications.
>>   <insert link about how you can't trust QoS tags that are made by other 
>> organizations, ways that it can be abused, etc>
>> This is possible because AQM algoithms don't have to drop the new packet 
>> that arrives, the algorithm can decide to drop the packet for one of the 
>> heavy flows rather than for one of the lightweight flows.
>> <insert references to currently favored AQM options here, PIE, fq_codel, 
>> cake, ???. Also links to failed approaches>
>> Turning on aqm on every bottleneck link makes the Internet usable for 
>> everyone, no matter what sort of application they are using.
>> <insert link on how to deal with equipment you can't configure by 
>> throttling bandwidth before the bottleneck oand/or doing ingress shaping of 
>> traffic>
>> While AQM makes the network usable, there is still additional room for 
>> improvement. While dropping packets does result in the TCP senders slowing 
>> down,and eventually stabilizing at around the right speed to keep the link 
>> fully utilized, the only way that senders have been able to detect problems 
>> is to discover that they have not received an ack for the traffic within 
>> the allowed time. This causes a 'bubble' in the flow as teh dropped packet 
>> must be retransmitted (and sometimes a significant amount of data after the 
>> dropped packet that did make it to the destination, but could not be acked 
>> because fo the missing packet).
>> This "bubble" in the data flow can be greatly compressed by configuring the 
>> AQM algorithm to send an ECN packet to the sender when it drops a packet in 
>> a flow. The sender can then adapt faster, slowing down it's new data, and 
>> re-sending the dropped packet without having to wait for the timeout. This 
>> has two major effects by allowing the sender to retransmit the packet 
>> sooner the dealy on the dropped data is not as long, and because the 
>> replacement data can arrive before the timeout of the following packets, 
>> they may not need to be re-sent. by configuring the AQM algorithm to send 
>> the ECN notification to the sender only when the packet is being dropped, 
>> the effect of failure of the ECN packet to get through to the sender (the 
>> notification packet runs into congestion and gets dropped, some network 
>> device blocks it, etc) is that the ECN enabled case devolves to match the 
>> non-ECN case in that the sender will still detect the dropped packet via 
>> the timeout waiting for the ack as if ENCN was not enabled.
>> <insert link to possible problems that can happen here, including the 
>> potential for an app to 'game' things if packets are marked at a different 
>> level than when they are dropped.>
>> So a very strong recommendation to enable Active Queue Management, while 
>> the different algorithms have different advantages and levels of testing, 
>> even the 'worst' of the set results in a night-and-day improvement for 
>> usability compared to unmanaged buffers.
>> Enabling ECN at the same point as dropping packets as part of enabling any 
>> AQM algorithm results in a noticable improvement over the base algorithm 
>> without ECN. When compared to the baseline, the improvement added by ECN is 
>> tiny compared to the improvement from enabling AQM.
>>   Is it fair to say that plain aqm vs aqm+ecn variation is on the same 
>> order of difference as the differences between the different AQM 
>> algorithms?
>> Future research items (which others here may already have done, and would 
>> not be part of my 'elevator pitch')
>> I believe that currently ECn triggers the exact same slowdown that a missed 
>> packet does, and it may be appropriate to have the sender do a less drastic 
>> slowdown.
>> It would be very interesing to provide soem way for the application sending 
>> the traffic to detect dropped packets and ECN responses. For example, a 
>> streaming media source (especially an interactive one like video 
>> conferencing) could adjust the bitrate that it's sending.
>> David Lang
>> _______________________________________________
>> aqm mailing list
> ________________________________________________________________
> Bob Briscoe,                                                  BT