Re: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02

Bob Briscoe <> Mon, 13 April 2015 15:11 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id 331391ACCF4 for <>; Mon, 13 Apr 2015 08:11:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 0.089
X-Spam-Status: No, score=0.089 tagged_above=-999 required=5 tests=[BAYES_50=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=ham
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id MeXCb9PXCRld for <>; Mon, 13 Apr 2015 08:11:27 -0700 (PDT)
Received: from ( []) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 01FE71AC43A for <>; Mon, 13 Apr 2015 08:11:24 -0700 (PDT)
Received: from ( by ( with Microsoft SMTP Server (TLS) id; Mon, 13 Apr 2015 16:11:20 +0100
Received: from ( by ( with Microsoft SMTP Server (TLS) id 8.3.348.2; Mon, 13 Apr 2015 16:11:22 +0100
Received: from ( by ( with Microsoft SMTP Server id; Mon, 13 Apr 2015 16:11:21 +0100
Received: from ([]) by (8.13.5/8.12.8) with ESMTP id t3DFBG3R002270; Mon, 13 Apr 2015 16:11:17 +0100
Message-ID: <>
X-Mailer: QUALCOMM Windows Eudora Version
Date: Mon, 13 Apr 2015 16:11:15 +0100
To: David Lang <>
From: Bob Briscoe <>
In-Reply-To: <>
References: <> <> <> <> <> <> <> <> <20150327183659.GI39886@verdi> <> <> <> <> <> <>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
X-Scanned-By: MIMEDefang 2.56 on
Archived-At: <>
Cc: "Scheffenegger, Richard" <>, Vishal Misra <>, KK <>, John Leslie <>, "" <>
Subject: Re: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 13 Apr 2015 15:11:31 -0000


Returning from a fortnight offlist...

I think your conception of how ECN works is incorrect. You describe 
ECN as if the AQM marks one packet when it drops another packet. You 
say that the ECN-mark speeds up the retransmission of the dropped 
packet. On the contrary, the idea of classic ECN [RFC3168] is that 
the ECN marks replace the drops. In all known testing (except 
pathological cases), classic ECN effectively eliminates drops for all 
ECN-capable packets.

Nonetheless, I do agree with your sentiment that the perfect is the 
enemy of the good. We can remove most of the really bloat-induced 
latency without ECN. So the message must be clear: Deploy AQM now. No 
need to wait for ECN. But implementations SHOULD allow ECN packets to 
be classifed into a separately configurable instance of the AQM algo. {Note 1}

Similarly, this WG made sure we did not deprecate RED in the AQM 
recommendations. Because, in existing equipment, even a poorly tuned 
RED is usually much better than a bloated buffer with no AQM.

Just as the WG mustn't confuse messages, you mustn't get confused by 
a discussion about the potential for more ambitious reductions in 
latency. Koen started the thread with reference to our presentation 
in the ICCRG in the IRTF, where 'R' in both cases stands for Research.

And I believe it is valid for the ECN benefits draft (in the IETF AQM 
WG) to point to the potential of ECN, by using insights from research 
in progress.

There are a number of different contributions to unnecessary latency, 
not just the one popularised as bufferbloat:

[In the following, I will use the term 'short message(s)' as 
shorthand for either a short interactive flow or a long flow 
consisting of short interactive messages (like in a game), or video 
frames or voice datagrams or anything where the perceived latency 
depends on the latency of each 'message', especially a string of 
messages with serial dependency as is typical in Web.]

#1 The well-known bufferbloat problem, where a long-running flow 
fills a bloated buffer, delays short messages.
   - AQM and/or flow queuing can remove this delay, without needing ECN.

#2 A loss causes head-of-line blocking for short message(s) while 
waiting for the retransmission.
   - AQM without ECN cannot remove this delay.
   - Flow queuing cannot remove this delay, if the losses are self-induced.
   - FEC can remove this delay without needing ECN, but the increased 
redundancy is equivalent to poorer utilisation (altho only the short 
flows need the redundancy).
   - ECN can remove this delay.

#3 A loss near the end of a short flow can lead to multi-RTT delay.
   - AQM without ECN cannot remove this delay.
   - Techniques like tail-loss probe and RTO-restart can mitigate 
this delay, without ECN, but not remove it.
   - ECN can remove this delay.

#4 The Reno/Cubic sawtooth causes variation in delay between 1 and 2 
base RTTs. This can affect short messages.
   - AQM without ECN cannot remove this delay unless configured to 
sacrifice utilisation.
   - Flow queuing removes this delay variation if caused by a 
separate long-running flow.{Note 2}
   - A change to the TCP algo (e.g. DCTCP) can remove this delay 
variation. Smaller sawteeth imply a much higher signalling rate, 
which in turn requires ECN, otherwise drop probability would be 
excessive. (This was the main point Koen was making.)
   - Therefore, ECN can remove this delay.

#5 Slow-starts{Note 3} cause spikes in delay.
   - AQM without ECN cannot remove this delay, and typically AQM is 
designed to allow such bursts of delay in the hope they will 
disappear of their own accord.
   - Flow queuing can remove the effect of these delay bursts on 
other flows, but only if it gives all flows a separate queue from the 
start.{Note 2}
   - Delay-based softening of slow-start, such as Hybrid Slow-Start 
in Linux, can mitigate these variations, but with increased risk of 
coming out of SS early, causing significantly longer completion time.
   - ECN with AQM based on the instantaneous queue limits this delay, 
without the risk of longer completion time.

#6 Slow-starts{Note 3} can cause runs of losses, which in turn cause delays.
   - AQM without ECN cannot remove these delays.
   - Flow queuing cannot remove these losses, if self-induced.
   - Delay-based SS like HSS can mitigate these losses, with 
increased risk of longer completion time.
   - ECN can remove these losses, and the consequent delays.

* AQM alone solves the main problem
* Flow queuing solves or mitigates most of the remaining secondary problems.
* ECN has the potential to solve all the remaining secondary problems 
(pending further research to prove some of them).

Whether flow queuing is applicable depends on the scale. The work I'm 
doing with Koen is to reduce the cost of the queuing mechanisms on 
our BNGs (broadband network gateways). We're trying to reduce the 
cost of per-customer queuing at scale, so per-flow queuing is simply 
out of the question. Whereas ECN requires no more processing than drop.

ECN has potential cheating problems, but we have per-customer queues anyway.
Using flow as the unit of allocation also has its own problems, with 
no proposed solutions.


{Note 1}: Your general point is that the perfect can be the enemy of 
the good. Here's the presentation I gave in TSVAREA straight after VJ 
presented CoDel in 2012 entitled "DCTCP & CoDel; the Best is the 
Friend of the Good."

{Note 2}:  A lone flow can cause this delay variation to itself, but 
that's irrelevant because, if the delay were not in the network it 
would be at the sender.

{Note 3}: Delay and loss spikes can equivalently be caused when 
Cubic's window rises to seek out newly available capacity after 
another flow finishes or the link rate varies.

At 05:16 30/03/2015, David Lang wrote:
>On Sat, 28 Mar 2015, Scheffenegger, Richard wrote:
>>Perhaps you would care to provide some text to address the 
>>misconception that you pointed out? (To wait for a 100% fix as a 
>>90% fix appears much less appealing, while the current state of art is at 0%)
>Ok, you put me on the spot :-) Here goes.
>>If you think that aqm-recommendations is not strogly enough worded. 
>>I think this particular discussion (to aqm or not) really belongs 
>>there. The other document (ecn benefits) has a different target in 
>>arguing for going those last 10%...
>so here is my "elevator pitch" on the problem. Feel free to take 
>anything I say here for any purpose, and I'm sure I'll get corrected 
>for anything I am wong on
>Problem statement: Transmit buffers are needed to keep the network 
>layer fully utilized, but excessive buffers result in poor latency 
>for all traffic. This latency is frequently bad enough to cause some 
>types of traffic to fail entirely.
>   <link to more background goes here, including how separate 
> benchmarks for throughput and latency have mislead people, "packet 
> loss considered evil", cheaper memory encouraging larger buffers, 
> etc. Include tests like netperf-wrapper and ping latency while 
> under load, etc. Include examples where buffers have resulted in 
> latencies so long that packets are retransmitted before the first 
> copy gets to the destination>
>Traditionally, transmit buffers have been sized to handle a fixed 
>number of packets. Due to teh variation in packet sizes, it is 
>impossible to tune this value to both keep the link fully utilized 
>when small packets dominate the trafific without having the queue 
>size be large enough to cause latency problems when large packets 
>dominate the traffic.
>Shifting to Byte Queue Lengths where queues are allowed to hold a 
>variable number of packets depending on how large they are makes it 
>possible to manually tune the transmit buffer size to get good 
>latency under all traffic conditions at a given speed. However, this 
>step forward revealed two additional problems.
>1. whenever the data rate changes, this value needs to be manually 
>changed (multi-link paths loose a link, noise degrades max 
>throughput on a link, etc)
>2. high volume flows (i.e. bulk downloads) can starve other flows 
>(DNS lookups, VoIP, Gaming, etc). this happens because space in tue 
>queue is on a first-com-first-served basis, so the high-volume 
>traffic fills the queue (at which point it starts to be dropped), 
>but all other traffic that tries to arrive is also dropped. It turns 
>out that these light flows tend to have a larger effect on the user 
>experience than heavier flows, because things tend to be serialized 
>behind the lighter flows (DNS lookup before doing a large download, 
>retrieving a small HTML page to find what additional resources need 
>to be fetched to display a page), or the user experience is directly 
>effected by light flows (gaming lag, VoIP drops, etc)
>Active Queue Management addresses these problems by adapting the 
>amount of data that is buffered to match the data transmission 
>capacity, and prevents high volume flows from starving low-volume 
>flows without the need to implement QoS classifications.
>   <insert link about how you can't trust QoS tags that are made by 
> other organizations, ways that it can be abused, etc>
>This is possible because AQM algoithms don't have to drop the new 
>packet that arrives, the algorithm can decide to drop the packet for 
>one of the heavy flows rather than for one of the lightweight flows.
><insert references to currently favored AQM options here, PIE, 
>fq_codel, cake, ???. Also links to failed approaches>
>Turning on aqm on every bottleneck link makes the Internet usable 
>for everyone, no matter what sort of application they are using.
><insert link on how to deal with equipment you can't configure by 
>throttling bandwidth before the bottleneck oand/or doing ingress 
>shaping of traffic>
>While AQM makes the network usable, there is still additional room 
>for improvement. While dropping packets does result in the TCP 
>senders slowing down,and eventually stabilizing at around the right 
>speed to keep the link fully utilized, the only way that senders 
>have been able to detect problems is to discover that they have not 
>received an ack for the traffic within the allowed time. This causes 
>a 'bubble' in the flow as teh dropped packet must be retransmitted 
>(and sometimes a significant amount of data after the dropped packet 
>that did make it to the destination, but could not be acked because 
>fo the missing packet).
>This "bubble" in the data flow can be greatly compressed by 
>configuring the AQM algorithm to send an ECN packet to the sender 
>when it drops a packet in a flow. The sender can then adapt faster, 
>slowing down it's new data, and re-sending the dropped packet 
>without having to wait for the timeout. This has two major effects 
>by allowing the sender to retransmit the packet sooner the dealy on 
>the dropped data is not as long, and because the replacement data 
>can arrive before the timeout of the following packets, they may not 
>need to be re-sent. by configuring the AQM algorithm to send the ECN 
>notification to the sender only when the packet is being dropped, 
>the effect of failure of the ECN packet to get through to the sender 
>(the notification packet runs into congestion and gets dropped, some 
>network device blocks it, etc) is that the ECN enabled case devolves 
>to match the non-ECN case in that the sender will still detect the 
>dropped packet via the timeout waiting for the ack as if ENCN was not enabled.
><insert link to possible problems that can happen here, including 
>the potential for an app to 'game' things if packets are marked at a 
>different level than when they are dropped.>
>So a very strong recommendation to enable Active Queue Management, 
>while the different algorithms have different advantages and levels 
>of testing, even the 'worst' of the set results in a night-and-day 
>improvement for usability compared to unmanaged buffers.
>Enabling ECN at the same point as dropping packets as part of 
>enabling any AQM algorithm results in a noticable improvement over 
>the base algorithm without ECN. When compared to the baseline, the 
>improvement added by ECN is tiny compared to the improvement from enabling AQM.
>   Is it fair to say that plain aqm vs aqm+ecn variation is on the 
> same order of difference as the differences between the different 
> AQM algorithms?
>Future research items (which others here may already have done, and 
>would not be part of my 'elevator pitch')
>I believe that currently ECn triggers the exact same slowdown that a 
>missed packet does, and it may be appropriate to have the sender do 
>a less drastic slowdown.
>It would be very interesing to provide soem way for the application 
>sending the traffic to detect dropped packets and ECN responses. For 
>example, a streaming media source (especially an interactive one 
>like video conferencing) could adjust the bitrate that it's sending.
>David Lang
>aqm mailing list

Bob Briscoe,                                                  BT