Re: [tcpm] [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S

Jonathan Morton <> Mon, 05 August 2019 15:56 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id A2584120192; Mon, 5 Aug 2019 08:56:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.748
X-Spam-Status: No, score=-1.748 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id UOe-0T31UnE4; Mon, 5 Aug 2019 08:56:13 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id B24A4120162; Mon, 5 Aug 2019 08:56:12 -0700 (PDT)
Received: by with SMTP id z28so25499958ljn.4; Mon, 05 Aug 2019 08:56:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=rCj3KubLzhyBvmt3m5sij9iCGDCmzKAOYbcx3Ijy/UE=; b=AuIcf2wP+i/uHF5LelV+I/CBg3bGbnkbEvCMFzmJ6yvBCevWde6CIgC9GV/1bAWbqN BjSMZUiFQphluXFLgS5i68BB/QYpSx1DPOu6n4KDNsT54DCtSAqPUPh2m3k+HTOMPo7V T5YBugS6ZuL/EOdeS6yMQrk+S1MPAGg+na8O4hDM9M5Q3PcoePWAv6OwVapiAnUD0Bh4 AxspfGoVmC/JpGYvPL+7XZt2leN4ivqHY1Ufyi6939CrUpiEGB9jmPeL/r6BCugJvszK IhyqXhffkA1aYI0IkvGLCbKx1m1EBHc8M0eLe8FPNvs52Slsc/brvSxtsEHnn5VBvwmZ iZsQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=rCj3KubLzhyBvmt3m5sij9iCGDCmzKAOYbcx3Ijy/UE=; b=jKpaSfUDDny24zx7yOqZufOZjN2opm7RID1N4RLhefIW5izMUX8lyp2MmEZPuMnsg8 BYuhJAg6kwVVwpfqAm066ZAAohIDp6P4Jgu085swH7C5kbI8q0l/Cyg7ZLwhxuEoi4Cr pfIiPXP95LgH8DK2ZrCORwPv4nkOD8gPdc4ciAUVa3iMEIoSv5qPFj4NeBKCZZAql/4I Ec6jl8bvNFED+R3JDsQqn/Keu8vY+Av7GNRuGTwSWxpky69s+lthYHzKmp/fPHLORjZY bbUpcODzkZ/Dc0qVNFKJgTWYYAqjc2FvTVGenitKquqNEM+wMmvtIyYe1mzc380hc5se M4/w==
X-Gm-Message-State: APjAAAXt35lOzHu25ZqfIDaUaTsKtiHaidjs6dyh2L6v0KZl8y3pWDxM GsHL25zRD3MNhDdnOXm/Sy8=
X-Google-Smtp-Source: APXvYqxQ0VHCvqunvmjjNc+SKehKFC0fho1tzHhaRQRnsPvpvdeEd2JYVTER9d3b1NKKdpo9NgDGfA==
X-Received: by 2002:a2e:9a13:: with SMTP id o19mr80649345lji.102.1565020570932; Mon, 05 Aug 2019 08:56:10 -0700 (PDT)
Received: from jonathartonsmbp.lan ( []) by with ESMTPSA id g12sm868113lfc.96.2019. (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Aug 2019 08:56:10 -0700 (PDT)
Content-Type: text/plain; charset="us-ascii"
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
From: Jonathan Morton <>
In-Reply-To: <LEXPR01MB046300B3D0F97AE65676D40E9CDA0@LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE>
Date: Mon, 05 Aug 2019 18:55:36 +0300
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <FRXPR01MB0710085AE4AEFA896819AFA69CD90@FRXPR01MB0710.DEUPRD01.PROD.OUTLOOK.DE> <> <LEXPR01MB046306842E5AB407A7BFC6619CDA0@LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE> <> <LEXPR01MB046300B3D0F97AE65676D40E9CDA0@LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE>
X-Mailer: Apple Mail (2.3445.9.1)
Archived-At: <>
Subject: Re: [tcpm] [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 05 Aug 2019 15:56:15 -0000

> On 5 Aug, 2019, at 3:16 pm, <> <> wrote:
> As I said, no corner-case engineering.

> In the market that I am aware of, there's a single regular bottleneck, which is the consumer access or terminal.

Let me explain one more time: this is *not* a corner case.  This is normal operation on today's Internet; faster core networks feed into larger numbers of slower edge networks in several stages.  If you haven't noticed it yourself, that's fortunate for you - but that "single regular bottleneck" is an illusory simplification, which only applies at relatively long timescales.

However, high-fidelity ECN markers *will* notice the difference, and the resulting congestion signals will appear at the receiver and be fed back to the sender.  That represents an engineering problem for which either a solution must be devised, or the fact that it is not a problem must be established.  At present, we're still working through the maths to determine the boundaries of acceptable operation.

It's even possible that conventional RFC-3168 AQMs notice it as well, depending on their design.  Codel, for example, is specifically designed to ignore transient queuing (if it persists for less than one 'interval', which is taken as an estimate of the RTT) and only act when a persistent standing queue exists.

And it's actually a very important problem when a dumb FIFO bottleneck is immediately followed by a slightly narrower AQM bottleneck.  In this case the dumb FIFO dilutes the benefit of the AQM significantly, because the AQM can't see that most of the queue exists in the upstream FIFO, and even if it could, it cannot apply any intelligence to it.  This is the scenario Sebastian is most concerned about, and for which I have tried to compensate in Cake with "ingress mode" shaping - because Cake is specifically designed to be deployed into that topology.

Most of the problem would go away if that dumb FIFO were merely replaced by a simple AQM.  This is a straightforward & deployable engineering solution that has been known for many years, and yet almost nobody has actually deployed it.  I would urge you to do what you can on that front; judging by your e-mail address, you probably have more influence where it counts than I do.

I repeat: this is normal operation on today's Internet.

> If your technical standardization activities help to make using the Internet more convenient in African countries without raising extra cost or requiring extra skills, I'm sure that's a fair market. I know that these countries are struggling to improve the services operated in their networks.

May I refer you to some useful work already done and widely deployed?  This is now the default in most Linux and Apple devices, and is also available in FreeBSD.  It is used by some of the better CPE devices as part of "Advanced QoS" and "Airtime Fairness", eg. in the Netgear R7000.

The problem is that it's not deployed at most of the actual bottlenecks, which mainly exist in ISPs' equipment if the core networks are indeed overprovisioned.  If it was deployed, congestion on the Internet would be a much easier problem than it is today.  And this should be especially applicable to less-developed areas of the Internet.

All it takes is enough ISPs manning up, talking to their hardware vendors, and saying: "We expect to have congestion occasionally/frequently (delete as applicable).  What AQM does your hardware support, and how do we turn it on?"  And if the answer is negative, being prepared to take business to a competitor who does.  The technology exists; money talks.

In the less developed parts of the world, one could easily jury rig an AQM router together using a Raspberry Pi and a couple of USB Ethernet dongles.  With the latest Raspberry Pi 4, that would work for up to a gigabit link; with older models, you can still handle 100Mbps.  These things are cheap enough to practically be disposable, and consume very little power.  If there's demand for that sort of thing, I and my colleagues could probably arrange for a ready-made SD card image to be built and published.

Or you could standardise on a consumer CPE router and build a no-knobs mesh-networking image for that.  There's a bunch of groups who regularly attend Battlemesh, who could help with that.  Be prepared to change models every few years as the old ones go out of production.

 - Jonathan Morton