Re: [tsvwg] New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-02.txt

Sebastian Moeller <> Mon, 22 March 2021 20:49 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id D49803A1103 for <>; Mon, 22 Mar 2021 13:49:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 0.252
X-Spam-Status: No, score=0.252 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id oEbgjsltxvCY for <>; Mon, 22 Mar 2021 13:49:27 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id C7E753A10FF for <>; Mon, 22 Mar 2021 13:49:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1616446119; bh=pU0e+XxflwYJlAuMc93vfdy+1agWQ8XczE/EZKb6o38=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=i+hr4IIvPvHU6lUu/SgSx2+FrGqUnIGQMNeCq/6rCq6dbsRIImKpqTdAopErLSulm O5WEg556PixQbG4jWQxfuL3sCwy1Sz5azYRfB+RYc+jNesqsXlKy1xjjxvTLBoLgR6 sf3uxsFKduPIZHXiP2d52eu1T0Fim5jQNsxZcSL8=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (mrgmx105 []) with ESMTPSA (Nemesis) id 1MAOJP-1lVMI02V35-00Bt5u; Mon, 22 Mar 2021 21:48:39 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.17\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Mon, 22 Mar 2021 21:48:37 +0100
Cc: Pete Heist <>, tsvwg IETF list <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <> <> <> <>
To: Bob Briscoe <>
X-Mailer: Apple Mail (2.3445.104.17)
X-Provags-ID: V03:K1:aJa6emHCsGzKCLfkDqUp580nhnpiUUJNzj9yreDSHIQocOAihTh tYpoAttUbtMie6u7lg+KSVm3wKD4TR+62BhZoWEzuNaeFIZSkCroKhZyEsrrSHDea6cBS3V cYoURhfrk/WLVdDNpjdqllBQqipSOpGwKEMprXDkehhdIwUJuATUZttD3dWfUlFXOkq9V/Y +G4O2AWjVrXP/MQqxweYA==
X-UI-Out-Filterresults: notjunk:1;V03:K0:GmMlTgZxLSA=:DTwb/19zsXGAfxnpc314/m l2G7bozQt4SwQBxFfLRZnw3oFqfT6fR0cdM6b5W1FIxZ8X35eDcQoWBKr43TcmCfIsKpLYbtG W6+qDUKWaPkDAR/5srmc/kHl5JGTOgH/BNm1pt1huvNjumO2Zc6EVJBDw/xRkK8d1ZQmEoBbq ogSx9dFQ3tfoYNNfgkWRLCq55aC+XsLuDj+P+9Et6PgsbUgW0Mal6cF/I6vjpfIAAcZT4qp41 sZO2aeA62A8dhaK0vQDEJwyrGWPNwP0Rla0Ds6goczxc80jmNAn0qOKbhCCpTX6/e/VQFScYw uWbFGJkI7mCI6gdLVew94oQp26GbAPUydXCURIRREe+c1d3VEc4w1oGWTttoutAf2+4CVgkTL lVx8ycRi1zVxM8l/Veu8RbgaP+sEUWummWx2ZIMOVu+2cD0N1Ly5gBZgL6JsoHnrSJzZ9Y3Qp 8+e2Rinco5gMDUCYjln6uKzRQcHoi9Z/iIN4/xUuMxAq2qeIdSqh+MRZtBRYoPBbm4qdJ01Rj 1BV3sIeMTLzKHwf3NmG/17t5KUiGaBI5hXOskQJ/Bz/3F/9oWKMI9dV5mXjMz+BTpnmfOsdda sHE0Ezrra3g3khEUNK3ciNJWbYf1GagsCWx2pwIfsKFiw35Q5Orfjqbr46BImtVHP41VBLiKJ vHlQeB+UiJStgD8JJD9V16TGjN5EwneaTE+2C7Txzl6p6WqS53xZOS/nYwvA4tELUHhJupLB4 FFpgX8Llp8YEyd7KiDxNPHJztSBLJW7iNo5yet6HEueQvce8OTa8T8WuvqTV+M13ADGDSeLHX cCzV+k01/FdKHzMxX4ZespJHBbU+yvrmzd04FU8r+eYwa0dxkFpAq6ONw4QBd9rClVxXNryje F5lg/y0QOKyktVJWDQZ5iZ5sT7WaTAB5whZUrF6qe09blM4vzCzGE4QLN+DJ/GtrqVqi56La5 g2Tb+/fgrwB4n87P6E0YF+ee7AUY/d+GrPPxnBd1EylPsP4X9Vq5IVTEdJMXy4H0TqHflCG5A 9/joGgKB1ia6lKYPTxmIOxxnaFRU1s+f7GCQ7KI5uB7bpJdACnWuWieU4vao0h9sUrsL1a16x y2hn/ptk0VOmi9xTU/FWtbOVvVvchUoU8PDGiuN9L7P4GLS9HiARZAO6jq3dY8DmVUzm37SHE GH6LmOqiJqumBRG2ep0OH3uJl0LJ0FGdfpm9zAgCJxeF6/4axnlM70KfbKq6wUEC3luSc=
Archived-At: <>
Subject: Re: [tsvwg] New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-02.txt
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 22 Mar 2021 20:49:32 -0000

Hi Bob,

More below, prefixed [SM].

On 22 March 2021 18:51:45 CET, Bob Briscoe <> wrote:
> Sebastian,
> On 09/03/2021 12:37, Sebastian Moeller wrote:
>>> On 08/03/2021 22:19, Pete Heist wrote:
>> [...]
>>>> IMO, fq_codel isn't ideal in this situation. When there actually is
>>>> congestion to manage, flow-fairness means that one user can
> dominate
>>>> the queue with many flows, which is exactly one of the problems
> we'd
>>>> like to avoid. Host fairness could improve on this, or better yet,
>>>> member fairness using the member database, as members can have more
>>>> than one IP/MAC. But, fq_codel is what's available now, and
> generally
>>>> usable enough in this case to get started with AQM.
>> [...]
>>> On Mar 9, 2021, at 01:25, Bob Briscoe <> wrote:
>> [...]
>>> PS. I agree it's v strange for an ISP to share capacity between its
> members by how many flows they are using.
>> [...]
>> I just want to note two things here:
>> a) even without a flow queueing AQM on the bottleneck, sharding works
> to get a higher part of the capacity since TCP tends to be more or less
> fair to itself
>> b) most ISP put a lid on games like that by simply also enforcing
> (policing or shaping) each users aggregate capacity to numbers
> mentioned in the contract...
>> So fq_codel, while by no means ideal here, does not really make
> thinks worst that the status quo, it is just that sharding can make
> pure flow fairness regress to less equitable sharing between classes
> other than flows.
> [BB] You seem to be using "the status quo" to mean TCP sharing out the 
> capacity between users. But that has never been the status quo - since 
> even before TCP congestion control was invented in 1988 per-customer 
> scheduling has been the norm for public ISPs {Note 1}. Irrespective of 
> access technology , whether DSL, DOCSIS cable, PON, mobile, satellite, 
> etc. They all share downstream capacity between customers at a node
> that 
> is either at the head of the bottleneck access link, or logically 
> limited to match the capacity of the customer's bottleneck access link.

[SM] Ah, no, with status quo I meant both a) and b) above. I considered my phrasing unambiguous, but still managed to confuse you, sorry for that.

To recapitulate, my point is sharding is unfortunate but hardly a problem unique to FQ, as of a) sharding will give gains even in the absence of FQ, and as of b) sharding from endusers might not actually be a problem ISP/transit networks need to worry about much, they can treat it as an SEP.

> You might be able to turn a few debating somersaults and claim that the
> problem is the absence of a per-customer scheduler, not the presence of an FQ scheduler where this scheduler would normally sit. If that allows you to sleep at night, you're welcome to live in your Alice through theLooking Glass world.

[SM] I probably could, but why would I? But since it seems you misunderstood me, maybe re-read what I wrote?

> {Note 1}: I can't immediately find evidence from the late 1980s of
> this. 
> All I can find is Appx B of TR-059 
> <;jsessionid=6924A1B1458AE2F6FEF15047BC95A7EC?doi=>
> from the DSL Forum, but that was only 2003 when QoS was added after 
> Diffserv was standardized. The closest I can get to something older is 
> the second half of RFC970 written in 1985, which was an early step 
> towards per-customer scheduling. You can skip over the first half that 
> is trying to solve an early form of bufferbloat that predated TCP 
> congestion control. Alternatively, I'm sure many people on this list 
> will be able to confirm that per-customer scheduling has 'always' been 
> the norm.

[SM] Thanks for digging this up. This nicely confirms my point b). I did not claim that to be a new development, even though I am not sure when ISPs switched from simple policers to user friendlier shapers?

>> For a cooperative use case, something like a per-member QFQ instance
> that equitably shares capacity between members with an fq_codel inside
> each of the QFQ classes seems like a better fit, no?
> Well, once you've got the per-member scheduling, an AQM in each
> member's 
> FIFO would suffice. But you can have your FQ-CoDel in there instead if 
> you must.

[SM] This is not so much a philosophical point as you might believe. I tried single and fq AQMs on my homelink, and for my use cases FQ works considerably better. And I know because I tried. BUT to do this for ingress I operate a traffic shaper on the wrong end of the bottleneck, if I could get my ISP to run fq-codel on the download side of my link at its BNG, I would be willing to pony up money for.
Again, I tried single queue AQMs and found them lacking. Currently I operate something close the proposed model, only I use cake for both the per-internal-IP isolation (member level) and for fq inside each of the top level aggregates. Works well for a family of five in spite of home schooling and video conferencing and remote desktop control, video streaming and downloads. 
I wish all L4S proponents here would take it upon themselves to try for only a week how the state of the art actually feels, but I digress.

> When a node is serving a few thousand members, you don't want to 
> unnecessarily have a thousand or so queues per member as well - at
> least 
> not if you can achieve similar performance without them.

[SM] That would be great. Data so far indicates however performance is not going to be similar.... at least not similar enough to make this a wort while endeavor. That said, if my ISP would offer even only RED on my ingress per-customer queue I would also already be willing to pay for. I would probably still supplement with FQ in my side, but worst case back-spill into the ISP's upstream buffers would be less of an issue.

> For instance, BT's network architect set me the task of simplifying
> BT's 
> QoS architecture, because even 5 queues per customer was considered to 
> be the main cause of system complexity. See Section 3.1.2 of this 
> report: 

[SM] I see, and I am not surprised. I also do not think your proposal is that much of a simplification, especially given the open ends like : "• Cons: Still we need convincing evidences that this would work, understand the packet loss trade-off especially for the interactive apps" and "• always plan sufficient capacity for sum of all VASs then no need for a separate priority for each value-added service" and "	• Isolation: between traffic within CVLAN e.g. femtocell, or third-party Wi-Fi network (such as FON), i.e. (customer vs non-customer traffic) can be achieved within one queue ( to be evaluated)" there seems to be an awful lot left to do before the proposal can replace the reference, no?

The head line problem "The use case is about serving HTTP Adaptive Streaming (HAS) in the presence of background TCP flows, when the downstream traffic per line is sufficient to saturate the queue at the BNG" is pretty much solved by cakes per-internal IP fairness as long as the per-IP share of capacity suffices for the HAS demands. If not the only solution is to statically carve out a reservation for that stream or grant priorities. (And nothing else is it you do, you allow the HAS streams to dominate the other TCP flows, which is fine if your users prefer the streams over the other data, but since that is not guaranteed to be true, doing that unconditionally seems not ideal). 
And as I might add, it does so without requiring the use of non-internet compatible transport protocols like DCTCP or TCP Prague (which in all fairness might turn out to be internet worthy, but has not convincingly done so so far).

Best Regards

> Bob
>> Best Regards
>> 	Sebastian