Re: [tcpPrague] [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

Sebastian Moeller <> Sun, 01 November 2020 09:55 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 420AC3A100A; Sun, 1 Nov 2020 01:55:55 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.649
X-Spam-Status: No, score=-1.649 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 8XRCBAeA_TGM; Sun, 1 Nov 2020 01:55:52 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 553F53A0976; Sun, 1 Nov 2020 01:55:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1604224507; bh=unF9f2Fnl31oD7szxkCT7ywkabzLt7gLW/aCaTUxwRM=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:In-Reply-To:References; b=gKRU+tWWsV84ih1dRrz3sbzQ9S0G481rJuFX1E8tN9mPTxNBo3sMh+QK44PKwdRMd 7LBeSpkgdGmWMJBoCaxvbAVvkfttt9uCh4Fp7LeHN81tri0A2Ir3ltecy8LGggeNmi Ix/GnpA9Aa09q+KNaqpNvr+npk5FlvsvfvZpq7gk=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (3c-app-gmx-bs25.server.lan []) (via HTTP); Sun, 1 Nov 2020 10:55:07 +0100
MIME-Version: 1.0
Message-ID: <trinity-64639b10-df53-4835-9328-408c01d211a1-1604224507500@3c-app-gmx-bs25>
From: Sebastian Moeller <>
To: "De Schepper, Koen (Nokia - BE/Antwerp)" <>
Cc: Bob Briscoe <>, tsvwg IETF list <>, iccrg IRTF list <>, TCP Prague List <>
Content-Type: text/plain; charset="UTF-8"
Date: Sun, 01 Nov 2020 10:55:07 +0100
Importance: normal
Sensitivity: Normal
In-Reply-To: <>
References: <> <trinity-fd329c3f-42fa-4cf6-b038-d5b3c4101f42-1604159383392@3c-app-gmx-bs36> <>
Content-Transfer-Encoding: quoted-printable
X-UI-Message-Type: mail
X-Priority: 3
X-Provags-ID: V03:K1:FU59iJhQm2QvieiB1I6mJOyL8zyJt1PfVs4WoFDTS3RiPZp+kH9fSNEmTLrQThSGgV8iQ OygEykakpWS5IjoPTSPNFQoDtdt82GUSMYr/1k/DZ/D7reawUC8bDGOot/oFS7USKTV/UyEPwVi4 5HomGVYjTFSDz8JpcRJikxa0G92yiZffawb9L+6jkakd1+BlAl+z61MrkfCIIF9GCTeeQV5TyLfT jA5InUJvxI40Z1HZS9HyTtYbZ+jTuFYfXcNk2SoKo6EE0LUp6zPQs4tmIbMFhT23mGBDVBhtXRa1 WQ=
X-UI-Out-Filterresults: notjunk:1;V03:K0:nnCDNn3S06U=:lfTmh4f/cXcYum6wBwB67E 6FudlZkBE4Fcd75+xoL7QUxR4Y7AFCs6mAmXoA4sw4U1kFwR+8XuLYBCQXZ9wJo2G9NqRe3Sv NftD/95XpZXG18OcfRsTQvS3nuMsm6q/tYaDyveM7DraA92eg/qVS42CRMQ27HLTl6EU519Fv xflgrq3wezB35nq4AzfX5+6sQkA7KS52mPwCN/4hhSPcCzbMTqjUL2HSLropERcS9DRATfZeT V86Ru4oMxIAUrd/STY2eaNIKgvFVu1akg+ok2deQmtkhPWCC8rmRj1AwiKxmU1tC17YwqnmxJ +larX55y+NfM4TpKcqVH8h2PwQaaOngJi0dB1gDVBDzWBKudvqzDT7lCa+iUbGLjv4tImcpYI gRMtUedLoabPtFRz2JZkcVggldBVGb5zYnoBiIiw4N/iEQ+A0irCAMEgBkXecEXMB5gTnQ5Z8 c7pXg1F530+tn+lQgRF2sTE/45jW/itrWNzSPVtfDK/i4wWIqaLRR0FV3NZWSIqm4D0JJ+t98 37ZDdhphDMfJqH64eI1HlS/dFIUG179Dtt8qVo1CkKa0ZjFDitLjzTvlGTe62izpJRz93VSq1 VAsq7ByY18FDU=
Archived-At: <>
Subject: Re: [tcpPrague] [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 01 Nov 2020 09:55:55 -0000

Hi Koen,

more below in-line, prefixed with [SM]

Gesendet: Samstag, 31. Oktober 2020 um 23:33 Uhr
Von: "De Schepper, Koen (Nokia - BE/Antwerp)" <>
An: "Sebastian Moeller" <>, "Bob Briscoe" <>
Cc: "tsvwg IETF list" <>, "iccrg IRTF list" <>, "TCP Prague List" <>
Betreff: RE: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

Hi Sebastian,
It was never our intent to develop the Congestion Control itself, as it was not in our interest (as a network provider). 

        [SM] I understand that, but it turns out that to asses functionality and safety of the proposed changes to ECN signalling you absolutely required a way to actually test under realistic conditions, and hence your hand was forced if ever to succeed. As a consequence TCP Prague developed into your umping ground for features you did not really want but had to accept... like the abysmal 15ms hack (good luck getting that into any OS networking stack).

There are plenty of bright people that can develop a Prague-compliant CC that will benefit their business (and at least comply to the safety measures). 

        [SM] But it still is incumbent upon you top demonstrate that the requirements are actually achievable and sufficient to guarantee functionality and safety (as that is the prove that L4S actually would work). I understand that most of the little conditions tested used DCTCP as protocol, so even for the one condition tested so far, short-RTT low HOP-count paths, most tests will need to be repeated with TCP Prague, not that I am holding my breath to see those results, given that I outlined a number of completely missing tests to evaluate functionality and safety outside the narrow conditions tested so far.

>From the start we realized that the small buffer is a potential safety issue for the typical r~=1/RTT congestion controls (also confirmed by our early tests and later by Pete’s tests too).

        [SM] IIRC, that was a well established fact and not a terribly novel insight...

 We hoped by showing that (once one puzzles out how) making a CC RTT independent for the unsafe range (small RTTs) is an easy obvious hack. To please you, we even made a “Sebastian-version” with f(rtt)=rtt+15ms that only removes the RTT benefit from the DualQ (one of your main complaints), and behaves further as a Classic RTT dependent flow. The Sebastian-version solves the safety issue and your fairness issue, but is not our preferred version.

        [SM] I do take offence in that name Sebastian-version, and in presenting that hack as anything but a work-around for a DualQ design error. As I explained patiently in the past, normal RTT dependence of TCP throughput is not a function of the endpoints (as as long as a single flow along a path can saturate the link, the end-points obviously are capable of reaching their goal independent of RTT, hence are RTT independent, I know this is within reason, as there are limits on the aviebale throughput depending on RTT and maximal window sizes, but these are not the issues at play here) but of the buffering in the network, most importantly the buffering at the bottlenecks. And hence that issue should be solved there.
BUT to add insult to injury, you are only proposing to make up for the 15ms DualQ might be configured with (might be as last time I looked at the code it looked more like classic's target was configured to 10ms, but I digress).

 We like more the one that solves the safety issue with f(rtt)=max(rtt, 15ms), and still allows L4S to benefit from the lower delay in all other cases.

        [SM] This is not a safety issue. It is a rate throttling issue. But humour me, how does curvy RED actually perform with all of these gross hacks, under which version does it achieve equitable sharing? 

Then thanks to your insisting to realizse literally what we have written, 

        [SM] So you intended to formulate requirements in your drafts as loose guiding principles without actual interpreting these as binding definitions? And you needed my input to realise basically what "requirement" means. Koen, I have a hard time believing that. 

we realized that the text was indeed too strict if read out of the larger context. In the context of the safety requirements section, we never wanted to enforce RTT fairness for flows with a bigger than the reference RTT, as they don’t pose a safety issue (it is a reality for all longer RTTs today, partially solved by Cubic already), and the quoted text could indeed be interpreted that way out of the larger context. Blindly trying to implement RTT independence for larger RTTs could even make it unsafe, if not done correctly. Of course, if wanted and beneficial, others are free to further optimize.
        [SM] You have been advertising L4S on pie-in-the-sky features like RTT-independence for years, in spite of having been told about it impossibility before, and now you seem to finally understand that. Again, I have a hard time believing that. I am also puzzled how this fits with how engineers typically are thought to work.

So I hope you understand now why we want to fix this text (and don’t see this as a failure), and hope you are ok too, to keep this safety requirement to the safety aspect only. 

        [SM] No, as I stated in the past, the 15ms issue, is a DialQ issue, and needs to be fixed in DualQ, then the RTT-independence requirement can be kept as purely inspirational. The current version with its hedge "as wide a range of RTTs as possible" is IMHO still better than the new "the minimum likely RTT and typical RTTs expected in the intended deployment scenario" because that is plainly wrong. By artificially coupling the CC response and the AQM (with the brain dead 15ms) your AQM no MUST not account for likely or expected RTTs but it MUST account for what the CCs are configured for, otherwise the uneqitable sharing issue will crop up again.
       Again this is a logical conclusion based on the unfortunate layer violating design choices you made, so please do not frame this as an issue of RTTs.
        IMHO, the fact that DualQ has this issue is a clear indication, that the cute coupling idea, simply is not suitable for the messy reality, and hacking around that by a requirement for transport protocols is insane, BUT then not clearly pointing out the dependency in the requirement section like "the CC's temporal response dynamic lower bound needs to be configured to match the L4S AQM expected on the path, or 15ms by default" is not going to make protocol implementers understand this subtle issue, as this is NOT an RTT bias issue to start with that is supposed to be covered by this requirement, but a DualQ misdesgn. Hiding this relations does not help anybody.

If you think the text is still not clearly representing the safety requirement of the RTT independence aspect, then your suggestions are very welcome.
        [SM] If you truly believe that, then I fear you have not understood the issue very well. Try to read this recommendation from the perspective of a naive protocol implementer. How will she know that she needs to look into the code/default configuration of the DualQ to be able to properly define the response dynamics lower bound, heck how is she supposed to understand that she needs to artificially slow down the CC response for some RTTs and make the CC control loop actively worse from the text of the requirement. IMHO the way this is phrased tries to hide the actual causalities at play here, and that never is a good idea in engineering or science.

As to your request to let the AQM make flows RTT independent, that is unfortunately impossible without knowing the RTT of each packet or identifying and treating the flows individually. 

        [SM] Well, it is less impossible than for the end-points. As long as you keep that requirement at all (and I can agree with getting rid of of it, as it is impossible to achieve realistically) you IMHO NEED to solve it at the component/layer responsible for that issue. Arguing it is hard for the AQM does not hold water as it is IMPOSSIBLE for the end-points in any useful generalized fashion. Giving the current proposals, I can almost guarantee that none of alternative transport protocols to TCP Prague will implement that  15ms hack at all (as the requirement is sufficiently vague to make it impossible to understand what is actually going on unless the implementer followed the L4S saga on this list*), and hence your safety issue will not be addressed at all. Again, I have a hard time believing that you assume that the changed text can be interpreted correctly without a lot of unwritten information, so how about you add a description of how a errr in DualQ makes a 15ms recommended?

So can you explain why you claim that what actually is easy (RTT-indep CC) is impossible and why you insist on what is impossible (RTT-indep AQM) should be done instead???
        [SM] Yes, easily. RTT dependence is a direct result of processes of competition at the bottleneck queue, where shorter RTT translates to tighter control loops and hence faster reactions to changing estimates of capacity, that way shorter RTT flows pick up the excess capacity the transiently appears when (longer RTT) flows go to multiplicative decrease. In addition the competition the bottleneck buffer can lead to a higher dropping probability for log RTT flows (precisely because they react slower). There are two ways to tackle this:
a) artificially slow down all flows to the same response dynamics as the longest RTT flow displays. That is the way you started on, but you clearly realised how sub-optimal this is, given that RTT are not really bounded to anything sane (think paths over geostationary satellites), so you already cheated by limiting that to the 15ms you foster on non L4S flows in your AQM.
       Now, I admit that technically it is not completely IMPOSSIBLE to generally solve this issue, by making all flows react as if their RTT would be >= say 500ms, but I do mantain, that nobody sane would accept that as a viable solution. Making it effectively possible but inacceptable.

b) ameliorate the competition in the bottleneck buffers. That is exactly par for the course for advanced queue management.

I have made the same arguments in the past, without convincing demonstrations that my theory is flawed, but feel free to poke holes into it. But again, hard to believe that you seem to have forgotten, that the two of us had pretty much that discussion before.

Best Regards

*) Comments from members of the BBR and Quicc communities indicate no sufficiently deep involvement as far as I can tell.

Thanks and regards,

From: Sebastian Moeller <>
Sent: Saturday, October 31, 2020 4:50 PM
To: Bob Briscoe <>
Cc: tsvwg IETF list <>; iccrg IRTF list <>; TCP Prague List <>; De Schepper, Koen (Nokia - BE/Antwerp) <>
Subject: Aw: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

Dear list,


please note that both of these "requirements" hedge against the fact that RTT independence is almost impossible to achieve from the endpoints (as it is a consequence of buffering at the bottleneck), originally bzy stating "as wide a range of RTTs as possible" (which given the phisical constraints in reality is equal to the empty set). Replacing that non-requirement with a differently worded version ("as much as possible") does not change anything substantially, except for making TCP Prague appear less of a failure. 

IMNHO this requirement for an L4S compatible transport, should be replaced with a requirement for an L4S compliant AQM (as it is limited queueing at the bottleneck that causes most the issue) MUST at least perform as well as a dumb FIFO, a mark which DualQ at short RTTs misses badly. But I guess, a no regressions against the status quo requirement, will probably not apply to L4S.


Best Regards




Gesendet: Freitag, 30. Oktober 2020 um 19:42 Uhr
Von: "Bob Briscoe" <[]>
An: "tsvwg IETF list" <[]>
Cc: "iccrg IRTF list" <[]>, "TCP Prague List" <[]>, "De Schepper, Koen (Koen)" <[]>
Betreff: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text


The co-authors of ECN L4S ID have been reviewing the correctness of the normative 'Prague' requirements.
This is the first of 2 emails, about 2 of the requirements that we think ought to be reworded a little.

If you agree with the rationale, but think the new wording still doesn't capture the requirement well, pls suggest sthg better.
If you disagree with the rationale, pls discuss.
4.3.  Prerequisite Congestion Response
   o  A scalable congestion control MUST reduce or eliminate RTT bias
      over as wide a range of RTTs as possible, or at least over the
      typical range of RTTs that will interact in the intended
      deployment scenario (see Appendix A.1.5[] for rationale).

A scalable congestion control MUST eliminate RTT bias as much as possible in the range between the minimum likely RTT and typical RTTs expected
in the intended deployment scenario  (see Appendix A.1.5 for rationale).

1/ "eliminate as much as possible" is stronger than "reduce or eliminate".
2/ This requirement was motivated by 'do no harm to others' relative to existing standard (RFC5681 Reno) congestion control. So there is no need to mandate that an L4S implementer does no harm to themselves, which window-based congestion controls tend to do at higher RTT. Of course, this doesn't preclude implementers reducing or eliminating RTT bias for larger than typical RTTs, but it removes any requirement to do so.


Bob Briscoe                     []