Re: [tsvwg] RTT-independence

[Bob Briscoe <[e-mail]>]

Sebastian,

On 18/12/2019 08:11, Sebastian Moeller wrote:
> Hi Bob,
>   more below in-line.
>
>> On Dec 18, 2019, at 01:23, Bob Briscoe <[e-mail]> wrote:
>>
>> Sebastian,
>>
>> I wanted to pick up your point about RTT-independence that you recently said has never been responded to.
>>
>> I have changed the subject line, because this is not an L4S vs SCE issue. With a shared queue it is just as much an issue for SCE as for L4S. With an FQ scheduler, it is just as much a non-issue for L4S as for SCE.
>>
>>
>> On 20/11/2019 14:18, Sebastian Moeller wrote:
>>> Dear Koen, dear all,
>>>
>>>
>>>
>>>> On Nov 20, 2019, at 13:40, De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]>
>>>>   wrote:
>>>>
>>>>
>>>> One of these opportunities here is to make L4S_TCP less RTT dependent. There have been many working implementations for less RTT-dependent CCs in the past. One that is widely deployed is Cubic, which is doing this for getting more throughput for longer RTTs. The only reason why it didn’t fly for lower RTTs on the current Internet is that they would hurt themselves (get lower throughput, competing with Reno).
>>>>
>>> 	[SM] Looking at the pfifo_fast results in Høiland-Jørgensen T, Hurtig P, Brunstrom A (2015) The Good, the Bad and the WiFi: Modern AQMs in a residential setting. Computer Networks 89:90–106. For Cubic/pfifo_fast (linux former default combination), I fail to see a strong indicator that cubic is RTT invariant or getting more thoughput at longer RTTs (except for the 10ms versus 50ms "hump"). What paper/data should I be looking at instead?
>>>
>>>
>>>>   
>>>> As we are able to define a new independent behavior and the RTT dependence in L4S is taken serious (some even call it a show stopper) this is even a must do opportunity for L4S. It is all a matter of perspective: show-stopper <-> opportunity.
>>>>
>>> 	[SM] I believe that working on more RTT independence is worth-while but no replacement for fixing the isolation system as it it is that equitable isolation from existing traffic that basically gives you the slate clean-"green field" development opportunity, no?
>> [BB] RTT-dependence is caused by the end-system, and should be solved in the end-system (whether or not FQ is also deployed to solve it in some places).
> 	[SM] I fully agree, and that is also a convincing argument, why increasing RTT independence can not be the solution for the observed isolation failure of the dual queue coupled AQM, glad that you agree.
[BB] There is no convincing argument yet. This is an opening sentence, 
from which neither I nor you can make any convincing argument. Please 
work through the whole argument below...


>
>
>
>> Importantly, RTT-dependence only needs to be addressed in the newly deployed scalable (L4S or SCE) end-systems, not pre-existing end-systems like Reno & Cubic, as I'll now explain.
>>
>> The problem is due to the tiny RTT you can get when you have both a tiny base RTT and a tiny queue. For example:
>>
>> Base RTT/ms
>> Queue/ms
>> Total RTT/ms
>> Close Scalable flow:	2.5
>> 0.5
>> 3
>> Distant Scalable flow:	100
>> 0.5
>> 100.5
>> Close Classic flow:	2.5
>> 15
>> 7.5
> 	[SM] Note, 15 + 2.5 = 17.5, a typo for sure. If I am wrong and you really ment 7.5, please elaborate why you only add 1/3 of the queueing delay, as I assume we are talking about saturating traffic here.
>
>
>> Distant Classic flow:	100
>> 15
>> 105
> 	[SM] 15 + 105 = 120
>
>> If all the above flows were RTT-dependent, the lowest 3ms RTT ('Close Scalable') would be the potential hog. For example against 105ms ('Distant Classic') if they were both 'window fair' like Reno, the throughput ratio would be 105:3 = 35:1
[BB] Sorry, I screwed up that table. I decided to use a 5ms queue 
instead of a 15ms queue at the last minute (in order to use the worst 
case for my argument), however I only changed the totals, and omitted to 
change the queue itself. Here's the corrected table:



	Base RTT/ms
	Queue/ms
	Total RTT/ms
Close Scalable flow: 	2.5
	0.5
	3
Distant Scalable flow: 	100
	0.5
	100.5
Close Classic flow: 	2.5
	5.0
	7.5
Distant Classic flow: 	100
	5.0
	105


All the arguments that followed remain the same though, because they 
relied on the totals not the constituent parts of the totals.

> 	[SM] I am not convinced that "window fairness" is in reality a useful property, throughput fairness seems much more desirable.
[BB] 'Window fairness' is not a useful property, it's just a property of 
a large class of congestion controls. The word 'fairness' sounds 
intrinsically like people should think it is desirable. So I try to 
avoid using it, and if I do, I try to always put it in quotes.

>
>> Whereas the Close vs Distant Classic case is cushioned by the 15ms queue. RTT ratio = 105:7.5 = 14:1, which was the Internet status quo pre-L4S (or SCE).
> 	[SM] If I fill in the modified numbers from above I get 115 / 17.5  ~ 6.6:1, but your point still stands.
[BB] Please now revisit the arguments below without the distraction of 
the typos.
>
>> So why is it enough for only scalable flows to reduce their RTT dependence? Assuming they do, the lowest 3ms RTT ('Close Scalable') doesn't go much faster than 'Distant Scalable'. If it's no longer a hog in relation to Distant Scalable, it's no longer a hog in relation to any of the other flows.
> 	[SM] That seem strictly depend on your definition of hog 105/120 is not equal still giving L4S an IMHO (hard to justify) edge over standard compliant traffic. But I am happy to be convinced by actual empirical numbers.
[BB] Oh dear. What is so important to you about such precise rate 
equality? Please be prepared to set aside your preconceptions about rate 
equality. Please re-read the reference at {Note 1}. No-one has ever been 
able to justify a goal of equal flow-rates, except for starvation 
avoidance (when rough rate equality is sufficient). Congestion levels 
have to be high for starvation to be a concern. So, the lower congestion 
is, the less important rate equality becomes.

There's a lot more to this, so please read and try really hard to 
understand /all/ the points in that reference at {Note 1}.

For instance 'rate fairness' is also not a useful goal because, like 
'window fairness', it makes the comparison in inappropriate units - if 
you try to say what is 'fair' at each instant without taking account of 
congestion over time, you'll get a much worse outcome (see the first 
argument in section 2.1.1 of the reference at {Note 1}).

I recall that you (if my memory serves me correctly) dismissed that 
argument in section 2.1.1  because you couldn't see how to calculate the 
correct shares at any one time. You'd need to read the references for 
that. But even if you can't work out the perfect solution, doesn't mean 
you cannot see that, for flows with different volumes, highly unequal 
flow rates give much shorter flow completion times than equal flow 
rates. The idea is to start you questioning your belief in equal flow rates.

Nonetheless, compared to 'window fairness', 'rate fairness' is a more 
useful /baseline/ on which to build other more useful behaviours, 
because at least it avoids starvation.

So, please understand that the reason I prefer an RTT-independent 
congestion control is not because I believe 'rate fairness' is a 
desirable goal. In the presence of very shallow queues, 'rate-fairness' 
is just less undesirable than 'window fairness', because 'rate-fairness' 
avoids starvation.


>
>> So the worst case reverts to the Close Classic flow, which isn't so bad because the total RTT is cushioned by the queue. So we will be no worse off than we were before L4S or SCE.
>>
>>
>> You will see we explained this at the end of Section V.B 'Congestion Control Roadmap' in the draft journal paper about L4S that we posted earlier this year:
>> “`Data Centre to the Home’: Deployable Ultra-Low Latency for All”
>>
>> But is RTT-independent TCP Prague real? No, not quite yet. At the time we designed this, we simulated it, but we're now working on it again, including implementing it in Linux, so you should see results soon. The approach to RTT-independence that we suggest is explained in section 3.2 of a paper we presented to the iccrg back in Jul 2017.
>> “Resolving Tensions Between Congestion Control Scaling Requirements“
> 	[SM] At this point I want to see real numbers from empirical testing, thanks for the references though.
>
>
>>
>> All the above papers are available via the L4S landing page: https://riteproject.eu/dctth/#papers
>>>> I'm also not against the recent vibrant energy, interest and engagement of people, but I think we can better use this energy in making things work. As you noticed, we can use the help to speed things up on the open source part.
>>>>
>>> 	[SM]... while at the same time requesting help for implementation and dealing with the consequences of said "code point and requirements".
>> If you recall, when I asked you not to keep repeating other people's arguments without adding anything, you said the RTT-dependence point was your contribution to the debate.
> 	[SM] Then we miscommunicated, my point is, that if you fix the dual queue couplked AQM than you can tackle the worthy goal of less RTT dependence at your own sweet time.
[BB] You never say what you mean by 'fix the dualQ coupled AQM'. I am 
not going to know what you mean because, from my perspective, the AQM 
does what it was designed to do very well. It doesn't determine per-flow 
shares of capacity. It merely emits congestion signals. Per-flow shares 
of capacity are determined by each sender's response to those signals.

You clearly want the AQM to do something different, but you don't say 
what. I assume you don't mean just "replace it with FQ" {Note 2}.

>> The L4S team identified this problem in early 2015 during testing with diverse RTTs.
> 	[SM] And then opted to keep pushing for internet-wide roll-out of L4S with this issue unsolved, color me not impressed.
[BB] We were pushing for Internet-wide roll-out of the /network/ part of 
L4S. We were from network companies and we thought others would do the 
end-system parts - hence the Prague requirements. We thought DCTCP was a 
sufficient starting point for others to build on. Once I realized that 
wasn't happening (except in Google) without a reference implementation, 
I left CableLabs and got funding to hire folks to work on TCP Prague 
with me.

>
>
>> When the Prague requirements were first agreed, we (Koen in particular) insisted that this should be included as a basic safety requirement, even tho some people said it wasn't important.
>> The WG has so far kept it as a MUST requirement, justified by the potentially large discrepancy in rates that can result, articulated here:
>>      https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08#appendix-A.1.5
>>
>> What I wanted to show by this email is that a significant amount of work has been done on this, the problem is well-understood and solutions are in progress. RTT-independent congestion controls have been produced before - not that I'm belittling how hard it is to resolve the tension between requirements identified in the 'Tensions' paper above.
> 	[SM] To repeat myself, this all supports the thesis that making RTT-independence the mechanism by which L4S equitably shares with standards-compliant traffic at the current state is premature.
[BB] It is not premature to develop an RTT-independence solution that we 
have shown to work in simulations, and that we know has been implemented 
successfully in other congestion controllers in the past.

> I very much would like to see the dual queue coupled AQM fixed, and once RTT independence (for all users of the L4S queue) is functional then the AQM can be changed again.
[BB] If you mean that we should bias the signals emitted by the DualQ to 
counterbalance the incorrect response of RTT-dependent senders, that 
would create a nightmare "incremental undeployment" problem. Then, 
no-one would ever be able to fix any of these broken (RTT-dependent) 
congestion controllers because, during the transition, the 
counterbalances added to AQMs would remain in the network and cause 
RTT-unfairness in the opposite direction. So reverse engineering the AQM 
to counter broken senders would freeze both into the Internet for ever.

So, please accept that is not the way it's going to happen. The DualQ 
Coupled AQM does not need to be fixed because it isn't broken - it is 
very much working as intended.

Network deployments take years and remain for years. End system 
deployments can continuously evolve.

My point at the start was that RTT-dependence is an aspect of the way 
DCTCP senders respond to the signals emitted by queues. So the correct 
place to fix RTT-dependence is in those senders.


>
>
>> So I ask you to take a more constructive approach. Banging on your keyboard in the style of "ner ner ne ner ner, I found a problem with L4S" doesn't help anyone,
> 	[SM] Let me be blunt, without the deserved push-back on the lists, "peaceful coexistence with single queue RFC3168 AQMs" would not have been tackled, the gist of the presentatuins so far has always been to ask the room, but do we ned to care for this at all? So IMHO pointing out L4S current short comings is important as otherwise little progress on the difficult issues would happen at all.
[BB] The concerns about single-queue RFC3168 AQMs did indeed succeed in 
redirecting my attention from RTT-independence, which I would otherwise 
still be focusing on. Olivier at least is still working on that.

I am concerned that still no-one has yet found a single-queue RFC3168 
AQM deployed on the public Internet, or a network operator who wants 
one. No doubt there are some out there, but I have no idea how large or 
small the problem really is.

It would be preferable if you would at least gather data to help 
prioritize this work, by quantifying the prevalence and impact of these 
problems. But I can't stop you just repeating known problems in email 
after email with no data to support your invective.

>
>> when the problem you've "found" exists with any low latency shared queue, including an SCE-based solution.
> 	[SM] Well, the something called "dual queue" AQM certainly needs to at least behave as if it had two queues in my book.
[BB] It does. The L queue isolates its traffic from the latency of the 
other, and they each emit different but coupled levels of congestion 
signals.

> L4S versus SCE is red herring here,
[BB] What I said was not pitting L4S against SCE. Quite the opposite.

I said that an SCE transport also needs to be RTT-independent, for the 
cases where it runs over a single queue SCE AQM (e.g. CNQ). I do not see 
any work being done on RTT-independence in SCE transports at all. And I 
do not see you criticising anyone for not having done this.

> my current issue is that the current implementation of L4S's reference AQM does not work as it should and as a normal reader would understand from reading the dualQ ID, IMHO that needs to be fixed.
[BB] Which text in the aqm-dualq-coupled draft gives this wrong 
impression? It clearly says:

       For the
       public Internet an L4S transport has to comply with the
       requirements in Section 4 of [I-D.ietf-tsvwg-ecn-l4s-id  <https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10#ref-I-D.ietf-tsvwg-ecn-l4s-id>] (aka.
       the 'Prague L4S requirements').


That clearly does not promise that the dualQ will behave as intended for 
a transport in the L queue that is RTT-dependent, i.e. that does not 
satisfy the stated requirements for that queue.

Whatever, I notice that Appendix C assumes an RTT-dependent L4S 
transport, which will need to be changed once we have empirical results 
for RTT-independent transports. And that assumption needs to be called 
out in the interim. I've made a note-to-self to do both these.

> I do not believe that the not yet solved RTT-independence goal is the best approach to solve the issue, but you are free to spend your time as you please.
It would help if you argued against the fairly straightforward reasoning 
(above, starting "So why is it enough for..."), rather than just 
generally saying you don't believe it.

>
>
>> Instead you could have read all the papers referenced from the L4S landing page, \\
>
> 	[SM} Regrettably, I have read quite a lot of that material, and I typically see too much hype and too little critical analysis for my taste.
>
>> understood how much everyone already understands the problem,
> 	[SM] I have no beef with your understanding of the issues, my complaint is that you do not act upon long-standing well-documented show-stopper issues. Not every documented bug can be dressed as a feature.
[BB] When you consider something to be an important bug, but I argue it 
is an important feature, the proper approach is to try to understand why 
something you thought was an accepted truth is being questioned.

Instead, what you've typed is effectively an accusation that I am a 
charlatan with evil intent. This frequent descent into personal slurs 
does not move anything forwards.

Think of all the material about FQ-CoDel. Is that hype, or is it 
enthusiasm about good results? To a sceptical reader who has not 
repeated the results themselves, I'm sure it seems like extreme hype.

I would ask you to presume the L4S results are all genuine until proven 
otherwise (of course, you should also hold a degree of scepticism in 
reserve). Then perhaps you would read enthusiasm as enthusiasm, rather 
than as hype.

(Nonetheless, I am making an editorial pass through the drafts damping 
down the enthusiasm/hype.)

Regards



Bob

{Note 1}: These arguments for allowing rate inequality are developed 
properly in "Per-Flow Scheduling and the End-to-End Argument 
<http://bobbriscoe.net/pubs.html#FQ_v_E2E>". I wrote it this summer 
specially for readers who believe in per-flow scheduling and who missed 
the debates in the research and IETF communities about this over the 
past dozen years.

{Note 2} I would never introduce per-flow rate controls into the network 
anyway. Because I do not want to be to blame for a future where 
high-rate applications (AR, VR, holographic, light-field, remote 
viewing, etc) have to ask permission from the network, just in order to 
maintain a certain amount of throughput in the presence of FQ with 
varying numbers of competing flows (when there is plenty of bandwidth so 
that a less than equal share is still nowhere near starvation). I do not 
think proponents of per-flow scheduling want such a politically 
distasteful shift of control towards network operators either, so I 
don't know why they choose not to see that they are pushing the Internet 
in this direction.

The main reason the early L4S proponents sunk years into developing the 
dualQ coupled AQM was to provide a way to do low latency without FQ.




>
> Best Regards
> 	Sebastian
>
>
>> how much has already been done to solve it, and then you could have tried to work out the details of a full solution.
>>
>>
>> Regards
>>
>>
>>
>> Bob
>>
>>
>>>
>>> Best Regards
>>> 	Sebastian
>>>
>>>
>>>
>>>>   
>>>> Regards,
>>>> Koen.
>>>>   
>>>> From: Ingemar Johansson S
>>>> <[e-mail]>
>>>>   
>>>> Sent: Wednesday, November 20, 2019 11:50 AM
>>>> To: Kyle Rose
>>>> <[e-mail]>; Ingemar Johansson S <[e-mail]>
>>>>
>>>> Cc: Sebastian Moeller
>>>> <[e-mail]>; G Fairhurst <[e-mail]>; [e-mail]; [e-mail]; De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]>; [e-mail]; Ingemar Johansson S <[e-mail]>
>>>>
>>>> Subject: RE: [tsvwg] L4S vs SCE
>>>>   
>>>> Hi
>>>>   
>>>> So given the imagined outcome that L4S fails.. two scenarios
>>>>   
>>>> If other SDOs or developers don’t pick up L4S then things are quite simple I guess, just declare the L4S drafts as deprecated, or is there more to do ?
>>>>   
>>>> But, if e.g. 3GPP somehow thinks that this is a good idea and adopts it.. Will the IETF send a message (LS?) to 3GPP with the message “please stop using L4S”, is this even a reasonable scenario?. After all, the fact that it is picked up by other SDOs, speaks against a failure ?
>>>>   
>>>> /Ingemar
>>>>   
>>>> From: Kyle Rose
>>>> <[e-mail]>
>>>>   
>>>> Sent: den 20 november 2019 11:14
>>>> To: Ingemar Johansson S
>>>> <ingemar.s.johansson=[e-mail]>
>>>>
>>>> Cc: Sebastian Moeller
>>>> <[e-mail]>; G Fairhurst <[e-mail]>; [e-mail]; Ingemar Johansson S <[e-mail]>; [e-mail]; De Schepper, Koen (Koen) <[e-mail]>; [e-mail]
>>>>
>>>> Subject: Re: [tsvwg] L4S vs SCE
>>>>   
>>>> On Wed, Nov 20, 2019 at 6:04 PM Ingemar Johansson S
>>>> <ingemar.s.johansson=[e-mail]>
>>>>   wrote:
>>>>
>>>>>        How do you expect an industry/commercial roll-out of L4S
>>>>> technology to behave, if the L4S experiment should terminated without
>>>>> adoption as a standard track RFC? Are they supposed to phase-out using
>>>>> ECT(1) as well, or is it understood that deployed L4S instances continue using
>>>>> L4S methods?
>>>>>
>>>> [IJ] The premise would be that L4S is declared a failure. I doubt that anybody has a good enough crystal ball to predict what happens. First it is necessary to come to the conclusion that L4S is a failure and I would argue that we are not yet there and I don't currently see that coming. Before that possible event I don't see it meaningful to speculate.
>>>>   
>>>> I'm going to have to go ahead and disagree with you strongly here. Given the potential consequences of cleaning up after a failed experiment without a plan worked out beforehand, this blithe approach is simply not acceptable.
>>>>   
>>>> In lots of cases, experiments are easy to terminate in an obvious way: for example, in one typical case, a code point can simply be abandoned, or (even better) a pollutable experimental code point returned to the available pool after some time. If that were the case here, it would not be difficult to enumerate a sequence of steps required to do so. It doesn't appear that's the case, however, so all the more reason to make sure we address this as part of the experiment setup.
>>>>   
>>>> A launch escape system of the necessary complexity should be a requirement of any experimental deployment. In this case, that might circumscribe the scope of the experiment itself.
>>>>   
>>>> Kyle
>>>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoe
>> http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/