Re: [tsvwg] path forward on L4S issue #16

[Sebastian Moeller <[e-mail]>]

Hi Ingemar,



> On Jun 11, 2020, at 13:07, Ingemar Johansson S <[e-mail]> wrote:
> 
> Hi
> Inline...
> /Ingemar
> 
>> -----Original Message-----
>> From: Sebastian Moeller <[e-mail]>
>> Sent: den 11 juni 2020 12:40
>> To: Ingemar Johansson S <[e-mail]>
>> Cc: Jonathan Morton <[e-mail]>; [e-mail]
>> Subject: Re: [tsvwg] path forward on L4S issue #16
>> 
>> Hi Ingemar,
>> 
>> as usual more below in-line.
>> 
>>> On Jun 11, 2020, at 10:23, Ingemar Johansson S
>> <[e-mail]> wrote:
>>> 
>>> Hi
>>> 
>>> Please see inline [IJ]
>>> /Ingemar
>>> 
>>>> -----Original Message-----
>>>> From: Sebastian Moeller <[e-mail]>
>>>> Sent: den 10 juni 2020 22:05
>>>> To: Ingemar Johansson S <[e-mail]>
>>>> Cc: Jonathan Morton <[e-mail]>; [e-mail]
>>>> Subject: Re: [tsvwg] path forward on L4S issue #16
>>>> 
>>>> Hi Ingemar,
>>>> 
>>>> more below in-line.
>>>> 
>>>>> On Jun 10, 2020, at 19:15, Ingemar Johansson S
>>>> <[e-mail]> wrote:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> As regards to dualQ (below), do you see any specific reason why it
>>>>> would not be possible to upstream (complexity/memory whatever) it or
>>>>> is your argument that it is just not done yet ?
>>>> 
>>>> 	[SM] Caveat: I am not involved in the Linux kernel and hence can not
>>>> predict with any certainty dualQ's likelihood of getting included.
>>>> BUT as
>>> you
>>>> know I am arguing for some time now, that there is a discrepancy
>>>> between the dualQ claims and its actual performance. In other words
>>>> it is not
>>> done,
>>>> and given the responses from the L4S team to this issue (like the
>>>> 15ms
>>> hack
>>>> in TCP Prague's CC response) I am more and more of the opinion, that
>>>> this
>>> is
>>>> not simply a case of dualQ just not being finished, but rather that
>>>> the
>>> current
>>>> dualQ design simply can not meet its promises and someone would need
>>>> to go back to the drawing board.
>>> 
>>> [IJ] Yes, and understand that there are widely differing views about
>>> the severity of this possible problem, if it really is a problem.
>> 
>> 	[SM] Sorry, not accepting that starving nonECT(1) flows at short
> RTTs
>> is a severe problem seems to be a position not even team L4S took. It has
>> been demonstrated from early on (it is visible in allmost all L4S papers
> and in
>> the measurements from team SCE) that the coupling heuristic that is an
>> essential component of dualQ's design simply does not give the required
>> isolation guarantees needed to save flows in the non-LL queue from the
>> consequences of its ~15:1 LL:no-LL queue scheduler. At its core dualQ is a
>> ~16:1 scheduler that by applying a number of heuristics will give rough
>> equivalence between the queues under a number of realistic conditions,
>> unfortunately it fails to do so under a number of different equally
> realistic
>> conditions. Team L4S went so far to propose an absolute ugly hack in TCP
>> Prague to paper over this mis-design in dualQ, accepting that this is a
> severe
>> problem.
>> 	So are you really taking the position that this is not really a
> problem?
>> Especially since, as i mentioned before, L4S really only has seen testing
> for
>> short RTT/few hop scenarios, and is expected to be rolled-out/used by
>> typically close by CDNs?
>> 
> 
> [IJ] I believe that this discussion belongs to issue #28 so I wont delve
> into it in this thread. 

	[SM] Fair enough, now whether #28 will ever see a discussion at all is a different question.



> 
>> 
>> 
>>> 
>>>> 
>>>>> 
>>>>> Also, do you have any comments to my three other questions, please
>>>>> refer to earlier email in the thread for the context.
>>>> 
>>>> 	[SM] I snipped these out of my reply since I had nothing meaningful
>>>> to add to those.
>>>> 
>>>> 
>>>>> 1) Do you have any public sources that confirm the numbers you quote
>>>>> below ?. I tried to look up data on this but it surely is not easy.
>>>> 
>>>> 	[SM] I do not know where Jonathan's numbers come from. But
>>>> https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-tcp
>>>> -ecn-
>>>> experience-with-enabling-ecn-on-the-internet-padma-bhooma-00 has
>> some
>>>> numbers from Apple, I believe Bob cited these numbers multiple times
>>>> in
>>> the
>>>> past.
>>>> Given the fact that 3gpp contains quite a lot of large carriers maybe
>>>> that would be a forum to ask for numbers?
>>> 
>>> [IJ] OK, that is the same reference that I have found, I was thinking
>>> that there is perhaps something more up to date available?
>> 
>> 	[SM] I believe the problem is, that almost nobody (except maybe for
>> the ripe atlas project) has the required testing nodes deployed required
> to
>> assess the likelihood/prevalence on CE marking on the internet. Ideally
> you
>> have a network of widely distributed testing nodes in leaf networks and
> try
>> to run saturating ECN enabled transfers between them, while recording the
>> number of CE marks per path (pair of testing nodes), nobody in academia
>> really has such a network, but a few major carriers should be able to set
>> something like this up easily, given the big names in support of L4S, I
> would
>> find it marvelous if these could cooperate to get the CE measurements
> done,
>> before proceeding with the L4S drafts.
>> 
>> 
>>> As regards to 3GPP
>>> access then I can very safely say that we can stop worrying, there is
>>> AFAIK no ECN marking (RFC3168 or anything else) in 3GPP base station
>>> nodes so there are no interop concerns here. There is a remote
>>> possibility that ECN marking can be enabled in e.g. industrial Cisco
>>> LTE routers I tried this a while back but had problems with ECN
>>> bleaching which is a configuration matter in 3GPP networks.
>> 
>> 	[SM] I meant asking the 3GPP members to help in setting up an
>> internet wide trawl for CE marked packets, for this leaf/edge or close to
> the
>> leaf/edge mesurement nodes seem required, and hence operators/carrieres
>> that connect end-users would be in a good position to measure CE mark
>> probabilities, no?
> 
> [IJ] OK, my misunderstanding. I leave this uncommented as I don't personally
> know of any such activities. 

	[SM] Nor do I, but if exact numbers of rfc3168 AQMs in the internet would be a potential major hurdle for my project, and I had major ISPs/mobile carriers in my team, this is an activity I would start... According to Bob, Ericsson, Sprint, Google, Nokia Networks, AT&T, Vodafone are all behind L4S so getting these numbers fresh out of the internet should not be an insurmountable obstacle, no?


> 
>> 
>> 
>>> 
>>>> 
>>>> 
>>>>> 2) Which foras are the vendors that manufacture CPEs active in (if
>>>>> any)
>>> ?.
>>>> 
>>>> 	[SM] I believe that OpenWrt certainly supports rfc3168 behaviour,
>>>> and there are CPE that run on modified OpenWrt, so the OpenWrt forum
>>>> might be a decent starting point?
>>> 
>>> [IJ] OK, thanks, I was of the impression, that OpenWrt is on the table
>>> in the implementation of fq-codel and the like, is this correct?,
>> 
>> 	[SM] SQM-scripts in OpenWrt allows to choose between cake and
>> fq_codel, but cake can be run in single queue mode. QOS-scripts in OpenWrt
>> IIRC defaults to fq_codel.
>> 
>> 
>>> if that is
>>> true then it perhaps fits bets with L4S issue #17 but perhaps it is
>>> relevant for #16 and #17?.
>> 
>> 	[SM] I believe that Pete Heist's recent data indicates that thanks
> to
>> the birthday paradoxon, stochastic fq as used in fq_codel and cake is not
> a
>> full remedy for L4S's changed response to CE-marks, so maybe that
>> separation needs to be reverted again.
>> 
>>> Has this extended beyond the early adopter do-it-your-self community?.
>>> I recall from earlier in this long tread that you found off the shelf
>>> routers in stores, were these shipped and ready with RFC3168 support
>>> or was an update necessary to make it RFC3168 capable?.
>> 
>> 	[SM] There is Evenroute's IQrouter, that you can order at amazon,
>> that runs a modified OpenWrt and does use fq (I think it defaults to cake
>> nowadays, and cake is an rfc3168 AQM), then there is the Turris Omnia (and
>> also the MOX I believe) that use fq_codel for their guest networks. And
> since
>> both of these are OpenWrt based they will default to fq_codel for all
>> interfaces.
> 
> [IJ] Thanks, looked up the Turris Omnia, it mentions automatic updates with
> rationale to protect against cyber threats. I interpret this nice feature as
> a possibility to update also for support L4S later on. This is course
> pending interest from the developer community. But atleast there are options
> to avoid the effects of issue #16 and #17 and avoid some of the issues with
> legacy only RFC3168 compatible AQMs, right ?.

	[SM] Yes, both turris and evenroute are good players that offer new firmwares quite often that fix issues, so are in a position to also do this for L4S. Mind you that response could well be to bleach ECT(1) unconditionally... Because we are putting the cart before the horse here, until it is demonstrated that L4S overall improves things bleaching ECT(1) seems to be the safer position, the onus really is on L4S as the new kid on the block, I would say...


> Something similar seem to apply to the Evenroute quote "Over-the-air
> firmware updates to stay current" .

	[SM] +1; both vendors are really good at keeping their promises about updates, but both do not force updates on their users, so having the machinery for automatic OTA updates is nice, but it will not guarantee that all deployed instances actually partake in that service.
	And then there is the long-tail issue with OpenWrt, people on older less beefy routers often see to stick to oudated versions since newer versions require more resources and can in some cases simply not run on old devices, so there will be legacy OpenWrt deployments in the field that require a fork-lift updata (CPE replacement) that is hard to schedule from outside, no? (This does not affect turris nor evenroute, as far as I understanf both still support all their ever deployed devices, but that is IMHO just a question of time until they will EOL/EOS).


Best Regards
	Sebastian


> 
>> 
>> 
>>> 
>>>> 
>>>>> 3) As regards to endpoints implementing RFC3168, do you refer to
>>>>> servers and clients or something else?. My interpretation is servers
>>>>> and clients and I don't believe that they are central  to this
>>>>> discussion, or do I miss something ?.
>>>> 
>>>> 	[SM] Well, it is these end-points that are going to suffer, when L4S
>>>> gets it wrong (when, not if). So these numbers give you an estimate
>>>> of the potential fall-out area.
>>> 
>>> [IJ] OK, then I understand.
>> 
>> 	[SM] Please note, that I can not speak for Jonathan, so this was
> just
>> my opinion on that matter.
>> 
>> 
>>> I was a bit mislead as I assumed that we discussed network nodes that
>>> do RFC3168 ECN marking, not end hosts that negotiate ECN. I would like
>>> to see these as separate parts of the problem.
>> 
>> 	[SM] Fair enough. To assess the total risk from L4S's changed CE-
>> response however, we need to look at all these things in integrated form.
> 
> [IJ] Sure, but it is still important to keep these things separate, after
> all one can alter the argument and say that end hosts can update to use L4S
> with BBRv2 or Prague..