Re: [tsvwg] path forward on L4S issue #16

[Sebastian Moeller <[e-mail]>]

Hi Ingemar,

as usual more below in-line.

> On Jun 11, 2020, at 10:23, Ingemar Johansson S <[e-mail]> wrote:
> 
> Hi
> 
> Please see inline [IJ]
> /Ingemar
> 
>> -----Original Message-----
>> From: Sebastian Moeller <[e-mail]>
>> Sent: den 10 juni 2020 22:05
>> To: Ingemar Johansson S <[e-mail]>
>> Cc: Jonathan Morton <[e-mail]>; [e-mail]
>> Subject: Re: [tsvwg] path forward on L4S issue #16
>> 
>> Hi Ingemar,
>> 
>> more below in-line.
>> 
>>> On Jun 10, 2020, at 19:15, Ingemar Johansson S
>> <[e-mail]> wrote:
>>> 
>>> Hi
>>> 
>>> As regards to dualQ (below), do you see any specific reason why it
>>> would not be possible to upstream (complexity/memory whatever) it or
>>> is your argument that it is just not done yet ?
>> 
>> 	[SM] Caveat: I am not involved in the Linux kernel and hence can not
>> predict with any certainty dualQ's likelihood of getting included. BUT as
> you
>> know I am arguing for some time now, that there is a discrepancy between
>> the dualQ claims and its actual performance. In other words it is not
> done,
>> and given the responses from the L4S team to this issue (like the 15ms
> hack
>> in TCP Prague's CC response) I am more and more of the opinion, that this
> is
>> not simply a case of dualQ just not being finished, but rather that the
> current
>> dualQ design simply can not meet its promises and someone would need to
>> go back to the drawing board.
> 
> [IJ] Yes, and understand that there are widely differing views about the
> severity of this possible problem, if it really is a problem.

	[SM] Sorry, not accepting that starving nonECT(1) flows at short RTTs is a severe problem seems to be a position not even team L4S took. It has been demonstrated from early on (it is visible in allmost all L4S papers and in the measurements from team SCE) that the coupling heuristic that is an essential component of dualQ's design simply does not give the required isolation guarantees needed to save flows in the non-LL queue from the consequences of its ~15:1 LL:no-LL queue scheduler. At its core dualQ is a ~16:1 scheduler that by applying a number of heuristics will give rough equivalence between the queues under a number of realistic conditions, unfortunately it fails to do so under a number of different equally realistic conditions. Team L4S went so far to propose an absolute ugly hack in TCP Prague to paper over this mis-design in dualQ, accepting that this is a severe problem.
	So are you really taking the position that this is not really a problem? Especially since, as i mentioned before, L4S really only has seen testing for short RTT/few hop scenarios, and is expected to be rolled-out/used by typically close by CDNs?


> 
>> 
>>> 
>>> Also, do you have any comments to my three other questions, please
>>> refer to earlier email in the thread for the context.
>> 
>> 	[SM] I snipped these out of my reply since I had nothing meaningful
>> to add to those.
>> 
>> 
>>> 1) Do you have any public sources that confirm the numbers you quote
>>> below ?. I tried to look up data on this but it surely is not easy.
>> 
>> 	[SM] I do not know where Jonathan's numbers come from. But
>> https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-tcp-ecn-
>> experience-with-enabling-ecn-on-the-internet-padma-bhooma-00 has some
>> numbers from Apple, I believe Bob cited these numbers multiple times in
> the
>> past.
>> Given the fact that 3gpp contains quite a lot of large carriers maybe that
>> would be a forum to ask for numbers?
> 
> [IJ] OK, that is the same reference that I have found, I was thinking that
> there is perhaps something more up to date available?

	[SM] I believe the problem is, that almost nobody (except maybe for the ripe atlas project) has the required testing nodes deployed required to assess the likelihood/prevalence on CE marking on the internet. Ideally you have a network of widely distributed testing nodes in leaf networks and try to run saturating ECN enabled transfers between them, while recording the number of CE marks per path (pair of testing nodes), nobody in academia really has such a network, but a few major carriers should be able to set something like this up easily, given the big names in support of L4S, I would find it marvelous if these could cooperate to get the CE measurements done, before proceeding with the L4S drafts. 


> As regards to 3GPP
> access then I can very safely say that we can stop worrying, there is AFAIK
> no ECN marking (RFC3168 or anything else) in 3GPP base station nodes so
> there are no interop concerns here. There is a remote possibility that ECN
> marking can be enabled in e.g. industrial Cisco LTE routers I tried this a
> while back but had problems with ECN bleaching which is a configuration
> matter in 3GPP networks. 

	[SM] I meant asking the 3GPP members to help in setting up an internet wide trawl for CE marked packets, for this leaf/edge or close to the leaf/edge mesurement nodes seem required, and hence operators/carrieres that connect end-users would be in a good position to measure CE mark probabilities, no?


> 
>> 
>> 
>>> 2) Which foras are the vendors that manufacture CPEs active in (if any)
> ?.
>> 
>> 	[SM] I believe that OpenWrt certainly supports rfc3168 behaviour,
>> and there are CPE that run on modified OpenWrt, so the OpenWrt forum
>> might be a decent starting point?
> 
> [IJ] OK, thanks, I was of the impression, that OpenWrt is on the table in
> the implementation of fq-codel and the like, is this correct?,

	[SM] SQM-scripts in OpenWrt allows to choose between cake and fq_codel, but cake can be run in single queue mode. QOS-scripts in OpenWrt IIRC defaults to fq_codel.


> if that is
> true then it perhaps fits bets with L4S issue #17 but perhaps it is relevant
> for #16 and #17?. 

	[SM] I believe that Pete Heist's recent data indicates that thanks to the birthday paradoxon, stochastic fq as used in fq_codel and cake is not a full remedy for L4S's changed response to CE-marks, so maybe that separation needs to be reverted again.

> Has this extended beyond the early adopter do-it-your-self community?. I
> recall from earlier in this long tread that you found off the shelf routers
> in stores, were these shipped and ready with RFC3168 support or was an
> update necessary to make it RFC3168 capable?.

	[SM] There is Evenroute's IQrouter, that you can order at amazon, that runs a modified OpenWrt and does use fq (I think it defaults to cake nowadays, and cake is an rfc3168 AQM), then there is the Turris Omnia (and also the MOX I believe) that use fq_codel for their guest networks. And since both of these are OpenWrt based they will default to fq_codel for all interfaces.


> 
>> 
>>> 3) As regards to endpoints implementing RFC3168, do you refer to
>>> servers and clients or something else?. My interpretation is servers
>>> and clients and I don't believe that they are central  to this
>>> discussion, or do I miss something ?.
>> 
>> 	[SM] Well, it is these end-points that are going to suffer, when L4S
>> gets it wrong (when, not if). So these numbers give you an estimate of the
>> potential fall-out area.
> 
> [IJ] OK, then I understand.

	[SM] Please note, that I can not speak for Jonathan, so this was just my opinion on that matter.


> I was a bit mislead as I assumed that we
> discussed network nodes that do RFC3168 ECN marking, not end hosts that
> negotiate ECN. I would like to see these as separate parts of the problem. 

	[SM] Fair enough. To assess the total risk from L4S's changed CE-response however, we need to look at all these things in integrated form.

Best Regards
	Sebastian


> 
>> 
>> Best Regards
>> 	Sebastian
>> 
>> 
>>> 
>>> /Ingemar
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: Sebastian Moeller <[e-mail]>
>>>> Sent: den 10 juni 2020 16:35
>>>> To: Ingemar Johansson S <[e-mail]>
>>>> Cc: Jonathan Morton <[e-mail]>; [e-mail]
>>>> Subject: Re: [tsvwg] path forward on L4S issue #16
>>>> 
>>>> Hi Ingemar,
>>>> 
>>>> to gently push back on some details.
>>>> 
>>>>> On Jun 10, 2020, at 15:59, Ingemar Johansson S
>>>> <[e-mail]> wrote:
>>>>> 
>>>>> [...]I understand that traffic shaping on outgoing interfaces can be
>>>>> applied in a Linux host but don't see why they become a problem
>>>>> especially as there are qdiscs that support dualQ.
>>>>> [...]
>>>> 
>>>> 	There seems to be a single out-of-the-mainline-Linux-tree repository
>>>> (https://protect2.fireeye.com/v1/url?k=e33cc533-bd9c7f5d-e33c85a8-
>>>> 869a14f4b08c-0ec6a27e7722e722&q=1&e=29721776-06f8-43e4-a1e6-
>>>> 67f0d2c15283&u=https%3A%2F%2Fgithub.com%2FL4STeam%2Flinux) for
>> both
>>>> the dual queue coupled AQM and TCP Prague.
>>>> 	I would not call that prrof of sufficient existence of "qdiscs that
>>>> support dualQ" to allow Linux system admins to switch over to
>>>> dualqand I
>>> do
>>>> not see how even inclusion into the mainline kernel* would this
>>>> solves the issue for currently deployed Linux machines, which often
>>>> use vendor
>>> kernels
>>>> which do not necessarily track mainline closely, especially for
>>>> server distributions.
>>>> 	I would respectfully argue that for safety considerations one should
>>>> look at the current state of the internet and not potential less
>>> problematic
>>>> states one would like to find the internet in...
>>>> 
>>>> Best Regards
>>>> 	Sebastian
>>>> 
>>>> 
>>>> *) As far as I can tell there have been no attempts at upstreaming
>>>> the
>>> dual
>>>> queue coupled AQM yet, so it is not clear what/if survives the
>>>> contact
>>> with
>>>> the linux kernel maintainers.