Re: [tsvwg] path forward on L4S issue #16

[Ingemar Johansson S <[e-mail]>]

OK, thanks. 
I believe that I will now close this thread from my side.

/Ingemar

> -----Original Message-----
> From: Sebastian Moeller <[e-mail]>
> Sent: den 11 juni 2020 13:56
> To: Ingemar Johansson S <[e-mail]>
> Cc: Jonathan Morton <[e-mail]>om>; [e-mail]
> Subject: Re: [tsvwg] path forward on L4S issue #16
> 
> Hi Ingemar,
> 
> 
> 
> > On Jun 11, 2020, at 13:07, Ingemar Johansson S
> <[e-mail]> wrote:
> >
> > Hi
> > Inline...
> > /Ingemar
> >
> >> -----Original Message-----
> >> From: Sebastian Moeller <[e-mail]>
> >> Sent: den 11 juni 2020 12:40
> >> To: Ingemar Johansson S <[e-mail]>
> >> Cc: Jonathan Morton <[e-mail]>om>; [e-mail]
> >> Subject: Re: [tsvwg] path forward on L4S issue #16
> >>
> >> Hi Ingemar,
> >>
> >> as usual more below in-line.
> >>
> >>> On Jun 11, 2020, at 10:23, Ingemar Johansson S
> >> <[e-mail]> wrote:
> >>>
> >>> Hi
> >>>
> >>> Please see inline [IJ]
> >>> /Ingemar
> >>>
> >>>> -----Original Message-----
> >>>> From: Sebastian Moeller <[e-mail]>
> >>>> Sent: den 10 juni 2020 22:05
> >>>> To: Ingemar Johansson S <[e-mail]>
> >>>> Cc: Jonathan Morton <[e-mail]>om>; [e-mail]
> >>>> Subject: Re: [tsvwg] path forward on L4S issue #16
> >>>>
> >>>> Hi Ingemar,
> >>>>
> >>>> more below in-line.
> >>>>
> >>>>> On Jun 10, 2020, at 19:15, Ingemar Johansson S
> >>>> <[e-mail]> wrote:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> As regards to dualQ (below), do you see any specific reason why it
> >>>>> would not be possible to upstream (complexity/memory whatever) it
> >>>>> or is your argument that it is just not done yet ?
> >>>>
> >>>> 	[SM] Caveat: I am not involved in the Linux kernel and hence can
> >>>> not predict with any certainty dualQ's likelihood of getting
included.
> >>>> BUT as
> >>> you
> >>>> know I am arguing for some time now, that there is a discrepancy
> >>>> between the dualQ claims and its actual performance. In other words
> >>>> it is not
> >>> done,
> >>>> and given the responses from the L4S team to this issue (like the
> >>>> 15ms
> >>> hack
> >>>> in TCP Prague's CC response) I am more and more of the opinion,
> >>>> that this
> >>> is
> >>>> not simply a case of dualQ just not being finished, but rather that
> >>>> the
> >>> current
> >>>> dualQ design simply can not meet its promises and someone would
> >>>> need to go back to the drawing board.
> >>>
> >>> [IJ] Yes, and understand that there are widely differing views about
> >>> the severity of this possible problem, if it really is a problem.
> >>
> >> 	[SM] Sorry, not accepting that starving nonECT(1) flows at short
> > RTTs
> >> is a severe problem seems to be a position not even team L4S took. It
> >> has been demonstrated from early on (it is visible in allmost all L4S
> >> papers
> > and in
> >> the measurements from team SCE) that the coupling heuristic that is
> >> an essential component of dualQ's design simply does not give the
> >> required isolation guarantees needed to save flows in the non-LL
> >> queue from the consequences of its ~15:1 LL:no-LL queue scheduler. At
> >> its core dualQ is a
> >> ~16:1 scheduler that by applying a number of heuristics will give
> >> rough equivalence between the queues under a number of realistic
> >> conditions, unfortunately it fails to do so under a number of
> >> different equally
> > realistic
> >> conditions. Team L4S went so far to propose an absolute ugly hack in
> >> TCP Prague to paper over this mis-design in dualQ, accepting that
> >> this is a
> > severe
> >> problem.
> >> 	So are you really taking the position that this is not really a
> > problem?
> >> Especially since, as i mentioned before, L4S really only has seen
> >> testing
> > for
> >> short RTT/few hop scenarios, and is expected to be rolled-out/used by
> >> typically close by CDNs?
> >>
> >
> > [IJ] I believe that this discussion belongs to issue #28 so I wont
> > delve into it in this thread.
> 
> 	[SM] Fair enough, now whether #28 will ever see a discussion at all
is
> a different question.
> 
> 
> 
> >
> >>
> >>
> >>>
> >>>>
> >>>>>
> >>>>> Also, do you have any comments to my three other questions, please
> >>>>> refer to earlier email in the thread for the context.
> >>>>
> >>>> 	[SM] I snipped these out of my reply since I had nothing
> >>>> meaningful to add to those.
> >>>>
> >>>>
> >>>>> 1) Do you have any public sources that confirm the numbers you
> >>>>> quote below ?. I tried to look up data on this but it surely is not
easy.
> >>>>
> >>>> 	[SM] I do not know where Jonathan's numbers come from. But
> >>>> https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-t
> >>>> cp
> >>>> -ecn-
> >>>> experience-with-enabling-ecn-on-the-internet-padma-bhooma-00 has
> >> some
> >>>> numbers from Apple, I believe Bob cited these numbers multiple
> >>>> times in
> >>> the
> >>>> past.
> >>>> Given the fact that 3gpp contains quite a lot of large carriers
> >>>> maybe that would be a forum to ask for numbers?
> >>>
> >>> [IJ] OK, that is the same reference that I have found, I was
> >>> thinking that there is perhaps something more up to date available?
> >>
> >> 	[SM] I believe the problem is, that almost nobody (except maybe for
> >> the ripe atlas project) has the required testing nodes deployed
> >> required
> > to
> >> assess the likelihood/prevalence on CE marking on the internet.
> >> Ideally
> > you
> >> have a network of widely distributed testing nodes in leaf networks
> >> and
> > try
> >> to run saturating ECN enabled transfers between them, while recording
> >> the number of CE marks per path (pair of testing nodes), nobody in
> >> academia really has such a network, but a few major carriers should
> >> be able to set something like this up easily, given the big names in
> >> support of L4S, I
> > would
> >> find it marvelous if these could cooperate to get the CE measurements
> > done,
> >> before proceeding with the L4S drafts.
> >>
> >>
> >>> As regards to 3GPP
> >>> access then I can very safely say that we can stop worrying, there
> >>> is AFAIK no ECN marking (RFC3168 or anything else) in 3GPP base
> >>> station nodes so there are no interop concerns here. There is a
> >>> remote possibility that ECN marking can be enabled in e.g.
> >>> industrial Cisco LTE routers I tried this a while back but had
> >>> problems with ECN bleaching which is a configuration matter in 3GPP
> networks.
> >>
> >> 	[SM] I meant asking the 3GPP members to help in setting up an
> >> internet wide trawl for CE marked packets, for this leaf/edge or
> >> close to
> > the
> >> leaf/edge mesurement nodes seem required, and hence
> >> operators/carrieres that connect end-users would be in a good
> >> position to measure CE mark probabilities, no?
> >
> > [IJ] OK, my misunderstanding. I leave this uncommented as I don't
> > personally know of any such activities.
> 
> 	[SM] Nor do I, but if exact numbers of rfc3168 AQMs in the internet
> would be a potential major hurdle for my project, and I had major
> ISPs/mobile carriers in my team, this is an activity I would start...
According to
> Bob, Ericsson, Sprint, Google, Nokia Networks, AT&T, Vodafone are all
> behind L4S so getting these numbers fresh out of the internet should not
be
> an insurmountable obstacle, no?
> 
> 
> >
> >>
> >>
> >>>
> >>>>
> >>>>
> >>>>> 2) Which foras are the vendors that manufacture CPEs active in (if
> >>>>> any)
> >>> ?.
> >>>>
> >>>> 	[SM] I believe that OpenWrt certainly supports rfc3168 behaviour,
> >>>> and there are CPE that run on modified OpenWrt, so the OpenWrt
> >>>> forum might be a decent starting point?
> >>>
> >>> [IJ] OK, thanks, I was of the impression, that OpenWrt is on the
> >>> table in the implementation of fq-codel and the like, is this
> >>> correct?,
> >>
> >> 	[SM] SQM-scripts in OpenWrt allows to choose between cake and
> >> fq_codel, but cake can be run in single queue mode. QOS-scripts in
> >> OpenWrt IIRC defaults to fq_codel.
> >>
> >>
> >>> if that is
> >>> true then it perhaps fits bets with L4S issue #17 but perhaps it is
> >>> relevant for #16 and #17?.
> >>
> >> 	[SM] I believe that Pete Heist's recent data indicates that thanks
> > to
> >> the birthday paradoxon, stochastic fq as used in fq_codel and cake is
> >> not
> > a
> >> full remedy for L4S's changed response to CE-marks, so maybe that
> >> separation needs to be reverted again.
> >>
> >>> Has this extended beyond the early adopter do-it-your-self
> community?.
> >>> I recall from earlier in this long tread that you found off the
> >>> shelf routers in stores, were these shipped and ready with RFC3168
> >>> support or was an update necessary to make it RFC3168 capable?.
> >>
> >> 	[SM] There is Evenroute's IQrouter, that you can order at amazon,
> >> that runs a modified OpenWrt and does use fq (I think it defaults to
> >> cake nowadays, and cake is an rfc3168 AQM), then there is the Turris
> >> Omnia (and also the MOX I believe) that use fq_codel for their guest
> >> networks. And
> > since
> >> both of these are OpenWrt based they will default to fq_codel for all
> >> interfaces.
> >
> > [IJ] Thanks, looked up the Turris Omnia, it mentions automatic updates
> > with rationale to protect against cyber threats. I interpret this nice
> > feature as a possibility to update also for support L4S later on. This
> > is course pending interest from the developer community. But atleast
> > there are options to avoid the effects of issue #16 and #17 and avoid
> > some of the issues with legacy only RFC3168 compatible AQMs, right ?.
> 
> 	[SM] Yes, both turris and evenroute are good players that offer new
> firmwares quite often that fix issues, so are in a position to also do
this for
> L4S. Mind you that response could well be to bleach ECT(1)
unconditionally...
> Because we are putting the cart before the horse here, until it is
> demonstrated that L4S overall improves things bleaching ECT(1) seems to be
> the safer position, the onus really is on L4S as the new kid on the block,
I
> would say...
> 
> 
> > Something similar seem to apply to the Evenroute quote "Over-the-air
> > firmware updates to stay current" .
> 
> 	[SM] +1; both vendors are really good at keeping their promises
> about updates, but both do not force updates on their users, so having the
> machinery for automatic OTA updates is nice, but it will not guarantee
that all
> deployed instances actually partake in that service.
> 	And then there is the long-tail issue with OpenWrt, people on older
> less beefy routers often see to stick to oudated versions since newer
> versions require more resources and can in some cases simply not run on
old
> devices, so there will be legacy OpenWrt deployments in the field that
> require a fork-lift updata (CPE replacement) that is hard to schedule from
> outside, no? (This does not affect turris nor evenroute, as far as I
understanf
> both still support all their ever deployed devices, but that is IMHO just
a
> question of time until they will EOL/EOS).
> 
> 
> Best Regards
> 	Sebastian
> 
> 
> >
> >>
> >>
> >>>
> >>>>
> >>>>> 3) As regards to endpoints implementing RFC3168, do you refer to
> >>>>> servers and clients or something else?. My interpretation is
> >>>>> servers and clients and I don't believe that they are central  to
> >>>>> this discussion, or do I miss something ?.
> >>>>
> >>>> 	[SM] Well, it is these end-points that are going to suffer, when
> >>>> L4S gets it wrong (when, not if). So these numbers give you an
> >>>> estimate of the potential fall-out area.
> >>>
> >>> [IJ] OK, then I understand.
> >>
> >> 	[SM] Please note, that I can not speak for Jonathan, so this was
> > just
> >> my opinion on that matter.
> >>
> >>
> >>> I was a bit mislead as I assumed that we discussed network nodes
> >>> that do RFC3168 ECN marking, not end hosts that negotiate ECN. I
> >>> would like to see these as separate parts of the problem.
> >>
> >> 	[SM] Fair enough. To assess the total risk from L4S's changed CE-
> >> response however, we need to look at all these things in integrated
form.
> >
> > [IJ] Sure, but it is still important to keep these things separate,
> > after all one can alter the argument and say that end hosts can update
> > to use L4S with BBRv2 or Prague..