Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE (Evolvability))

[Ingemar Johansson S <[e-mail]>]

Hi
Please see inline

/Ingemar

> -----Original Message-----
> From: Sebastian Moeller <[e-mail]>
> Sent: den 4 januari 2020 22:38
> To: Ingemar Johansson S <[e-mail]>
> Cc: Ingemar Johansson S
> <ingemar.s.johansson=[e-mail]>
> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
> (Evolvability))
> 
> Hi Ingemar,
> 
> more below in-line
> 
> 
> > On Jan 4, 2020, at 20:10, Ingemar Johansson S
> <[e-mail]> wrote:
> >
> > Hi
> > Please see inline
> >
> > /Ingemar
> >
> >> -----Original Message-----
> >> From: Sebastian Moeller <[e-mail]>
> >> Sent: den 4 januari 2020 16:30
> >> To: Ingemar Johansson S <[e-mail]>
> >> Cc: Ingemar Johansson S
> >> <ingemar.s.johansson=[e-mail]>; [e-mail]
> >> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
> >> (Evolvability))
> >>
> >> Hi Ingemar,
> >>
> >> more below in-line.
> >>
> >>
> >>> On Jan 4, 2020, at 15:41, Ingemar Johansson S
> >> <[e-mail]> wrote:
> >>>
> >>> Hi
> >>> See inline
> >>> /Ingemar
> >>>
> >>>> -----Original Message-----
> >>>> From: Sebastian Moeller <[e-mail]>
> >>>> Sent: den 3 januari 2020 17:37
> >>>> To: Ingemar Johansson S <[e-mail]>
> >>>> Cc: Ingemar Johansson S
> >>>> <ingemar.s.johansson=[e-mail]>; Black, David
> >>>> <[e-mail]>; Bob Briscoe <[e-mail]>; Roland
> >>>> Bless <[e-mail]>; [e-mail]; [e-mail]
> >>>> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
> >>>> (Evolvability))
> >>>>
> >>>>
> >>>> Hi Ingemar,
> >>>>
> >>>> more below in-line.
> >>>>
> >>>>> On Jan 3, 2020, at 12:18, Ingemar Johansson S
> >>>> <[e-mail]> wrote:
> >>>>>
> >>>>> Hi
> >>>>> Please see inline
> >>>>> /Ingemar
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Sebastian Moeller <[e-mail]>
> >>>>>> Sent: den 3 januari 2020 10:28
> >>>>>> To: Ingemar Johansson S
> >>>>>> <ingemar.s.johansson=[e-mail]>; Black,
> David
> >>>>>> <[e-mail]>; Bob Briscoe <[e-mail]>;
> Roland
> >>>>>> Bless <[e-mail]>
> >>>>>> Cc: Ingemar Johansson S <[e-mail]>;
> >> tsvwg-
> >>>>>> [e-mail]; [e-mail]
> >>>>>> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs
> >>>>>> SCE
> >>>>>> (Evolvability))
> >>>>>>
> >>>>>> Hi Ingmar,
> >>>>>>
> >>>>>> On January 3, 2020 10:10:31 AM GMT+01:00, Ingemar Johansson S
> >>>>>> <ingemar.s.johansson=[e-mail]> wrote:
> >>>>>>> Hi
> >>>>>>> Yes I am aware that SCE works with RFC6040 tunnels, it does
> >>>>>>> however seem to work well with RFC4301 ditto.
> >>>>>>> Now I have no idea as regards to how large extent RFC4301 is
> >>>>>>> deployed on the internet and whether this is a large problem for
> >>>>>>> SCE or not. I notice however that RFC4301 is no problem for L4S.
> >>>>>>
> >>>>>>      [SM] I beg to differ, L4S will not play nice with either a
> >>>>>> rfc3168 or SCE- capable AQM on a tunneled segment, as it will
> >>>>>> completely misinterpret the received CE marks and will not
> >>>>>> respond appropriately with a sufficiently large reduction of the
> >>>>>> congestion window*. The currently still preferred solution to
> >>>>>> this issue, if I understand Bob correctly, is still to reason
> >>>>>> that such ECN-enabled AQMs along tunneled path segments (which
> I
> >>>>>> understand to be always inside the core of an AS as towards the
> >>>>>> edges packets get
> >>>>>> decapsulated) are rare enough to allow completely ignoring them.
> >>>>>
> >>>>> [IJ] This is more of a concern with traditional loss/ECN based
> >>>>> congestion control algorithms like Reno, Cubic (and DCTCP).
> >>>>
> >>>> 	[SM] How so? As far as I can tell the failure mode is, that L4S
> >>> flows will
> >>>> not yield to the rfc3168 AQMs CE marks in the way that the AQM
> >>>> expects/is designed for, thereby crowding out the 1/sqrt(p) flows
> >>>> and securing
> >>> themselves
> >>>> the lion's share of the bandwidth (assuming the hop does not need
> >>>> to dip
> >>> deep
> >>>> into dropping-instead-of-marking territory)). BBR, might bully over
> >>>> the
> >>> AQM's
> >>>> signals, but I am not sure whether that actually is an improvement
> >>>> over
> >>> the
> >>>> status quo (unless purely viewed from BBR's perspective), but that
> >>>> is a
> >>> different
> >>>> kettle of fish.
> >>>>
> >>>>> If one look more into the
> >>>>> future (and actually present) I see algorithms like BBRv2 that
> >>>>> strive one BDP queue and will therefore not strive to bloat the
> >>>>> queues as much as the forementioned CC algos.  It seems to me like
> >>>>> there is incentive to go towards more of these. And as you point
> >>>>> out below there is work ongoing work by Bob and others to add
> >>>>> RFC3168
> >>> compatibility to
> >>>> TCP Prague.
> >>>>
> >>>> 	[SM] Which albeit quite belated is a welcome development. I am
> >>> looking
> >>>> forward to the data demonstrating that approach working as intended
> >>>> and as required (I am especially interested to see how long this is
> >>>> going to take
> >>> and
> >>>> whether it introduces a "latency-bolus" in the queue). But that
> >>>> still
> >>> leaves all
> >>>> other transports the L4S folks envision classifying into the
> >>>> L4S-queue to
> >>> worry
> >>>> about. Doing that for TCP Prague is only a first mile-stone, sure
> >>>> being
> >>> proof of
> >>>> principle it might be an important mile stone, but if coexistence
> >>>> with the
> >>> existing
> >>>> internet is of serious concern (which it should) it is only the
> >>> beginning...
> >>>> 	I consider this to be a side-effect/consequence of redefining what
> >>>> a
> >>> CE-
> >>>> mark means, as the SCE proposal demonstrates that is not the only
> >>>> option
> >>> to
> >>>> implement 1/p style signaling.
> >>>>
> >>>>
> >>>>> Speaking of the SCE capable AQMs, how much does it take to make
> >> them
> >>>>> L4S capable instead, an update of RFC8290 ?
> >>>>
> >>>> 	[SM] Note that RFC8290 is not really SCE capable to begin with
> >>>> (how could it, it predates the SCE drafts considerably). Be that as
> >>>> it may,
> >>> IMHO it
> >>>> would take a) as a precondition a robust and reliable indicator of
> >>>> how a
> >>> flow will
> >>>> respond to congestion signaling, 1/p or 1/sqrt(p), unfortunately
> >>>> ECT(1) is
> >>> neither
> >>>> robust nor reliable*, and b) a consensus that overloading the ECN
> >>> CE-codepoint
> >>>> with two semantic different meanings is reliable, robust (and safe
> >>>> for the
> >>> rest of
> >>>> the network).
> >>>> 	The more relevant question for me is what it would take to remedy
> >>>> these two obvious issues in the L4S design... and what would it
> >>>> take in
> >>> addition
> >>>> to start tackling the failure of the dual queue coupled AQM to do
> >>>> the one
> >>> thing is
> >>>> was intended/designed for reliably and robustly.
> >>>
> >>> [IJ] I understand that
> >>> a) delves into the flow rate fairness that has been discussed at
> >>> length in other mail treads.
> >>
> >> 	[SM] I am puzzled, a) above is about how it mark/differentiate
> >> between flows that should be treated to 1/p versus those that should
> >> be treated to 1/sqrt(p) congestion signaling. IMHO this has exactly
> >> zero to
> > do
> >> with flow rate fairness and all with how to make the two types of
> > congestion
> >> responses coexist peacefully in the existing internet. But please
> > elaborate
> >> what you referred to above with "flow rate fairness".
> >> 	Really, the constant attacks against FQ solution are a RED HERRING
> > in
> >> the context whether L4S is ready to reach experimental stage. My
> >> issue is not that L4S might not be the theoretically best solution
> >> for the
> > challenge it
> >> was designed to solve, my issue is that practically the current
> >> implementations ARE NO SOLUTION at all, as they do not work reliably
> >> and robustly, period.
> >>
> >>
> >>>    I don't see the possible rate unfairness that DualQ gives as
> >>> unsafe
> >>
> >> 	[SM] Try the shoe on the other foot then, if the IETF would command
> >> to change the dual queue coupled AQM so that it gives the same
> >> priority/bandwidth advantage to non-L$S flows instead, would you
> >> still be okay with that? If your answer is yes, you can predict which
> >> changes to
> > dual
> >> queue coupled AQM I will propose next... I take it that you also
> >> consider
> > 15
> >> ms the ideal ref-delay for the non-L4S's queue PI2 instance without
> >> being able to tell what makes 15 so magically wonderful in that
context?
> >>
> >>
> >>> and
> >>> it is quite clear to me that
> >>>    this is something that relates more to how endpoint congestion
> >>> control algorithms are deigned
> >>>    than on the specific design of DualQ.
> >>
> >> 	[SM] Again I disagree, the dual queue coupled AQM's sole reason to
> >> exists is to equitably share between the two traffic classes the L4S
> > approach
> >> categorizes the world into. Failure to do so reliably and robustly is
> >> NOT
> > a
> >> function of the endpoints  but of your isolation mechanism (as an
> >> isolator that can be easily gamed by the endpoints does not even
> >> deserve that name).
> >> 	I realize that this is an inconvenient truth and that both Bob and
> > Koen
> >> always try to change this into an unrelated discussions about say
> >> RTT- independence, but that is simply a distraction strategy as I
> >> have alluded
> > to
> >> multiple times, not least because so far the L4S team has utterly
> >> failed
> > to
> >> present data demonstrating that they managed to make TCP Prague,
> >> their demonstration transport, RTT-independent enough to paper over
> >> the dual queue coupled AQM's design mistakes. But to be explicit,
> >> even if you manage to fix this for TCP Prague, unless you only admit
> >> TCP Prague flow
> > the
> >> the L4S queue you have won very little. If it apprears that I repeat
> > myself,
> >> then simply because part of my audience does not seem to be listening
> >> or engaging with my arguments at all.
> >>
> >>
> >>> I understand that we disagree on
> >>> this and therefore it
> >>>    is best to ask a wider group to comment this.
> >>> b) Means that a RCF3168 ECN capable flow whose packets have been CE
> >>> marked upstreams are put
> >>>   in the L4S queue in the DualQ downstreams and therefore causing
> >>> packet reordering for the RFC3168 flow.
> >>
> >> 	[SM] Exactly, by mis-classifying CE packets into the L4S queue one
> >> risks out-of-order delivery for non-L4S flows, as a direct
> >> consequence of abusing a single codepoint as binary classifier. If
> >> this single codepoint
> > would
> >> be the only option this might be worth discussing/accepting, but
> >> since
> > there
> >> are quite a number of different classifier schemes ECT(1) really
> >> should be taken off the table IMHO.
> >> 	Except, that for the reordering to happen, you do not even need
> >> packets in both directions, even in one direction reordering can be
> > inflicted
> >> upon non-L4S flows, if a CE-marked packet hits the dual queue coupled
> >> AQM it will traverse the L4S queue and will be given priority over
> >> packets of
> > the
> >> same flow in the non-L4S queue, so as long as the non-L4S queue is
> >> not empty reordering can happen. That is a side-effect most easily
> >> dealt with
> > by
> >> employing a full independent bit for proper classification, or by
> > defaulting all
> >> CE-marked packets to the non-L4S queues (that way reordering is
> >> restricted to the new L4S flows and there s a real incentive to
> >> carefully look again
> > at the
> >> unfortunate choice of 15ms ref_delay for the non-L4S queue, a win-win
> >> situation form the perspective of non-L4S traffic).
> >>
> >>
> >>>   The severity of this reordering assumes a scenario where
> >>>    i) congestion occurs simultaneously both on the upstreams and
> >>> downstreams,
> >>
> >> 	[SM] I am not sure that you need simultaneous congestion in both
> >> directions, and secondly, I believe this situation to be more
> > probable/likely
> >> than you seem to assume (this is partly a consequence of the often
> >> highly asymmetric links to end-users).
> >>
> >>>       I have so far seen this as quite unlikely
> >>>    ii) congestion occurs upstreams and the uncongested DualQ
> >>> downstreams deliberately builds a large queue,
> >>>       I have not see that DualQ does this.
> >>>   I have possibly missed something, please clarify
> >>
> >> 	As above for packet reordering you do not need simultaneous bi-
> >> directional congestion. My point however is that this again is one of
> >> the situations where the L4S team opted to make the non-L4S traffic
> >> pay all
> > the
> >> cost of the "impedance mismatch" between the current network and the
> >> "brave new world" that L4S promises.
> >
> > [IJ] I believe that you misunderstood the scenario. It is not about
> bidirectional congestion (or additional congestion in the reverse ACK
path).
> > It is only about the congestion in the forward path.
> 
> 
> 	[SM] Indeed I misread your upstream downstream references as
> denoting the general traffic direction and not simply the position in
relation
> to the L4S node. Thanks for clearing this up and persisting through my
block-
> headedness.
> 
> 
> > Consider a path between a sender and a receiver with two intermediate
> > nodes A and B, where node A is upstreams i.e closer to the sender.
> > Both nodes implement AQMs that can CE mark ECT packets. Node B
> > implements a DualQ AQM in which ECT(1) and CE marked packets are
> directed to the L4S queue.
> > This correctly means that CE marked packets for an RFC3168 flow will
> > also go into the L4S queue, which can cause packet reordering for the
> RFC3168 flow.
> 
> 	[SM] Yes, I agree.
> 
> > For this to actually happen you need any of (repeat of my last email)
> >     i) congestion occurs simultaneously both on node A and B
> >          at least it should happen within the same RTT
> >           I have so far seen this as quite unlikely
> 
> 	[SM] Currently the likelihood is close to zero, by virtue of the
lack of
> L4S AQMs in the wild, so yes this is unlikely. But I for example use an
rfc-3168
> compliant AQM on ingress and egress of my network and hence ECN-
> enabled flows (on egress) will see CE-marks when leaving slow-start and
> later in their life time (when ever they exceed their share of the link).
If an
> L4S AQM is instantiated on say a peering link (as envisioned or at least
hinted
> at by some of the L4S design descriptions) congestion on that link can and
will
> be quite decoupled from congestion and ECN-marking on my own access
> link. Can I put a number to quantify this? No, I can not, at least not any
> believable number more precise than non-zero.
> 
> 
> >     ii) congestion occurs in node A and the
> >         uncongested DualQ in node B builds a (large) queue
> >         I have not seen that DualQ does this.
> 
> 	[SM] If two (or more) packets of a single flow arrive back to back
at
> the L4S AQM with the last CE-marked, can you guarantee that that the dual
> queue coupled AQM will not reorder them even without saturating load,
> after all it is designed to give the L4S-queue (limited) priority over the
non-
> L4S queue?

[IJ] I cannot guarantee anything, my observation is however that possible
packet bursts are very likely ironed out in node A as it CE marks packets
for the RFC3168 and is therefore congested. This should mean that the queues
in node B should be empty and packets will be forwarded as they arrive.
Still there is a likelihood that packets arriving in the L4S buffer are
scheduled before packets arriving in the RFC3168 buffer and that can cause
packet reordering for the RFC3168 flow. 
The questions are :
 * How large will the reordering depth be?, my gut feeling is that this will
be in the order of sub milliseconds but I can only speculate. 
 * Is this worse than the reordering that we see already today ?
 * Is the reordering depth more than RACK can handle ?

> 
> 
> > How likely is it that any of these scenario happens ?
> 
> 	[SM] This is a very good question for which I am I hope to get an
> answer from real experimentally acquired data by the L4S proponents.
> Please note that since rfc3168 will probably stay in effect there is
simply no
> guarantee that this scenario might not become more probable in the future.


> 
> 
> Best Regards
> 	Sebastian