Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE (Evolvability))

[Sebastian Moeller <[e-mail]>]

Hi Ingemar,


more below in-line.

To anybody not interested in the detailed discussion below, please ignore this e-mail, thanks.


> On Jan 5, 2020, at 10:43, Ingemar Johansson S <[e-mail]> wrote:
> 
> Hi
> Please see inline
> 
> /Ingemar
> 
>> -----Original Message-----
>> From: Sebastian Moeller <[e-mail]>
>> Sent: den 4 januari 2020 22:38
>> To: Ingemar Johansson S <[e-mail]>
>> Cc: Ingemar Johansson S
>> <ingemar.s.johansson=[e-mail]>
>> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
>> (Evolvability))
>> 
>> Hi Ingemar,
>> 
>> more below in-line
>> 
>> 
>>> On Jan 4, 2020, at 20:10, Ingemar Johansson S
>> <[e-mail]> wrote:
>>> 
>>> Hi
>>> Please see inline
>>> 
>>> /Ingemar
>>> 
>>>> -----Original Message-----
>>>> From: Sebastian Moeller <[e-mail]>
>>>> Sent: den 4 januari 2020 16:30
>>>> To: Ingemar Johansson S <[e-mail]>
>>>> Cc: Ingemar Johansson S
>>>> <ingemar.s.johansson=[e-mail]>; [e-mail]
>>>> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
>>>> (Evolvability))
>>>> 
>>>> Hi Ingemar,
>>>> 
>>>> more below in-line.
>>>> 
>>>> 
>>>>> On Jan 4, 2020, at 15:41, Ingemar Johansson S
>>>> <[e-mail]> wrote:
>>>>> 
>>>>> Hi
>>>>> See inline
>>>>> /Ingemar
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Sebastian Moeller <[e-mail]>
>>>>>> Sent: den 3 januari 2020 17:37
>>>>>> To: Ingemar Johansson S <[e-mail]>
>>>>>> Cc: Ingemar Johansson S
>>>>>> <ingemar.s.johansson=[e-mail]>; Black, David
>>>>>> <[e-mail]>; Bob Briscoe <[e-mail]>; Roland
>>>>>> Bless <[e-mail]>; [e-mail]; [e-mail]
>>>>>> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
>>>>>> (Evolvability))
>>>>>> 
>>>>>> 
>>>>>> Hi Ingemar,
>>>>>> 
>>>>>> more below in-line.
>>>>>> 
>>>>>>> On Jan 3, 2020, at 12:18, Ingemar Johansson S
>>>>>> <[e-mail]> wrote:
>>>>>>> 
>>>>>>> Hi
>>>>>>> Please see inline
>>>>>>> /Ingemar
>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: Sebastian Moeller <[e-mail]>
>>>>>>>> Sent: den 3 januari 2020 10:28
>>>>>>>> To: Ingemar Johansson S
>>>>>>>> <ingemar.s.johansson=[e-mail]>; Black,
>> David
>>>>>>>> <[e-mail]>; Bob Briscoe <[e-mail]>;
>> Roland
>>>>>>>> Bless <[e-mail]>
>>>>>>>> Cc: Ingemar Johansson S <[e-mail]>;
>>>> tsvwg-
>>>>>>>> [e-mail]; [e-mail]
>>>>>>>> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs
>>>>>>>> SCE
>>>>>>>> (Evolvability))
>>>>>>>> 
>>>>>>>> Hi Ingmar,
>>>>>>>> 
>>>>>>>> On January 3, 2020 10:10:31 AM GMT+01:00, Ingemar Johansson S
>>>>>>>> <ingemar.s.johansson=[e-mail]> wrote:
>>>>>>>>> Hi
>>>>>>>>> Yes I am aware that SCE works with RFC6040 tunnels, it does
>>>>>>>>> however seem to work well with RFC4301 ditto.
>>>>>>>>> Now I have no idea as regards to how large extent RFC4301 is
>>>>>>>>> deployed on the internet and whether this is a large problem for
>>>>>>>>> SCE or not. I notice however that RFC4301 is no problem for L4S.
>>>>>>>> 
>>>>>>>>     [SM] I beg to differ, L4S will not play nice with either a
>>>>>>>> rfc3168 or SCE- capable AQM on a tunneled segment, as it will
>>>>>>>> completely misinterpret the received CE marks and will not
>>>>>>>> respond appropriately with a sufficiently large reduction of the
>>>>>>>> congestion window*. The currently still preferred solution to
>>>>>>>> this issue, if I understand Bob correctly, is still to reason
>>>>>>>> that such ECN-enabled AQMs along tunneled path segments (which
>> I
>>>>>>>> understand to be always inside the core of an AS as towards the
>>>>>>>> edges packets get
>>>>>>>> decapsulated) are rare enough to allow completely ignoring them.
>>>>>>> 
>>>>>>> [IJ] This is more of a concern with traditional loss/ECN based
>>>>>>> congestion control algorithms like Reno, Cubic (and DCTCP).
>>>>>> 
>>>>>> 	[SM] How so? As far as I can tell the failure mode is, that L4S
>>>>> flows will
>>>>>> not yield to the rfc3168 AQMs CE marks in the way that the AQM
>>>>>> expects/is designed for, thereby crowding out the 1/sqrt(p) flows
>>>>>> and securing
>>>>> themselves
>>>>>> the lion's share of the bandwidth (assuming the hop does not need
>>>>>> to dip
>>>>> deep
>>>>>> into dropping-instead-of-marking territory)). BBR, might bully over
>>>>>> the
>>>>> AQM's
>>>>>> signals, but I am not sure whether that actually is an improvement
>>>>>> over
>>>>> the
>>>>>> status quo (unless purely viewed from BBR's perspective), but that
>>>>>> is a
>>>>> different
>>>>>> kettle of fish.
>>>>>> 
>>>>>>> If one look more into the
>>>>>>> future (and actually present) I see algorithms like BBRv2 that
>>>>>>> strive one BDP queue and will therefore not strive to bloat the
>>>>>>> queues as much as the forementioned CC algos.  It seems to me like
>>>>>>> there is incentive to go towards more of these. And as you point
>>>>>>> out below there is work ongoing work by Bob and others to add
>>>>>>> RFC3168
>>>>> compatibility to
>>>>>> TCP Prague.
>>>>>> 
>>>>>> 	[SM] Which albeit quite belated is a welcome development. I am
>>>>> looking
>>>>>> forward to the data demonstrating that approach working as intended
>>>>>> and as required (I am especially interested to see how long this is
>>>>>> going to take
>>>>> and
>>>>>> whether it introduces a "latency-bolus" in the queue). But that
>>>>>> still
>>>>> leaves all
>>>>>> other transports the L4S folks envision classifying into the
>>>>>> L4S-queue to
>>>>> worry
>>>>>> about. Doing that for TCP Prague is only a first mile-stone, sure
>>>>>> being
>>>>> proof of
>>>>>> principle it might be an important mile stone, but if coexistence
>>>>>> with the
>>>>> existing
>>>>>> internet is of serious concern (which it should) it is only the
>>>>> beginning...
>>>>>> 	I consider this to be a side-effect/consequence of redefining what
>>>>>> a
>>>>> CE-
>>>>>> mark means, as the SCE proposal demonstrates that is not the only
>>>>>> option
>>>>> to
>>>>>> implement 1/p style signaling.
>>>>>> 
>>>>>> 
>>>>>>> Speaking of the SCE capable AQMs, how much does it take to make
>>>> them
>>>>>>> L4S capable instead, an update of RFC8290 ?
>>>>>> 
>>>>>> 	[SM] Note that RFC8290 is not really SCE capable to begin with
>>>>>> (how could it, it predates the SCE drafts considerably). Be that as
>>>>>> it may,
>>>>> IMHO it
>>>>>> would take a) as a precondition a robust and reliable indicator of
>>>>>> how a
>>>>> flow will
>>>>>> respond to congestion signaling, 1/p or 1/sqrt(p), unfortunately
>>>>>> ECT(1) is
>>>>> neither
>>>>>> robust nor reliable*, and b) a consensus that overloading the ECN
>>>>> CE-codepoint
>>>>>> with two semantic different meanings is reliable, robust (and safe
>>>>>> for the
>>>>> rest of
>>>>>> the network).
>>>>>> 	The more relevant question for me is what it would take to remedy
>>>>>> these two obvious issues in the L4S design... and what would it
>>>>>> take in
>>>>> addition
>>>>>> to start tackling the failure of the dual queue coupled AQM to do
>>>>>> the one
>>>>> thing is
>>>>>> was intended/designed for reliably and robustly.
>>>>> 
>>>>> [IJ] I understand that
>>>>> a) delves into the flow rate fairness that has been discussed at
>>>>> length in other mail treads.
>>>> 
>>>> 	[SM] I am puzzled, a) above is about how it mark/differentiate
>>>> between flows that should be treated to 1/p versus those that should
>>>> be treated to 1/sqrt(p) congestion signaling. IMHO this has exactly
>>>> zero to
>>> do
>>>> with flow rate fairness and all with how to make the two types of
>>> congestion
>>>> responses coexist peacefully in the existing internet. But please
>>> elaborate
>>>> what you referred to above with "flow rate fairness".
>>>> 	Really, the constant attacks against FQ solution are a RED HERRING
>>> in
>>>> the context whether L4S is ready to reach experimental stage. My
>>>> issue is not that L4S might not be the theoretically best solution
>>>> for the
>>> challenge it
>>>> was designed to solve, my issue is that practically the current
>>>> implementations ARE NO SOLUTION at all, as they do not work reliably
>>>> and robustly, period.
>>>> 
>>>> 
>>>>>   I don't see the possible rate unfairness that DualQ gives as
>>>>> unsafe
>>>> 
>>>> 	[SM] Try the shoe on the other foot then, if the IETF would command
>>>> to change the dual queue coupled AQM so that it gives the same
>>>> priority/bandwidth advantage to non-L$S flows instead, would you
>>>> still be okay with that? If your answer is yes, you can predict which
>>>> changes to
>>> dual
>>>> queue coupled AQM I will propose next... I take it that you also
>>>> consider
>>> 15
>>>> ms the ideal ref-delay for the non-L4S's queue PI2 instance without
>>>> being able to tell what makes 15 so magically wonderful in that
> context?
>>>> 
>>>> 
>>>>> and
>>>>> it is quite clear to me that
>>>>>   this is something that relates more to how endpoint congestion
>>>>> control algorithms are deigned
>>>>>   than on the specific design of DualQ.
>>>> 
>>>> 	[SM] Again I disagree, the dual queue coupled AQM's sole reason to
>>>> exists is to equitably share between the two traffic classes the L4S
>>> approach
>>>> categorizes the world into. Failure to do so reliably and robustly is
>>>> NOT
>>> a
>>>> function of the endpoints  but of your isolation mechanism (as an
>>>> isolator that can be easily gamed by the endpoints does not even
>>>> deserve that name).
>>>> 	I realize that this is an inconvenient truth and that both Bob and
>>> Koen
>>>> always try to change this into an unrelated discussions about say
>>>> RTT- independence, but that is simply a distraction strategy as I
>>>> have alluded
>>> to
>>>> multiple times, not least because so far the L4S team has utterly
>>>> failed
>>> to
>>>> present data demonstrating that they managed to make TCP Prague,
>>>> their demonstration transport, RTT-independent enough to paper over
>>>> the dual queue coupled AQM's design mistakes. But to be explicit,
>>>> even if you manage to fix this for TCP Prague, unless you only admit
>>>> TCP Prague flow
>>> the
>>>> the L4S queue you have won very little. If it apprears that I repeat
>>> myself,
>>>> then simply because part of my audience does not seem to be listening
>>>> or engaging with my arguments at all.
>>>> 
>>>> 
>>>>> I understand that we disagree on
>>>>> this and therefore it
>>>>>   is best to ask a wider group to comment this.
>>>>> b) Means that a RCF3168 ECN capable flow whose packets have been CE
>>>>> marked upstreams are put
>>>>>  in the L4S queue in the DualQ downstreams and therefore causing
>>>>> packet reordering for the RFC3168 flow.
>>>> 
>>>> 	[SM] Exactly, by mis-classifying CE packets into the L4S queue one
>>>> risks out-of-order delivery for non-L4S flows, as a direct
>>>> consequence of abusing a single codepoint as binary classifier. If
>>>> this single codepoint
>>> would
>>>> be the only option this might be worth discussing/accepting, but
>>>> since
>>> there
>>>> are quite a number of different classifier schemes ECT(1) really
>>>> should be taken off the table IMHO.
>>>> 	Except, that for the reordering to happen, you do not even need
>>>> packets in both directions, even in one direction reordering can be
>>> inflicted
>>>> upon non-L4S flows, if a CE-marked packet hits the dual queue coupled
>>>> AQM it will traverse the L4S queue and will be given priority over
>>>> packets of
>>> the
>>>> same flow in the non-L4S queue, so as long as the non-L4S queue is
>>>> not empty reordering can happen. That is a side-effect most easily
>>>> dealt with
>>> by
>>>> employing a full independent bit for proper classification, or by
>>> defaulting all
>>>> CE-marked packets to the non-L4S queues (that way reordering is
>>>> restricted to the new L4S flows and there s a real incentive to
>>>> carefully look again
>>> at the
>>>> unfortunate choice of 15ms ref_delay for the non-L4S queue, a win-win
>>>> situation form the perspective of non-L4S traffic).
>>>> 
>>>> 
>>>>>  The severity of this reordering assumes a scenario where
>>>>>   i) congestion occurs simultaneously both on the upstreams and
>>>>> downstreams,
>>>> 
>>>> 	[SM] I am not sure that you need simultaneous congestion in both
>>>> directions, and secondly, I believe this situation to be more
>>> probable/likely
>>>> than you seem to assume (this is partly a consequence of the often
>>>> highly asymmetric links to end-users).
>>>> 
>>>>>      I have so far seen this as quite unlikely
>>>>>   ii) congestion occurs upstreams and the uncongested DualQ
>>>>> downstreams deliberately builds a large queue,
>>>>>      I have not see that DualQ does this.
>>>>>  I have possibly missed something, please clarify
>>>> 
>>>> 	As above for packet reordering you do not need simultaneous bi-
>>>> directional congestion. My point however is that this again is one of
>>>> the situations where the L4S team opted to make the non-L4S traffic
>>>> pay all
>>> the
>>>> cost of the "impedance mismatch" between the current network and the
>>>> "brave new world" that L4S promises.
>>> 
>>> [IJ] I believe that you misunderstood the scenario. It is not about
>> bidirectional congestion (or additional congestion in the reverse ACK
> path).
>>> It is only about the congestion in the forward path.
>> 
>> 
>> 	[SM] Indeed I misread your upstream downstream references as
>> denoting the general traffic direction and not simply the position in
> relation
>> to the L4S node. Thanks for clearing this up and persisting through my
> block-
>> headedness.
>> 
>> 
>>> Consider a path between a sender and a receiver with two intermediate
>>> nodes A and B, where node A is upstreams i.e closer to the sender.
>>> Both nodes implement AQMs that can CE mark ECT packets. Node B
>>> implements a DualQ AQM in which ECT(1) and CE marked packets are
>> directed to the L4S queue.
>>> This correctly means that CE marked packets for an RFC3168 flow will
>>> also go into the L4S queue, which can cause packet reordering for the
>> RFC3168 flow.
>> 
>> 	[SM] Yes, I agree.
>> 
>>> For this to actually happen you need any of (repeat of my last email)
>>>    i) congestion occurs simultaneously both on node A and B
>>>         at least it should happen within the same RTT
>>>          I have so far seen this as quite unlikely
>> 
>> 	[SM] Currently the likelihood is close to zero, by virtue of the
> lack of
>> L4S AQMs in the wild, so yes this is unlikely. But I for example use an
> rfc-3168
>> compliant AQM on ingress and egress of my network and hence ECN-
>> enabled flows (on egress) will see CE-marks when leaving slow-start and
>> later in their life time (when ever they exceed their share of the link).
> If an
>> L4S AQM is instantiated on say a peering link (as envisioned or at least
> hinted
>> at by some of the L4S design descriptions) congestion on that link can and
> will
>> be quite decoupled from congestion and ECN-marking on my own access
>> link. Can I put a number to quantify this? No, I can not, at least not any
>> believable number more precise than non-zero.
>> 
>> 
>>>    ii) congestion occurs in node A and the
>>>        uncongested DualQ in node B builds a (large) queue
>>>        I have not seen that DualQ does this.
>> 
>> 	[SM] If two (or more) packets of a single flow arrive back to back
> at
>> the L4S AQM with the last CE-marked, can you guarantee that that the dual
>> queue coupled AQM will not reorder them even without saturating load,
>> after all it is designed to give the L4S-queue (limited) priority over the
> non-
>> L4S queue?
> 
> [IJ] I cannot guarantee anything,

	[SM] That is a problem, don't you agree? This is something that is empirically testable, the fact that is seems to have not been tested yet, shows that there are still avenues available to kick L4S's tires in the lab and that (experimental) roll-out into the wider internet seems premature.


> my observation is however that possible
> packet bursts are very likely ironed out in node A as it CE marks packets
> for the RFC3168 and is therefore congested. This should mean that the queues
> in node B should be empty and packets will be forwarded as they arrive.

	[SM] But as you conceded, we have no idea whether that is actually sufficient for the dual queue coupled AQM to not re-order packets.


> Still there is a likelihood that packets arriving in the L4S buffer are
> scheduled before packets arriving in the RFC3168 buffer and that can cause
> packet reordering for the RFC3168 flow. 
> The questions are :
> * How large will the reordering depth be?, my gut feeling is that this will
> be in the order of sub milliseconds but I can only speculate. 

	[SM] The way TCP works the number of milliseconds seems less important than the maximum out of sequence-ness, I think about dupACKs here. Sure for the case where only a single CE-marked packet, jumps the queue nothing really will happen (as there will only be one dupACK and after that cumulative ACKs should increase again), except that the flow's tolerance against re-ordering along the rest of the path is now compromised... But the fact remains that this compromise for non-L4S flows only exists, because the ECT(1) as classifier breaks down for CE-marked packets; how about defaulting all CE-marked packets to the non-L4S queue instead, in that case L4S-traffic woulld pay the price for L4S' unfortunate choice of classifier (but due to recommending RACK L4S traffic should be more robust against re-ordering anyway)...

> * Is this worse than the reordering that we see already today ?

	[SM] In the limit yes, because we currently have this additional re-ordering "source" in the internet. 

> * Is the reordering depth more than RACK can handle ?

	[SM] How is that going to help standards-compliant TCPs not employing RACK? RACK, as far as I can tell recommends to keep the permissible re-ordering window <= the unloaded RTT*, so at short (and low bandwidth) paths even a few packets worth of re-ordering might cause issues even with RACK.

Best Regards
	Sebastian



*) https://tools.ietf.org/html/draft-ietf-tcpm-rack-06
"4.  The RACK reordering window MUST be bounded and this bound SHOULD
       be one round trip."