Re: [tsvwg] FW: New Version Notification for draft-white-tsvwg-l4sops-00.txt

[Sebastian Moeller <[e-mail]>]

Hi Greg,

more below in-line.

> On Aug 4, 2020, at 01:35, Greg White <[e-mail]> wrote:
> 
> Apologies in advance for the mangled URLs in the quoted text below.  My IT department has implemented a system that mangles URLs on inbound email.
>  
> From: Sebastian Moeller <[e-mail]>
> Date: Thursday, July 30, 2020 at 3:43 PM
> To: Greg White <[e-mail]>
> Cc: "[e-mail]" <[e-mail]>
> Subject: Re: [tsvwg] FW: New Version Notification for draft-white-tsvwg-l4sops-00.txt
>  
> Hi Greg,
>  
> my comments for sections 1 & 2 below (more will follow):
>  
> "1.  Introduction
>  
>    In the majority of network paths, including paths where the
>    bottleneck link utilizes packet drops (either due to buffer overrun
>    or active queue management) in response to congestion, as well as
>    paths that implement a 'flow-queuing' scheduler such as fq_codel or
>    Cobalt, and those that implement dual-Q-coupled AQM, L4S traffic
>    coexists well with classic congestion controlled traffic."
>  
> [SM] This description ignores the birthday paradox issue that Pete Heist demonstrated. A misbehaving L4S will cause less havok there but it is not without side-effects on stochastic flow queueing systems, we might as well acknowledge that, no?
> [GW] IIRC that issue can occur in fq_codel, but is much less likely in Cobalt (due to set-associative hashing).  Is that correct?

	[SM2] As Jonathan wrote less likely in cake, but still possible. Also, my hypothesis is that fq_codel is more widely in use than cake.


>  
> "   On network paths where the bottleneck link implements a shared-queue
>    (FIFO) with an Active Queue Management algorithm that provides
>    Explicit Congestion Notification signaling according to RFC3168, it
>    has been demonstrated that when a set of long-running flows
>    comprising both "Classic" congestion controlled flows and L4S-
>    compliant congestion controlled flows compete for bandwidth, the
>    classic congestion controlled flows may achieve lower throughput when
>    compared to the L4S congestion controlled flows.  This 'unfairness'
>    between the two classes appears to be more pronounced on longer RTT
>    paths (e.g. 50ms and above) and/or at higher link rates (e.g. 50 Mbps
>    and above)."
>  
> This is rather cautious I would use more drastic terms, the observed unfairness approaches starvation of the non-L4S flows and this text makes it sound like it is a minor almost theoretical concern.
>  
> [GW] We can definitely wordsmith the text, but we need to be sure that we are characterizing the situation accurately and not trying to portray things in drastic terms (or glossy ones) unless it is warranted.  

	[SM2] Well, the thing is Pete Heist's testing (see https://github.com/heistp/sce-l4s-ect1#typical-internet-jitter) demonstrated that L4S with rfc3168 detection, which as far as I can tell is still required for a L4S-compatible transport, is quite bad at achieving fairness, even with RED. 
Sidenote: calling 50ms "longer" RTT paths supports my claim, that L4S is really ONLY ever designed and more cricially TESTED for low-hop count low RTT paths.


> My belief is that it isn’t warranted to use drastic terms here.  

	[SM2] I accept your believe, but think that we should base operator recommendations on hard data if possible, that way our respective beliefs do not muddy the waters.


> I think it is a fairly widely held view that single queue RFC3168 bottlenecks are uncommon at best and the majority of the ones that do exist are likely to be RED AQMs.  

	[SM2] Data please, or we are back at Bob's line or reasoning, that because he is bad at measuring something that something might not even exist.

> The testing with RED AQMs showed approximate fairness in most cases (though more testing would be welcome).

	[SM2] What is your definition of "approximate fairness"? Can we agree on calling imbalances inside of 2 orders of base2-magnitude (aka faktor of 4) as approximately fair and everything beyond 3 orders of bas2-magnitude (factor 8) starvation? If you disagree with my numbers, please propose your own set, but let's get away from the idea that approximate fairness and starvation would be concepts that can not be defined numerically.


>  We’ll need to get a sense from the WG as to how to characterize the situation, given what we know (and what we don’t know). 

	[SM2] With all due respect this WG has not been great at giving feed-back on simple questions like "what is starvation" in the past; as much as I would like to be wrong, I do not see this changing now.


>  
> "  The root cause of this unfairness is that RFC3168 does not
>    differentiate between packets marked ECT0 (used by classic senders)
>    and those marked ECT1 (used by L4S senders), and provides an
>    identical congestion signal (CE marks) to both classes, whereas the
>    two classes respond differently to that congestion signal."
>  
> How about keeping causality intact and frame this as a consequence of L4S redefining what CE means. There are reasons for doing that, but this text leaves it unclear who caused this problem (or rather the text implicates rfc3168, which IMHO violates temporal causality).
>  
> [GW] Point taken.  I’ll try to rearrange that.

	[SM2] Thanks!

>  
> "The result is that the
>    classic senders respond to the CE marks provided by the bottleneck by
>    yielding capacity to the L4S flows.  While this has not been
>    demonstrated to cause starvation of the classic flows, the resulting
>    rate imbalance can be a cause of concern."
>  
> Mmmh, https://protection.greathorn.com/services/v2/lookupUrl/43da5eeb-7a88-4655-a9ba-bec785235113/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=sce.dnsmgr.net&path=/results/ect1-2020-04-23-final/l4s-s2-twoflow/l4s-s2-twoflow-ns-cubic-vs-prague-codel1q_20ms_-160ms_tcp_delivery_with_rtt.svg
> pretty much demonstrated starvation and that is with TCP-Prague with rfc3168 detection. 
>  
>  
> [GW] Ok, I will update the text to be more clear.   It seems that it can be demonstrated in contrived setups that it is possible to cause starvation.  That condition was a single queue CoDel (do these exist in the wild?) with heavily modified settings. CoDel with default settings did not show unfairness.   

	[SM2] Okay, no please proof that these conditions do not actually exist in the real world and you are scot free... And no, changing the target value is not a contrived set-up, for example have a look at the PIE RFCs where there it is mentioned that for short RTT paths adjusting the latency target is recommended, and for codel/fq_codel/cake the same logic applies. This gets ironic, given the fact that L4S itself is really only taylored for similarly short RTT paths...

>  
>  
> "2.  Per-Flow Fairness
>  
>    There are a number of factors that influence the relative rates
>    achieved by a set of congestion controlled flows sharing a queue in a
>    bottleneck link.
>  
>    TODO: discuss startup & convergence times, short flows, RTT-
>    unfairness, differences in deployed CC algorithms, etc.
>  
>    TODO: also mention that flow sharding is commonplace, so per-flow
>    fairness does not imply per-application fairness"
>  
>  
> This obviously needs flashing out before one can meaningfully comment, but regarding the last section, you realize that application not the relevant classification for intermediate nodes? I would assume that for an ISP a (potentially weighted) per-end-host fairness would be the exact tool to avoid sharding as work-around... This is obviously not a new idea or comment, so if you opt for elaborating on fairness, please also include known remedies. Or better avoid that discussion here altogether.
>  
> [GW] Thanks. Yes per-end-host fairness could be useful in some situations.  AFAIK most residential ISPs implement some form of per-customer fairness, which could be argued is more appropriate in that context. I guess the point is that per-flow fairness isn’t always (or even usually) the most important metric for most users/systems.  It seems important to discuss this, seeing as per-flow unfairness is the main concern that has been raised, but I realize this section could be challenging to write in way that all will agree with.

	[SM2] Okay, I see why you want to discuss that here. Please keep in mind, that while per-flow-fairness is not the ultimate solution it clearly is further along that path than L4S's "anything goes". Specifically everything that works around the strikter fq-sheduling will also wreck havoc on "anything goes"-scheduling, so IMHO the argument that FQ is not perfect is beyond the point, as it seems in all relevant criteria to be better or equal to "anything goes" (mostly better) with the sole exemption of computational complexity. I really believe that this document could do without opening that can of worms...

Best Regards
	Sebastian

>  
> Best Regards
>   Sebastian
>  
>  
> > On Jul 30, 2020, at 13:05, Greg White  wrote:
> > 
> > TSVWG members-
> > 
> > I've posted a rough draft of Operational Guidance for L4S deployment.  This is not much more than an outline at this point, and is almost certainly incomplete even at that, so please read it with that in mind. 
> > 
> > -Greg
> > 
> > 
> > From: "[e-mail]" 
> > Date: Thursday, July 30, 2020 at 4:53 AM
> > To: Greg White 
> > Subject: New Version Notification for draft-white-tsvwg-l4sops-00.txt
> > 
> > A new version of I-D, draft-white-tsvwg-l4sops-00.txt
> > has been successfully submitted by Greg White and posted to the
> > IETF repository.
> > 
> > Name:          draft-white-tsvwg-l4sops
> > Revision:      00
> > Title:         Operational Guidance for Deployment of L4S in the Internet
> > Document date: 2020-07-30
> > Group:         Individual Submission
> > Pages:         7
> > URL:            https://protection.greathorn.com/services/v2/lookupUrl/f7c346c8-8443-44e5-b71e-435217709fda/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=www.ietf.org&path=/internet-drafts/draft-white-tsvwg-l4sops-00.txt
> > Status:         https://protection.greathorn.com/services/v2/lookupUrl/e5e7af28-9419-4d0e-a559-650861096bf9/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=datatracker.ietf.org&path=/doc/draft-white-tsvwg-l4sops/
> > Htmlized:       https://protection.greathorn.com/services/v2/lookupUrl/3b91a498-7692-49a5-aeb2-601ef0ba245a/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=tools.ietf.org&path=/html/draft-white-tsvwg-l4sops-00
> > Htmlized:       https://protection.greathorn.com/services/v2/lookupUrl/813cf103-4e26-44a1-8c2f-a2648d5dacc0/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=datatracker.ietf.org&path=/doc/html/draft-white-tsvwg-l4sops
> > 
> > 
> > Abstract:
> >   This is an early, work-in-progress draft - a start at getting some of
> >   the ideas from the mailing list and email exchanges on paper.
> > 
> >   This draft is intended to provide guidance to operators of end-
> >   systems, operators of networks, and researchers in order to ensure
> >   reasonable fairness between L4S and Classic flows sharing a single-
> >   queue RFC3168 bottleneck link.  This draft identifies opportunites to
> >   prevent and/or detect and resolve fairness problems in such networks.
> > 
> > 
> > 
> > 
> > Please note that it may take a couple of minutes from the time of submission
> > until the htmlized version and diff are available at tools.ietf.org.
> > 
> > The IETF Secretariat
> > 
> > 
> > 
>