Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Sebastian Moeller <> Wed, 11 September 2019 11:44 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 763C712088A for <>; Wed, 11 Sep 2019 04:44:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.648
X-Spam-Status: No, score=-1.648 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 45poe4bW7sCO for <>; Wed, 11 Sep 2019 04:44:10 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 9521E1200F8 for <>; Wed, 11 Sep 2019 04:44:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1568202241; bh=ulXNBicnaPyvptoiUcVASpsTlitRMJbmZA6bkSK7A7g=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=VDvFx1HerGJSRJcpuASQgdX+VsRzGml4kOOFfz24A6Zk2LkIfS9TN3FFG8bdQoXcZ XNxaS3oipXfZ9TnHS0hv4ldgBE6788mPhoxLfHRZnAVoJ08QqGXCctbe0nD6inxOeg dNAFh/nsbkVEbF/IweafcqgEN1vjMKR3KL0BJSdU=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (mrgmx003 []) with ESMTPSA (Nemesis) id 0MaIsi-1ho8k51kyG-00Jtqm; Wed, 11 Sep 2019 13:44:01 +0200
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Wed, 11 Sep 2019 13:43:59 +0200
Cc: "" <>, "" <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <LEJPR01MB1178B6C102455F1F9886D49A9CBB0@LEJPR01MB1178.DEUPRD01.PROD.OUTLOOK.DE> <> <LEJPR01MB11789EE6D8B7C732393BD1439CB70@LEJPR01MB1178.DEUPRD01.PROD.OUTLOOK.DE> <> <> <>
To: Greg White <>
X-Mailer: Apple Mail (2.3445.104.11)
X-Provags-ID: V03:K1:bFXzIz26G2OsdWZf7nOb9KBv1eLL5TsbCHtKRaYgd7WfWsWurjP lmBHFP5IKNWiEokCuXhTOCYTjjWeaIpgwVMP6vOrEdSDA3BNfvv8Jk1/2tR5mMHEVtMOz/9 Q459wkKbmdM+ql9c0HNTICMkclVDac4gEekKgkOHS7hjj1Yx+wUdYILt8Rj9Pe3qE/qYUIw YzO2Yo30D6tyD1+79fxsw==
X-UI-Out-Filterresults: notjunk:1;V03:K0:t7arYmSzgAU=:syEDWUlvzF1CZ54KY9Eiz1 cykeeWgNNJZm6Ni+w9OlsOlP8Ntg9YkFzFV6QFA1JUvPmupbPITra6w2py5b2CU72CJgENnu6 iEs3ZTIf5HOInhE/sbvIVSWNaK7fmoT5ce8wPLVdmNEFXw4/w60TlxR5KetdOraPPsgIuze7H 8Egu0SfP1i/PqiJj+i+SocEU0bUGhSPqBBrrBich1TENC3+XRwGFiu+PjHnZfzhV/1qw9Lfni wbfR3sW6b6PkhBYOfvKuSTCLGkdE4o5n0TfnDUlBNI99QSJU1Rr9X7lddp/fPMjqxfMyec4+d bnjWnDjr/2Z8onuz9/WQ2cDNsJ6zfWvzfoLcZDh5gYqjiudmq5c29SYJqbEmv1DfGWCrxIVCN XcnEsWw0LpBR1n2IuVaElOZHCGIzDl0Ja82We4CB99uYR5rbCRD9FDusxedfLiHRX19TE9iiy Z5oAnyWYKyQCi31DlHhmhKt5xSU/8Jwc8XuUBAzHMKCAM4RA5iXvd/I4AXjIUjF95+k/kVK0T O5bt8UKKv3uo/BIBXx583+uiXvCY8WqIaCC41S2V9mdCwIHLpHECdmiZif4Crp/Uw5AeCsWZ0 Iyou8h6AzAI0o7HWkV3S+4QMVvJpuwMZfEwLrurKMugcp7Wbli1O7bwH7O8KSDmKxeg8TRspV ZO6N2tg8jXDCPsuTBjRr5j62Z4i3j9ygD2hOQRUF/qHIYIXPmo+6t7Zt051JY7yqBiuIZpded AEbl9tgRKKIVgCYfF7P+p7dwtin7LVYH9awVVgFWEi+/4Qom8+b5zdZ0D1LCEDY6MVL7XBWID 7DGeFMr/zU12KnzFjKTA/oXaAkfLJv5Xm2Rq7VEb+/AOPqs3yTBB3E7J3qig4NX7BDWEz3+Eh SxgQwiOJU/o+45jbjka2lRwnaHPsSm4dIEbBPSzEwmh70/qJ3ZLzMfjpdTIQxx8RHPL6B36tK mmjBuPtGiqdhY6Qjd12D1MI8KpTvipFVmewFOXkHFJjFNfmgZk+W4AosC2NwYiM4FU2YXqmPp UVeDf0r9X+kRUvJgg/0VwkJyV01jhW9t1m8B3WPVe//EnY72DhCyEfm6AdDHq9GDKGisLd4p8 zowT/3jMM4ME0yEGYV6axniCOTwLwJGZ5vCpuv2zCtlwYxedcjpOBwteimyB7Qdut7hl1bLU5 4PnzXQv6MCF1XvwsOL3YHBdktLh8ByhwOYrWJTGg41yziBOJA7CCG/jySl5ecOiIMi1Z2bd9H Ax861r8MtIJxjwIf2
Archived-At: <>
Subject: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 11 Sep 2019 11:44:12 -0000

Hi Greg,

thanks for your response, comments prefixed with [SM]

> On Sep 10, 2019, at 23:28, Greg White <> wrote:
> Hi Sebastian, 
> See below marked [GW].
> On 9/10/19, 1:23 PM, "Sebastian Moeller" <> wrote:
>    Hi Greg,
>> On Sep 10, 2019, at 20:29, Greg White <> wrote:
>> Yes, my focus (both from the perspective of the systems that I work on, and from the perspective of where I think NQB differentiation is most valuable) is on last mile and WiFi.       
>    	[SM] Except with the current very loose definition of which flows qualify for NQB marking, NQB seems to only merit AC_BE, which is already the default, so is NQB actually going to be valuable for wifi?
>> In terms of interconnection, there could be multiple ways to handle this.  My view is that the endpoint application has the best a priori knowledge of its sending behavior, so it is good to leverage that (and either validate it or not, depending on what is most appropriate for the situation).  
>    	[SM] not validating the behavior but giving special powers (like the originally proposed AC-VI/AC_VO on wifi) seems like a genuinely optimistic ideas. 
> [GW] What validation is done today for applications that mark their packets as AC_VO or AC_VI?  

	[SM] As far as I know almost no validation, but do to the traditional DSCP bleaching over the internet and the reluctance of most applications to twiddle with DSCP bits, most traffic will be in the AC_BE, but admittedly rather by chance than by design. In addition the IEEE's default DSCP to AC mappings are quite conservative (like mapping EF to AC_VI not AC_VO) making high volume traffic in the higher ACs relatively unlikely with such DSCPs that might survive internet passage. But with NQB's desirable approach to be transmitted e2e, that is going to change.

> I think that WMM is another one of those cases where the tragedy of the commons seems imminently possible, yet it largely hasn't occurred.  

	[SM] Mmmh, with 2.4GHz radios in a dense AP-concentration situation, we are close (and this is by beacon traffic alone).

> But, I hear you about being more clear about which flows are intended to be marked NQB. We certainly don't want to recommend something that creates the tragedy.

	[SM] Great, I think we can all agree on this ;)

> Perhaps we've left it too ambiguous in the draft, so that needs some review.  It was not the intent that capacity-seeking flows (even L4S) mark their packets as NQB.

	[SM] Great, maybe this can be made more explicit?

>  The target flows would be ones where the sender has a degree of confidence that it will not exceed the available capacity of the path.  

	[SM] With a variable bandwidth/rate path like wifi, the sender's confidence might not be the best measure? I would guess the hops operating the RF link would be in a better position to predict the path capacity in the immediate future?

> Also, I think validation of NQB behavior is a very useful tool, and it is in fact mandatory to implement for DOCSIS links.

	[SM] +1.

>  Other links (including WiFi) could implement it as well.  

	[SM] I fear I am less optimistic, and would argue that without this implemented wifi should be very careful to prioritize NQB (unless we can agree on NQB qualifying behavior that makes harm very unlikely).

> For example, WiFi APs or Stations could monitor queue depth or queue latency for the AC_VI (or AC_VO, whichever NQB maps to) queue and re-direct NQB traffic to AC_BE, either in a flow-aware way (probably preferred) or possibly even in a flow-blind way.  In other situations it may not be necessary (e.g. in controlled environments or on links that support FQ), or it may not be feasible.  

	[SM] All good ideas; the question to me is more in light of the lack of such mechanisms would it not be safer to default to AC_BE and only selectively elevate NQB traffic if the radio controller/AP is NQB aware? 
	I ask because I just did a flent rrul_CS8 (that is bi-directional greedy TCP traffic with one flow dscp-marked for each CS) test-run on my home net, and saw my macbook's sent CS6 and CS7 flows almost starve all other traffic, including CS6/CS7 marked traffic from the AP; I am not confident, that the hardware out there is ready for significant traffic volumes on AC_VO/AC_VI yet... (happy to share the flent data file on request).
	Now, I realize that NQB is not intended for bulk flows, but as long as the marking is not actively verified (and maybe even bleached on detected mis-marking) the intention does not matter that much, NQB-mismarking will give a considerable bandwidth and latency improvement. With access links getting faster and faster, wifi is becoming more and more a (transient) bottleneck making this issue sensitive.

>> This would argue for networks to adopt a policy of not bleaching the NQB value on traffic arriving from interconnections, and utilizing queue protection if necessary to validate NQB behavior.   That said, there are other ways that network operators can identify NQB traffic without solely relying on endpoint marking (and DSCP preservation across interconnects).  I know some operators are investigating such techniques.      
>    	[SM] if the name has any relevance, one could treat each flow as non-queue-building until there is enough evidence to change that classification (like fq_codel treats all new flows as sparse initially until their observed bahavior merits a different classification).
> [GW] Exactly.  That is one mechanism that has been discussed.  A possible downside to that approach is that once the flow has demonstrated that it is QB (by building a queue in the NQB buffer) and subsequent packets are classified to the QB buffer, it is possible that the QB buffer was empty when those packets arrived, and they end up being transmitted before the tail packets in the NQB buffer, i.e. out of order delivery.  Flow startup for traditional TCP flows (even those that support RACK) can be sensitive to out of order delivery, so if this happens frequently it could be an issue.

	Interestingly it was a similar observation that made me initially think, DOCSIS queue protection would track each flow's packets in the queue so that on data-driven change of classification all queued packets of that flow could be moved from one queue to the other (I obviously did not consider the cost of tracking very deeply). Even though at that point it might be easier to directly go for stochastic flow queueing, as in that configuration re-ordering is not an issue and it should be much easier to verify compliance to required behavior if the flows are isolated... 

Best Regards