Re: [tsvwg] Traffic protection as a hard requirement for NQB

Sebastian Moeller <> Tue, 10 September 2019 20:17 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 2F22412082E for <>; Tue, 10 Sep 2019 13:17:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.348
X-Spam-Status: No, score=-2.348 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id jhnHDbxAhkia for <>; Tue, 10 Sep 2019 13:17:47 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id E57CB120288 for <>; Tue, 10 Sep 2019 13:17:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1568146620; bh=t6WMprTduAUC7hcQkvWl2pjKmYppkRd90Nb2tp9UnOo=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=SveUHKKglhQ/9CxVLH6rMhDwerXCyuq/6HoTEp9HiigUFnS6aJGaZsLeS/Ufse83k EX63gxw+f+poSjkKHcpYFmHa5ZQu4LPMqR4v8TSUQUvW+W8S3WCaVGyMrkG6lhBTgs /j+HWBX9pW0iROW1jH4tWKi+iczCO/q9Ya3zwoUU=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from hms-beagle2.lan ([]) by (mrgmx102 []) with ESMTPSA (Nemesis) id 0M3MAG-1iPaTE01eC-00r3Y7; Tue, 10 Sep 2019 22:17:00 +0200
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Tue, 10 Sep 2019 22:16:57 +0200
Cc: "Black, David" <>, "" <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <> <> <> <> <>
To: Bob Briscoe <>
X-Mailer: Apple Mail (2.3445.104.11)
X-Provags-ID: V03:K1:14uTm8rsACVaNvGC0wdmM4Ady3cX33oLUeDCumwp1aPuxmxlxdx OsSB9rUoSR5oKwowGgG4tZoE4XS+ApWhDFE8KyzQeXCBUEj8cjfKQryJWuqZeXvLmKWRiWl QYcCszAF5sahhlFVYU1L+E7LFAj+k+64eGLByfjdcW2AW7Kni8uxmU8KRa2ePIl542f+bAZ boTPvcvWlqDB5VefqZEyw==
X-UI-Out-Filterresults: notjunk:1;V03:K0:9YwTKJW9VuU=:RJWmJbGX2oVCxUr+AudaQg UQE4Z2u8EE2ByHFsp3T1t6qhN1F/shBhpzugo5Ny+TIwtdiKTqB7VnmPzH16iKb7O747u46SX EBRLSfCBWrHIBYv1BfKS9Q8m6qL54Eb4JuvBDY+CfjfyRmj85yKriPALdJoUkOdE/NI+F1jYm l/702mjDnhomxAG27mCP49bDaon6JXXM+8R3mQmTZpbREYQWUmZLiKy57zzTBeuXcQOPhe5wN LlzsDSjxiQ2PZig/Swzo00Dlv6qID+OVKE6K8a8rdKTAqkYhpQPoVGpSwq3FulJzVSCwcXIID sgLc2TiDaQGym1y9S26c0AEaNrgERetF8hV3L5+6h6k05MFwhe+j8eCGNCUI1lA2VODFF0Gic VpJYm/BoucZwa1YoUV3NnW95/PilhpyEsB/tYHfBXMjzWP1DMNgMopV8GnPVbxQg1a+1k8agJ Kt6is6pk95xcCHhze+9F4WHSST6l2yAGCuS5Eghf8BAezjwuJj8X4+DF2xoX0ojMIzv9qFWaG s0+/TV+4yfXLb0T5IfyWf/BMeLNTB28yxC6aUwsaO0Mr09+BK1Bav7m7yr1BLxr6soXT0u63d 47Io+GwE0KkezRPReqI2ZPDLff5paWApxRYC7EzyNtpCE/BMHM0sWJUkLF/ND50T2cUii8BpA dURwELhELExufLG3hZflMWdBZcAPOcES/vwjWxvdzHuI8x1pk7j+/TpxF+17Nf7NL1TBEyRH6 eIdCfuUWAY4joPl5c4P5UTvK8Rd3a+dBwBe2P9UJYB8NHG3oryOmt8MwySpo/h3t0RFG1pNL9 aX5BldH+q0TUtkMgDoLARJIvOK/vsKp/ciPdIz7LmGkMbleoWqqPy+UWROgvkb1CW9fnUFvun K/Znc+nYaLRhKFayEK+Bcwx5MBXoFzNHBipECNdVrk81TjHtvatqPJ+ztmm5kHSSPgpuMD671 u9fPlwsyhG4HOZr56Q0i7b11uNbQiEisi+2XkZZZZj3Hs0boQFy+2lWdtc9cEqOdnrzpGt/LE 48HZnmQSqx2xo01XxUKuuS51CMcct+GEkAVtyRg1qOP9J/9VdQt/EB8t20Jt5CT1xb0mSm6xW 3DH3yddjf7X/AG/0vSwAb66gTDPwysPGTvLCv7I73l4C1SSfFW2HGkxDVGygBDH/kdVwEVCHZ HtXRUbkXu9YgB42p2OY+T5ZDS/LAhyTByP64yNCQEWuEiNV3fntQs/rY4sWVv68VLPMI6eha2 UxmQz3GL58DGR6cGp
Archived-At: <>
Subject: Re: [tsvwg] Traffic protection as a hard requirement for NQB
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 10 Sep 2019 20:17:51 -0000

Dear Bob,

> On Sep 10, 2019, at 17:13, Bob Briscoe <> wrote:
> Sebastian,
> On 07/09/2019 16:48, Sebastian Moeller wrote:
>> Dear Bob,
>>> On Sep 5, 2019, at 20:23, Bob Briscoe <> wrote:
>>> ==Incentive Alignment?==
>>> To judge whether there's any tragedy of the commons (incentive alignment), I'll put up a straw-man NQB configuration that complies with the following requirement in the draft:
>>>    a useful property of nodes that support separate queues for NQB and QB
>>>    flows would be that for NQB flows, the NQB queue provides better
>>>    performance (considering latency, loss and throughput) than the QB
>>>    queue; and for QB flows, the QB queue provides better performance
>>>    (considering latency, loss and throughput) than the NQB queue.
>>> Background: NQB has become possible largely because Internet access link rates have typically become fast enough that the serialization delay of a packet can be sub-millisecond, and therefore a queue of a few packets introduces delay that is small relative to other non-optional sources of delay like propagation. In these cases we no longer need priority scheduling for low delay.
>>> Config:
>>> 	• Scheduler:
>>> 		• WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least 60Mb/s for NQB flows.
>>> 		• Scheduler quantum: 1500B.
>>> 	• Buffering:
>>> 		• The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
>>> 		• The QB buffer is deeper (say 200ms) with an AQM target delay of say 10ms.
>>> Traffic: Let's introduce some example NQB traffic:
>>> 	• 2 paced flows of VoIP datagrams, each avg 50B payload plus 58B of Eth/IP/UDP/RTP headers at 33pkt/s
>>> 		• bit-rate: 29kb/s /flow
>>> 		• serialization delay: 7.2us / pkt
>>> 	• 2 streams of 1000B game sync datagrams at 30pkt/s =>
>>> 		• bit-rate: 240kb/s /flow
>>> 		• serialization delay: 67us / pkt
>>> 	• plus occasional DNSSec datagrams, worst case 1500B
>>> 		• serialization delay: 100us / pkt
>>> 	• Perhaps 540kb/s in all, which is about 0.9% of the min NQB capacity.
>>> Worst-case NQB queuing delay calculation for the above traffic model:
>>> Each NQB flow paces out its own packets, but one might happen to arrive while a packet from any of the other NQB flows is already queued. Worst case n-1 = 4 other NQB packets queued, where n is the number of application flows. And if there's traffic in the QB queue, each NQB packet will have to wait for one quantum (100us) while the other queue is served. Worst-case NQB delay is then:
>>> (67us * 2 + 7.2us + 100us) + (100us * 4) = 641us,
>>> It's v unlikely that this worst-case would arise, but it gives an indication of where the tail of the delay distribution will lie.
>> 	[SM] I just re-thought this section, and believe it to be a strawman argument.
>> In my reality the NQB draft states:
>> "7.  Relationship to L4S
>>    The dual-queue mechanism described in this draft is intended to be compatible with [I-D.ietf-tsvwg-l4s-arch]."
>> In turn [I-D.ietf-tsvwg-l4s-arch], drags in I-D.ietf-tsvwg-aqm-dualq-coupled which sizes the shared dualQ buffer to allow for 250ms at the egress bandwidth ("limit = MAX_LINK_RATE * 250 ms               % Dual buffer size"), and not 3 ms.
>> In the example above it is ONLY the extremely shallow buffer of 3 ms that "aligns" the incentives such that QB traffic gets unhappy in the NQB queue. So either  -D.ietf-tsvwg-aqm-dualq-coupled needs to change, or your argument above needs adaption to 250ms. And even if we would allow 50% of that for the QB queue, that still leaves 125ms. Since you are heavily involved in the other L4S drafts this is a rather surprising lack of candidness about how the NQB queue is actually supposed to be operated.
>> Feel free to demonstrate again how with a 125 ms buffer (that is 0.125/((1500*8)/(120*1000^2)) = 1250 packets or 1250*1500/1000^2 = 1.875 MB, a whole lot of flows are one with their business long before they transmitted 1,8 MB)  there are no incentives for QB flows to mark themselves as NQB unless there is a stringent monitor and enforcement regime operated at the bottleneck NQB queue operating element to weed out misbehaving flows. But please plug in the numbers that the l4S family of drafts actually recommend to use, so it becomes relevant for this draft.
> You're right that, when the NQB draft says it is compatible with l4s-arch, it means it is compatible with the structure of the architecture, not necessarily the specific recommended numbers in one of the example implementations in aqm-dualq-coupled.

	[SM] Well, as far as I can tell these are the only numbers around.

> Indeed, in Appendix A of aqm-dualq-coupled, note b says that two separate buffers might be preferred to a shared 250ms buffer [and the Low Latency buffer would not need to be more than a few ms].

	[SM] Indeed, except I can not find the part in brackets in that draft. Setting the Low Latency buffer to only a few milliseconds, will then make the whole low latency queue susceptible of the attack scenario I described before, is that really an improvement (and helping your cause])? 

> That said, here's a strawman (i.e. not tested, but for you to bash) for adding NQB support to the DualPI2 example in appendix A of aqm-dualq-coupled:
> * Classify NQB packets as well as L4S ECN into the Low Latency queue
> * Don't subject NQB packets to the L4S AQM (unless their ECN field is also L4S).

	[SM] You are talking about the ECN marking component only here?

> * During overload, subject all packets to the same drop prob (Classic, L4S and NQB)

	[SM] Is there any other option, unless one wants to effectively starve all other traffic "classes"?

> * add a 3ms sojourn limit for packets marked NQB (but not L4S), after which they are dropped

	[SM] Great idea, looking forward seeing this in the next dualQ draft! Until then I will assume 250ms/2 as the likely default value. I also note that this adds another complication to dualQ, basically multiplexing a dropping regime into an ECN based system.

> * It might be necessary to tune down the L4S scheduling weight to bolster the incentive against mismarking, e.g. 3/4 not 15/16.

	[SM] You mean the part in the dualQ draft that says " if L4S traffic is over-aggressive or unresponsive, the scheduler weight for Classic traffic will at least be large enough to ensure it does not
      starve."? I would not actually direct attention to this part of the dualQ rfc, as for a system that claims to share between L4S and standard-compliant flows only giving 1/16th to the standard-compliant flows is a problem.

> The alternative specified for DOCSIS is to use a 10ms low latency buffer but also to implement per-flow queue protection (which can be disabled by the operator).

	[SM] If we are talking about [DOCSIS-MULPIv3.1]  Cable Television Laboratories, Inc., "MAC and Upper Layer Protocols Interface Specification, CM-SP-MULPIv3.1-I18-190422", April 22, 2019,
              < CM-SP-MULPIv3.1>.

In Annex N I read: I read: 
BUFFER_SIZE                // The size of the buffer for the LL service flow [B]
                           //  (see Section C. A value of 10 ms multiplied
                           // by the Maximum Sustained Rate (MAX RATE) is recommended

Naively, I would say this is only 10 ms if MAX RATE equals 1, no? What am I missing.

Best Regards

> Bob
>> Best Regards
>> 	Sebastian
> -- 
> ________________________________________________________________
> Bob Briscoe