Re: [tsvwg] Traffic protection as a hard requirement for NQB

[Sebastian Moeller <[e-mail]>]

Hi Bob,


> On Sep 5, 2019, at 20:23, Bob Briscoe <[e-mail]> wrote:
> 
> David & Sebastien,
> 
> [These responses cover come of Sebastien''s points that were common to David's reply as well - I'll pick out the remaining ones specific to Sebastien's email in a subsequent response. Sorry for the long response, but I've tried hard to cover this rigorously.]
> 
> On 03/09/2019 22:46, Black, David wrote:
>> [Writing as an individual]
>> 
>> The core of my current views is this:
>> 
>> 
>>> ... “traffic protection” (e.g., “queue
>>> protection” or a suitably configured FQ AQM) appears to be necessary in
>>> general to keep queue-building traffic out of the NQB traffic aggregate, as
>>> allowing such traffic degrades the properties of the NQB PHB.
>>> 
>> I see serious potential here for a "tragedy of the commons" outcome where lots of QB traffic is sent marked NQB because the resulting latency is (at least initially) lower.
> ===Misunderstanding?==
> 
> 
> Using a term like 'tragedy of the commons', implies your argument might be based on a different understanding of what NQB is to mine. I'm trying to unpick what you think NQB is from your sentence "the resulting latency is (at least initially) lower" for a QB flow that marks itself NQB?
> 
> Please confirm that you (both) understand that, with NQB, the low latency is not provided by scheduling in the network. It is primarily a property of NQB traffic sources, and the only necessary property of the network's per-hop behaviour for NQB is to isolate NQB from QB traffic (i.e. in a separate queue for the NQB class). There is no priority scheduling requirement. 

	[SM] Assuming our NQB endpoints employ reasonably competent oracles to actually make that feasible.... especially in the light of transient "cross-traffic". Also the NQB senders will have to actually time their transmissions such that their packets arrive almost perfectly "desynchronized" (spread out such that each of their packets just hit our non-scheduling bottleneck hop just before the "egress" time-slot assigned to each flow). This is not going to work without some form of feed-back from the bottleneck... In my understanding the non-queue build hence requires the bottlenecks interaction, a process I would naively call scheduling; and yes since the only node that has enough information to actually synchronize the end-points I would call this scheduling in the network. Either that or our NQB flows will build up (transient) queues at the bottleneck. What a I missing here?

> 
> [A comment I would make to the authors about the draft: it needs to give some example approaches to scheduling between NQB and QB, or at least talk about this scheduling.]
> 
>>>> I think we should be wary of making traffic protection a hard requirement
>>>> at such an early stage in our knowledge of the NQB behaviour. I believe this
>>>> is a case where the market, not the IETF, ought to decide whether protection
>>>> is required.
>>>> 
>> I want to see clear and convincing evidence that the "tragedy of the commons" outcome is highly unlikely, which this text ...
>> 
>> 
>>>> The draft claims incentives can be aligned by an implementation being arranged to ensure that NQB traffic benefits from NQB marking and QB
>>>> traffic benefits from QB marking. Incentives are hard to guess, so that may or
>>>> may not be true. However, I don't think we (the tsvwg/IETF) can state
>>>> categorically that it is not true.
>>>> 
>> ... is not.
>> 
>> The above text not only reverses the burden of proof but also changes the question from whether a "tragedy of the commons" is possible to whether it will always (categorically) happen - the resulting straw proposition is then easy to demolish.  Nice try, no sale ... 
>> 
> [BB] OK, I'll turn the burden of proof back round again... 
> 
> As this insertion is long, I've used sub-headings.
> ==Incentive Alignment?==
> 
> 
> To judge whether there's any tragedy of the commons (incentive alignment), I'll put up a straw-man NQB configuration that complies with the following requirement in the draft:
>    a useful property of nodes that support separate queues for NQB and QB
>    flows would be that for NQB flows, the NQB queue provides better
>    performance (considering latency, loss and throughput) than the QB
>    queue; and for QB flows, the QB queue provides better performance
>    (considering latency, loss and throughput) than the NQB queue.
> 
> Background: NQB has become possible largely because Internet access link rates have typically become fast enough that the serialization delay of a packet can be sub-millisecond, and therefore a queue of a few packets introduces delay that is small relative to other non-optional sources of delay like propagation. In these cases we no longer need priority scheduling for low delay.
> 
> Config: 
> 	• Scheduler: 
> 		• WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least 60Mb/s for NQB flows. 
> 		• Scheduler quantum: 1500B.
> 	• Buffering:
> 		• The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
> 		• The QB buffer is deeper (say 200ms) with an AQM target delay of say 10ms.
> 
> Traffic: Let's introduce some example NQB traffic:
> 	• 2 paced flows of VoIP datagrams, each avg 50B payload plus 58B of Eth/IP/UDP/RTP headers at 33pkt/s 
> 		• bit-rate: 29kb/s /flow
> 		• serialization delay: 7.2us / pkt
> 	• 2 streams of 1000B game sync datagrams at 30pkt/s => 
> 		• bit-rate: 240kb/s /flow
> 		• serialization delay: 67us / pkt
> 	• plus occasional DNSSec datagrams, worst case 1500B
> 		• serialization delay: 100us / pkt
> 	• Perhaps 540kb/s in all, which is about 0.9% of the min NQB capacity.
> Worst-case NQB queuing delay calculation for the above traffic model: 
> Each NQB flow paces out its own packets, but one might happen to arrive while a packet from any of the other NQB flows is already queued. Worst case n-1 = 4 other NQB packets queued, where n is the number of application flows. And if there's traffic in the QB queue, each NQB packet will have to wait for one quantum (100us) while the other queue is served. Worst-case NQB delay is then:
> (67us * 2 + 7.2us + 100us) + (100us * 4) = 641us, 
> It's v unlikely that this worst-case would arise, but it gives an indication of where the tail of the delay distribution will lie.
> 
> Tragedy?
> Now, why do you think "the resulting latency is (at least initially) lower" for a QB flow that marks itself NQB? Why do you think incentives are misaligned (i.e. a tragedy of the commons)?
> 
> Certainly, as the draft says, there is a danger of QB being mismarked due to accidents or malice. However, there's no /incentive/ to mismark to /get/ lower latency, in the strawman config above.

	[SM] But this also gives a great incentive for latency sensitive capacity seeking traffic to mis-mark itself into the NQB treatment, 0.6ms delay versus >=10ms. Why is that not an incentive, I fail to see. If a flow is sufficiently short (and say for browsing many are done in iw10) or bursty this delay advantage will be real...
	

> 	• If a QB flow mismarked as NQB causes higher latency, it will cause it to itself.
> 	• If a long-running flow behaves in a QB way, one assumes it needs to build a queue of perhaps 10ms for full utilization, so in a 3ms buffer it will experience high loss and badly under-utilize the link.
> 	• If a short QB flow mismarked itself as NQB, it would experience its self-induced latency during start-up. 
> Nonetheless, If there already happens to be a long-running QB flow in the QB queue, it would indeed get lower delay than if it marked itself truthfully and added to the QB queue. But, with the shallow NQB buffer, the mismarked flow would find it very hard to avoid high loss, especially tail loss.

	[SM] But at 30 packets buffer depth tail loss is hardly that much worse than head drop, no?

> 
> This last point is what I was thinking of when I said "Incentives are hard to guess". Even if this was tried experimentally,

	[SM] What bothers me personally a bit, is the "if this was tried" bit in the above instead of actually running a real experiment.

> it would be hard to know whether the harm from high risk of loss would dominate the benefit from the reduction in delay. But if one accepts that a loss is usually equivalent to more than 1RTT of delay, it seems there is no incentive for QB flows to mismark.

	[SM] Well, my hypothetical bursty web flow will just be fine as long as it does not send bursts larger than 3 ms.

> 
>> or as Sebastian writes:
>> 
>> 
>>> 	[SM] I would have thought it would be on those claiming a laxer
>>> enforcement model to proof that that model is sufficient?
>>> 
>> I agree.  As for requirements language ...
>> 
>> 
>>>> While there is no operational experience of NQB deployments, I think the market (i.e. most early-adopting operators) will want the warm feeling of
>>>> some form of traffic protection. But as we get more experience we might
>>>> find incentives really are aligned. And we might find accidents and malice are
>>>> not a significant problem.
>>>> 
>>>> 	[SM] @ the chairs, how hard would it be to retroactively change from
>>>> a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be
>>>> nice, but if it is hard I would vote for a MUST as that seems to be the safer
>>>> option.
>>>> 
>> ... a common IETF approach to this sort of uncertain situation is "MUST implement, SHOULD use" with discussion of when the "SHOULD" may not apply and/or possible consequences of ignoring the "SHOULD."  The underlying rationale is that the functionality will be available for use if/when the need for it is discovered in a fashion that surprises the network operator.  It is much easier to relax "MUST implement" based on implementation experience than it is to upgrade "SHOULD implement" as the former doesn't have the potential to break any "running code" implementations.
> [BB] As this insertion is long, I've used sub-headings...
> ==Incentive Alignment was the point of NQB==
> 
> 
> As I understand it, the latest NQB draft was a merge of:
> 	• NQB from the DOCSIS community 
> 	• and LoLa from the GSMA community. 
> The whole point of LoLa was to trade off loss and latency incentives to avoid the complexities of Diffserv policing.

	[SM] This seems a sane approach, make low-latency costly in another dimension and users will think hard what they actually require.


> The point of NQB also concerned incentives, but avoided the LoLa tradeoff between lower latency and higher loss. Instead, both QB and NQB got better performance by marking themselves truthfully than they did by mismarking.

	[SM Now, show that this holds true for conditions instead of one example...

> 
> 1/ Incentives are a first order concern. If incentives are not aligned, that affects all typical cases, because every party can be assumed to be acting in its own interests.
> 
> 2/ The NQB draft introduces traffic protection in the context of accidents and malice. They are second order concerns, i.e. not expected to be the common case, but still potentially of concern.

	[SM] You realize that DDOS as a service is a thing people pay for over the internet? I would say malice is much more common than you make it sound.

> 
> I might be more sympathetic to "MUST implement" if incentives were not aligned. But even if incentives were misaligned, I don't think it would be normal or appropriate for the IETF to mandate protection...
> 
> ==MUST is not for Nannying==
> 
> 
> Quoting the bible on the use of MUST [RFC2119]:
> 
>    Imperatives of the type defined in this memo must be used with care
>    and sparingly.  In particular, they MUST only be used where it is
>    actually required for interoperation or to limit behavior which has
>    potential for causing harm (e.g., limiting retransmissions)  For
>    example, they must not be used to try to impose a particular method
>    on implementors where the method is not required for
>    interoperability.
> 
> a) There is certainly no interoperability concern here.
> 
> b) "Limiting behaviour which has potential to cause harm" means limiting harm to others; it does not mean it is appropriate to mandate that an NQB implementation MUST limit harm to itself. That is nannying. Implementers are perfectly able to make that decision for themselves.
> 
> For instance, regular Diffserv with prioritization has much stronger protection concerns than NQB because, not only is there potential for accidents and malice, but incentives are not aligned. Nonetheless, Diffserv was divided into standards track RFC2474 about the DS field (interop), and informational RFC2475 about domain structure, policing, etc. There is no normative text in RFC2475 and I don't think the IETF ever mandated that a border router MUST implement Diffserv policing.

	[SM] But unlike NQB "normal" standard DSCPs were not supposed to survive passage though the internet e2e.

> 
> ==Continual Run-time Protection is Unnecessary==
> 
> 
> Even if we did want to mandate protection, it would be inappropriate to mandate continual run-time protection. Because it is common for accidents and malice to be weeded out before run-time.

	[SM] This seems quite optimistic in light of giving gaming traffic as a user for the NQB marking, and building a system in which relatively small periodic bursts are sufficient to degrade NQB performance. If countering malice is only an after thought it is not going to work.

> 
> For instance, in the upstream of a mobile network, the RAN scheduler only protects one user's traffic from another's. It would not protect one NQB app of a user from another NQB app of the same user. To do that, a step could be added to the app-store validation process to verify correct use of NQB by an app.

	[SM] That puts an awful lot of trust into the walled-garden vetting processes.

> 
> Similarly, on other platforms virus checkers weed out malicious software at installation time. 
> 
> ==Mandating a Layering Violation?==
> 
> 
> Saying "MUST implement" some form of traffic protection would turn NQB from a simple L2 or L3 class-based queue isolation scheme (with a "SHOULD" recommendation to access L4) into an approach that is non-compliant unless it accesses L4 headers.

	[SM] Then require at least a pure src/dst IP address tuple for your "flow" definition to at least protect different end points from each user, and for IPv6 also include the floe label (just declare IPv4 to be out of scope), if layering violation are of such concern. IMHO the fact that (to my knowledge) most internet traffic is either UDP or TCP indicates that "port" addresses really are useful, and maybe the "layering violation" is that the IP header omits these fields, but I digress.

> 
> Low latency is a big deal in 5G.

	[SM] Which so far means in 5G marketing...


> The 3GPP complied with normal network layering principles when it required the 5G RLC layer to compress and encrypt its payload (the DSCP at L3 gets translated into a 5QI in the RLC header at L2, which is equivalent of a QCI in LTE).
> 
> How will it look to the 3GPP if the IETF introduces a low latency DSCP that mandates access to L4 headers in order to use it? 
> 
> 
>> 
>>>> As an analogy, when TCP congestion control was first developed, it was
>>>> 
>>> known that end-systems could run a subverted TCP algorithm or just use
>>> unresponsive UDP. At that time, the view could have been taken that per-
>>> flow scheduling would have to be a 'MUST' requirement for all Internet
>>> bottlenecks.   As it has turned out, the Internet does have /per-user/
>>> scheduling at bottlenecks,
>>> 
>> I understand the TCP congestion control history, and the even-more-relevant ECN for TCP history, where there was a serious potential for a "tragedy of the commons" outcome - ignoring ECN congestion indications was a potential new way to cause congestion collapse of the Internet (fortunately, that did not happen).  There are at least a couple of aspects that seem rather different in the current situation:
>> - At the time, the IETF community had strong influence over the important TCP stacks. 
>> 	The network stacks have gotten much more diverse since then, and a lot of NQB traffic
>> 	can be expected to come from applications that have a much weaker relationship to the IETF community.
>> - Switching away from TCP to UDP was a high hurdle, e.g., as TCP uses stream sockets, whereas UDP uses datagram sockets, and ECN functionality was in the core of TCP implementations.
>> 	The hurdle here is much lower, particularly for applications that already use UDP.
>> 
> [BB] That's all true. Yes, it increases the risk, particularly of accidents and bugs.
> 
> Don't get me wrong, I'm not trying to argue down the risk here. I'm saying it's not the IETF's role to mandate what to do about the risk (no matter how high it is). The arguments above against nannying and mandating self-protection all still apply. 
> 
>> 
>>>> In the TCP case, it turned out that a delicate balance of incentives proved
>>>> sufficient to allow most Internet equipment to be simpler and cheaper.
>>>> There is a poorly understood balance of incentives in the NQB case. So let's
>>>> not require equipment to be more complex than it might need to be, at least
>>>> not yet.
>>>> 
>> I concur with Sebastian that the onus is on the proposers of the complexity/cost savings to demonstrate that the resulting designs are safe for the Internet.  I'm reminded by analogy of something that my software engineering professor said back when I was in grad school - "I can make the code run arbitrarily fast if it doesn't have to be correct." [Mary Shaw]
> [BB] I hope I've explained that, even if risks are high (of self-interest, accidents or malice), it is not the IETF's role to mandate that implementers protect against these risks.

	[SM] The whole premise of the NQB draft is that NQB-ness is obeservable data:
"In an unmanaged
   environment, how can networks trust endpoint marking, and why
   wouldn't all applications mark their packets as NQB?

   As an answer the last question, it is worthwhile to note that the NQB
   designation and marking would be intended to convey verifiable
   traffic behavior, not needs or wants."

Requiring "verifiable behavior" and then not actually verifying it indicates that either the requirement or the enforcement policy is wrong and should be changed, you seem to argue for changing the NQB requirement then?

	Sebastian

> 
> 
> Bob
> 
>> 
>> Thanks, --David
>> 
>> 
>>> -----Original Message-----
>>> From: Sebastian Moeller 
>>> <[e-mail]>
>>> 
>>> Sent: Tuesday, September 3, 2019 5:15 AM
>>> To: Bob Briscoe
>>> Cc: Black, David; 
>>> [e-mail]
>>> 
>>> Subject: Re: [tsvwg] Traffic protection as a hard requirement for NQB (was:
>>> WG adoption of draft-white-tsvwg-nqb!)
>>> 
>>> 
>>> [EXTERNAL EMAIL]
>>> 
>>> Dear Bob,
>>> 
>>> allow me to chime in.
>>> 
>>> 
>>>> On Sep 2, 2019, at 16:47, Bob Briscoe <[e-mail]>
>>>>  wrote:
>>>> 
>>>> David,
>>>> 
>>>> Thanks for your closing remarks on the NQB adoption call.
>>>> You say your last point is open for discussion, so I will dive straight in to start 
>>>> that discussion.
>>>> 
>>>> On 30/08/2019 16:40, Black, David wrote:
>>>> 
>>>>> 	• [snip]
>>>>> 	• The criticisms on this list of the “queue protection” requirement in
>>>>> 
>>> the draft are largely accurate.   The draft needs at least an Editor’s Note that
>>> this material will be revised, as while the DOCSIS mechanism is an example of
>>> how to do queue protection, it is not appropriate to require implementation
>>> of that mechanism.   A plausible plan that I have discussed with the authors is
>>> to write a set of functional/behavioral requirements for NQB “traffic
>>> protection” that can be satisfied by a “queue protection” mechanism such as
>>> the DOCSIS mechanism, or by a suitably configured FQ AQM implementation.
>>> [snip]
>>> 
>>>>> In addition, related to item 2), my expectation (which is open to further
>>>>> 
>>> discussion) that “traffic protection” will be a “MUST” requirement, perhaps
>>> with some well-specified exceptions (including explanations of why the
>>> exceptions are ok).   This is because “traffic protection” (e.g., “queue
>>> protection” or a suitably configured FQ AQM) appears to be necessary in
>>> general to keep queue-building traffic out of the NQB traffic aggregate, as
>>> allowing such traffic degrades the properties of the NQB PHB.
>>> 
>>>> I think we should be wary of making traffic protection a hard requirement
>>>> 
>>> at such an early stage in our knowledge of the NQB behaviour. I believe this
>>> is a case where the market, not the IETF, ought to decide whether protection
>>> is required.
>>> 
>>> 	[SM] The draft says "... it is worthwhile to note that the NQB
>>> designation and marking would be intended to convey verifiable traffic
>>> behavior, not needs or wants." in the light of this requirement it seems
>>> obvious that a hop willing to honor that DSCP should/must actually verify the
>>> traffic behavior, no? Requiring behavior to according to a set of requirements
>>> but not enforcing these requirements seems very very optimistic.
>>> 
>>> 
>>> 
>>>> The draft claims incentives can be aligned by an implementation being 
>>>> arranged to ensure that NQB traffic benefits from NQB marking and QB
>>>> traffic benefits from QB marking. Incentives are hard to guess, so that may or
>>>> may not be true. However, I don't think we (the tsvwg/IETF) can state
>>>> categorically that it is not true.
>>>> 
>>>> 	[SM] I would have thought it would be on those claiming a laxer
>>>> enforcement model to proof that that model is sufficient?
>>>> 
>>>> 
>>>> The draft makes the point that, even if incentives are aligned, queue-
>>>> building traffic could be mismarked as NQB, either accidentally or maliciously.
>>>> That's a sound reason for an implementer to include traffic protection, but I
>>>> don't think it's a good reason for us (the tsvwg/IETF) to require them to.
>>>> 
>>>> While there is no operational experience of NQB deployments, I think the
>>>> market (i.e. most early-adopting operators) will want the warm feeling of
>>>> some form of traffic protection. But as we get more experience we might
>>>> find incentives really are aligned. And we might find accidents and malice are
>>>> not a significant problem.
>>>> 
>>>> 	[SM] @ the chairs, how hard would it be to retroactively change from
>>>> a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be
>>>> nice, but if it is hard I would vote for a MUST as that seems to be the safer
>>>> option.
>>>> 
>>>> 
>>>> So I think the current 'SHOULD' is the right call. It could be beefed up with
>>>> 
>>> warnings on the risks of not providing protection - not least the risk no early
>>> adopter will want to use such an implementation.
>>> 
>>>> 
>>>> 
>>>> As an analogy, when TCP congestion control was first developed, it was
>>>> known that end-systems could run a subverted TCP algorithm or just use
>>>> unresponsive UDP. At that time, the view could have been taken that per-
>>>> flow scheduling would have to be a 'MUST' requirement for all Internet
>>>> bottlenecks.
>>>> 
>>>> As it has turned out, the Internet does have /per-user/ scheduling at
>>>> bottlenecks,
>>>> 
>>>> 	[SM] Question is this universally true? I am not 100% sure. My access
>>>> link is limited by my contracted rate, but due to oversubscription there is no
>>>> guarantee that on a congested link between my home and my ISP's traffic-
>>>> shaper/the wider internet I get a share that reflects my "per-user" fraction of
>>>> the shared capacity.
>>>> 	I fail to see any scheduler beyond my ISPs traffic shaper that has a
>>>> wholistic-enough view to classify traffic based on "user" (think NATed IPv4
>>>> address, but variable length IPv6 prefixes, plus a router's ipv6/64), could you
>>>> elaborate please?
>>>> 
>>>> 
>>>> but there has been little need for /per-flow/ scheduling for capacity sharing
>>>> (yes, FQ exists, but it's not needed for capacity sharing).
>>>> 
>>>> In the TCP case, it turned out that a delicate balance of incentives proved
>>>> sufficient to allow most Internet equipment to be simpler and cheaper.
>>>> There is a poorly understood balance of incentives in the NQB case. So let's
>>>> not require equipment to be more complex than it might need to be, at least
>>>> not yet.
>>>> 
>>>> 	[SM] I believe for malicious actors NQB will be a attractive DSCP as it
>>>> promised to allow doing harm even at low bandwidth (just send a low
>>>> average rate in lumpy bursts and I would expect an NQB-honoring L4S
>>>> scheduler to get into trouble*, without requiring a large offensive traffic
>>>> load, keeping it cheap and hard to detect yet sufficiently disruptive to rob the
>>>> low latency queue of its intended functionality).
>>>> 
>>>> 
>>>> *) This is a hypothesis I have not confirmed in any way, but it seems to be in
>>>> line with how I understand dual queue aqm to work (so it might be more of a
>>>> reflection of my level of understanding and not so much of dual queue aqm).
>>>> Please correct me if this seems wrong.
>>>> 
>>>> 
>>>> u
>>>> 
>>>> Bob
>>>> 
>>>> 
>>>> 
>>>>> Thanks, --David (TSVWG co-chair, will be shepherd for NQB draft).
>>>>> ----------------------------------------------------------------
>>>>> David L. Black, Senior Distinguished Engineer
>>>>> Dell EMC, 176 South St., Hopkinton, MA  01748
>>>>> [phone] New    Mobile: [phone]
>>>>> 
>>>>> [e-mail]
>>>>> 
>>>>> ----------------------------------------------------------------
>>>>> 
>>>>> 
>>>> --
>>>> 
>>>> 
>>> __________________________________________________________
>>> ______
>>> 
>>>> Bob Briscoe
>>>> 
>>>> http://bobbriscoe.net/
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               
> http://bobbriscoe.net/