Re: [tsvwg] Traffic protection as a hard requirement for NQB

[Bob Briscoe <[e-mail]>]

David & Sebastien,

[These responses cover come of Sebastien''s points that were common to 
David's reply as well - I'll pick out the remaining ones specific to 
Sebastien's email in a subsequent response. Sorry for the long response, 
but I've tried hard to cover this rigorously.]

On 03/09/2019 22:46, Black, David wrote:
> [Writing as an individual]
>
> The core of my current views is this:
>
>> ... “traffic protection” (e.g., “queue
>> protection” or a suitably configured FQ AQM) appears to be necessary in
>> general to keep queue-building traffic out of the NQB traffic aggregate, as
>> allowing such traffic degrades the properties of the NQB PHB.
> I see serious potential here for a "tragedy of the commons" outcome where lots of QB traffic is sent marked NQB because the resulting latency is (at least initially) lower.


      ===Misunderstanding?==


Using a term like 'tragedy of the commons', implies your argument might 
be based on a different understanding of what NQB is to mine. I'm trying 
to unpick what you think NQB is from your sentence "the resulting 
latency is (at least initially) lower" for a QB flow that marks itself NQB?

Please confirm that you (both) understand that, with NQB, the low 
latency is not provided by scheduling in the network. It is primarily a 
property of NQB traffic sources, and the only necessary property of the 
network's per-hop behaviour for NQB is to isolate NQB from QB traffic 
(i.e. in a separate queue for the NQB class). There is no priority 
scheduling requirement.

[A comment I would make to the authors about the draft: it needs to give 
some example approaches to scheduling between NQB and QB, or at least 
talk about this scheduling.]

>>> I think we should be wary of making traffic protection a hard requirement
>>> at such an early stage in our knowledge of the NQB behaviour. I believe this
>>> is a case where the market, not the IETF, ought to decide whether protection
>>> is required.
> I want to see clear and convincing evidence that the "tragedy of the commons" outcome is highly unlikely, which this text ...
>
>>> The draft claims incentives can be aligned by an implementation being arranged to ensure that NQB traffic benefits from NQB marking and QB
>>> traffic benefits from QB marking. Incentives are hard to guess, so that may or
>>> may not be true. However, I don't think we (the tsvwg/IETF) can state
>>> categorically that it is not true.
> ... is not.
>
> The above text not only reverses the burden of proof but also changes the question from whether a "tragedy of the commons" is possible to whether it will always (categorically) happen - the resulting straw proposition is then easy to demolish.  Nice try, no sale ...
[BB] OK, I'll turn the burden of proof back round again...

As this insertion is long, I've used sub-headings.


      ==Incentive Alignment?==


To judge whether there's any tragedy of the commons (incentive 
alignment), I'll put up a straw-man NQB configuration that complies with 
the following requirement in the draft:

    a useful property of nodes that support separate queues for NQB and QB
    flows would be that for NQB flows, the NQB queue provides better
    performance (considering latency, loss and throughput) than the QB
    queue; and for QB flows, the QB queue provides better performance
    (considering latency, loss and throughput) than the NQB queue.

_Background_: NQB has become possible largely because Internet access 
link rates have typically become fast enough that the serialization 
delay of a packet can be sub-millisecond, and therefore a queue of a few 
packets introduces delay that is small relative to other non-optional 
sources of delay like propagation. In these cases we no longer need 
priority scheduling for low delay._

Config_:

  * Scheduler:
      o WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at
        least 60Mb/s for NQB flows.
      o Scheduler quantum: 1500B.
  * Buffering:
      o The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
      o The QB buffer is deeper (say 200ms) with an AQM target delay of
        say 10ms.


_Traffic_: Let's introduce some example NQB traffic:

  * 2 paced flows of VoIP datagrams, each avg 50B payload plus 58B of
    Eth/IP/UDP/RTP headers at 33pkt/s
      o bit-rate: 29kb/s /flow
      o serialization delay: 7.2us / pkt
  * 2 streams of 1000B game sync datagrams at 30pkt/s =>
      o bit-rate: 240kb/s /flow
      o serialization delay: 67us / pkt
  * plus occasional DNSSec datagrams, worst case 1500B
      o serialization delay: 100us / pkt
  * Perhaps 540kb/s in all, which is about 0.9% of the min NQB capacity.

_Worst-case NQB queuing delay calculation_ for the above traffic model:
Each NQB flow paces out its own packets, but one might happen to arrive 
while a packet from any of the other NQB flows is already queued. Worst 
case n-1 = 4 other NQB packets queued, where n is the number of 
application flows. And if there's traffic in the QB queue, each NQB 
packet will have to wait for one quantum (100us) while the other queue 
is served. Worst-case NQB delay is then:

    (67us * 2 + 7.2us + 100us) + (100us * 4) = 641us,

It's v unlikely that this worst-case would arise, but it gives an 
indication of where the tail of the delay distribution will lie.

_Tragedy?_
Now, why do you think "the resulting latency is (at least initially) 
lower" for a QB flow that marks itself NQB? Why do you think incentives 
are misaligned (i.e. a tragedy of the commons)?

Certainly, as the draft says, there is a danger of QB being mismarked 
due to accidents or malice. However, there's no /incentive/ to mismark 
to /get/ lower latency, in the strawman config above.

  * If a QB flow mismarked as NQB causes higher latency, it will cause
    it to itself.
  * If a long-running flow behaves in a QB way, one assumes it needs to
    build a queue of perhaps 10ms for full utilization, so in a 3ms
    buffer it will experience high loss and badly under-utilize the link.
  * If a short QB flow mismarked itself as NQB, it would experience its
    self-induced latency during start-up. Nonetheless, If there already
    happens to be a long-running QB flow in the QB queue, it would
    indeed get lower delay than if it marked itself truthfully and added
    to the QB queue. But, with the shallow NQB buffer, the mismarked
    flow would find it very hard to avoid high loss, especially tail loss.


This last point is what I was thinking of when I said "Incentives are 
hard to guess". Even if this was tried experimentally, it would be hard 
to know whether the harm from high risk of loss would dominate the 
benefit from the reduction in delay. But if one accepts that a loss is 
usually equivalent to more than 1RTT of delay, it seems there is no 
incentive for QB flows to mismark.

> or as Sebastian writes:
>
>> 	[SM] I would have thought it would be on those claiming a laxer
>> enforcement model to proof that that model is sufficient?
> I agree.  As for requirements language ...
>
>>> While there is no operational experience of NQB deployments, I think the market (i.e. most early-adopting operators) will want the warm feeling of
>>> some form of traffic protection. But as we get more experience we might
>>> find incentives really are aligned. And we might find accidents and malice are
>>> not a significant problem.
>>>
>>> 	[SM] @ the chairs, how hard would it be to retroactively change from
>>> a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be
>>> nice, but if it is hard I would vote for a MUST as that seems to be the safer
>>> option.
> ... a common IETF approach to this sort of uncertain situation is "MUST implement, SHOULD use" with discussion of when the "SHOULD" may not apply and/or possible consequences of ignoring the "SHOULD."  The underlying rationale is that the functionality will be available for use if/when the need for it is discovered in a fashion that surprises the network operator.  It is much easier to relax "MUST implement" based on implementation experience than it is to upgrade "SHOULD implement" as the former doesn't have the potential to break any "running code" implementations.
[BB] As this insertion is long, I've used sub-headings...


      ==Incentive Alignment was the point of NQB==


As I understand it, the latest NQB draft was a merge of:

  * NQB from the DOCSIS community
  * and LoLa from the GSMA community.

The whole point of LoLa was to trade off loss and latency incentives to 
avoid the complexities of Diffserv policing. The point of NQB also 
concerned incentives, but avoided the LoLa tradeoff between lower 
latency and higher loss. Instead, both QB and NQB got better performance 
by marking themselves truthfully than they did by mismarking.

1/ Incentives are a first order concern. If incentives are not aligned, 
that affects all typical cases, because every party can be assumed to be 
acting in its own interests.

2/ The NQB draft introduces traffic protection in the context of 
accidents and malice. They are second order concerns, i.e. not expected 
to be the common case, but still potentially of concern.

I might be more sympathetic to "MUST implement" if incentives were not 
aligned. But even if incentives were misaligned, I don't think it would 
be normal or appropriate for the IETF to mandate protection...


      ==MUST is not for Nannying==


Quoting the bible on the use of MUST [RFC2119]:

    Imperatives of the type defined in this memo must be used with care
    and sparingly.  In particular, they MUST only be used where it is
    actually required for interoperation or to limit behavior which has
    potential for causing harm (e.g., limiting retransmissions)  For
    example, they must not be used to try to impose a particular method
    on implementors where the method is not required for
    interoperability.

a) There is certainly no interoperability concern here.

b) "Limiting behaviour which has potential to cause harm" means limiting 
harm to others; it does not mean it is appropriate to mandate that an 
NQB implementation MUST limit harm to itself. That is nannying. 
Implementers are perfectly able to make that decision for themselves.

For instance, regular Diffserv with prioritization has much stronger 
protection concerns than NQB because, not only is there potential for 
accidents and malice, but incentives are not aligned. Nonetheless, 
Diffserv was divided into standards track RFC2474 about the DS field 
(interop), and informational RFC2475 about domain structure, policing, 
etc. There is no normative text in RFC2475 and I don't think the IETF 
ever mandated that a border router MUST implement Diffserv policing.


      ==Continual Run-time Protection is Unnecessary==


Even if we did want to mandate protection, it would be inappropriate to 
mandate continual run-time protection. Because it is common for 
accidents and malice to be weeded out before run-time.

For instance, in the upstream of a mobile network, the RAN scheduler 
only protects one user's traffic from another's. It would not protect 
one NQB app of a user from another NQB app of the same user. To do that, 
a step could be added to the app-store validation process to verify 
correct use of NQB by an app.

Similarly, on other platforms virus checkers weed out malicious software 
at installation time.


      ==Mandating a Layering Violation?==


Saying "MUST implement" some form of traffic protection would turn NQB 
from a simple L2 or L3 class-based queue isolation scheme (with a 
"SHOULD" recommendation to access L4) into an approach that is 
non-compliant unless it accesses L4 headers.

Low latency is a big deal in 5G. The 3GPP complied with normal network 
layering principles when it required the 5G RLC layer to compress and 
encrypt its payload (the DSCP at L3 gets translated into a 5QI in the 
RLC header at L2, which is equivalent of a QCI in LTE).

How will it look to the 3GPP if the IETF introduces a low latency DSCP 
that mandates access to L4 headers in order to use it?


>
>>> As an analogy, when TCP congestion control was first developed, it was
>> known that end-systems could run a subverted TCP algorithm or just use
>> unresponsive UDP. At that time, the view could have been taken that per-
>> flow scheduling would have to be a 'MUST' requirement for all Internet
>> bottlenecks.   As it has turned out, the Internet does have /per-user/
>> scheduling at bottlenecks,
> I understand the TCP congestion control history, and the even-more-relevant ECN for TCP history, where there was a serious potential for a "tragedy of the commons" outcome - ignoring ECN congestion indications was a potential new way to cause congestion collapse of the Internet (fortunately, that did not happen).  There are at least a couple of aspects that seem rather different in the current situation:
> - At the time, the IETF community had strong influence over the important TCP stacks.
> 	The network stacks have gotten much more diverse since then, and a lot of NQB traffic
> 	can be expected to come from applications that have a much weaker relationship to the IETF community.
> - Switching away from TCP to UDP was a high hurdle, e.g., as TCP uses stream sockets, whereas UDP uses datagram sockets, and ECN functionality was in the core of TCP implementations.
> 	The hurdle here is much lower, particularly for applications that already use UDP.
[BB] That's all true. Yes, it increases the risk, particularly of 
accidents and bugs.

Don't get me wrong, I'm not trying to argue down the risk here. I'm 
saying it's not the IETF's role to mandate what to do about the risk (no 
matter how high it is). The arguments above against nannying and 
mandating self-protection all still apply.

>
>>> In the TCP case, it turned out that a delicate balance of incentives proved
>>> sufficient to allow most Internet equipment to be simpler and cheaper.
>>> There is a poorly understood balance of incentives in the NQB case. So let's
>>> not require equipment to be more complex than it might need to be, at least
>>> not yet.
> I concur with Sebastian that the onus is on the proposers of the complexity/cost savings to demonstrate that the resulting designs are safe for the Internet.  I'm reminded by analogy of something that my software engineering professor said back when I was in grad school - "I can make the code run arbitrarily fast if it doesn't have to be correct." [Mary Shaw]
[BB] I hope I've explained that, even if risks are high (of 
self-interest, accidents or malice), it is not the IETF's role to 
mandate that implementers protect against these risks.


Bob

>
> Thanks, --David
>
>> -----Original Message-----
>> From: Sebastian Moeller <[e-mail]>
>> Sent: Tuesday, September 3, 2019 5:15 AM
>> To: Bob Briscoe
>> Cc: Black, David; [e-mail]
>> Subject: Re: [tsvwg] Traffic protection as a hard requirement for NQB (was:
>> WG adoption of draft-white-tsvwg-nqb!)
>>
>>
>> [EXTERNAL EMAIL]
>>
>> Dear Bob,
>>
>> allow me to chime in.
>>
>>> On Sep 2, 2019, at 16:47, Bob Briscoe <[e-mail]> wrote:
>>>
>>> David,
>>>
>>> Thanks for your closing remarks on the NQB adoption call.
>>> You say your last point is open for discussion, so I will dive straight in to start
>>> that discussion.
>>> On 30/08/2019 16:40, Black, David wrote:
>>>> 	• [snip]
>>>> 	• The criticisms on this list of the “queue protection” requirement in
>> the draft are largely accurate.   The draft needs at least an Editor’s Note that
>> this material will be revised, as while the DOCSIS mechanism is an example of
>> how to do queue protection, it is not appropriate to require implementation
>> of that mechanism.   A plausible plan that I have discussed with the authors is
>> to write a set of functional/behavioral requirements for NQB “traffic
>> protection” that can be satisfied by a “queue protection” mechanism such as
>> the DOCSIS mechanism, or by a suitably configured FQ AQM implementation.
>> [snip]
>>>> In addition, related to item 2), my expectation (which is open to further
>> discussion) that “traffic protection” will be a “MUST” requirement, perhaps
>> with some well-specified exceptions (including explanations of why the
>> exceptions are ok).   This is because “traffic protection” (e.g., “queue
>> protection” or a suitably configured FQ AQM) appears to be necessary in
>> general to keep queue-building traffic out of the NQB traffic aggregate, as
>> allowing such traffic degrades the properties of the NQB PHB.
>>> I think we should be wary of making traffic protection a hard requirement
>> at such an early stage in our knowledge of the NQB behaviour. I believe this
>> is a case where the market, not the IETF, ought to decide whether protection
>> is required.
>>
>> 	[SM] The draft says "... it is worthwhile to note that the NQB
>> designation and marking would be intended to convey verifiable traffic
>> behavior, not needs or wants." in the light of this requirement it seems
>> obvious that a hop willing to honor that DSCP should/must actually verify the
>> traffic behavior, no? Requiring behavior to according to a set of requirements
>> but not enforcing these requirements seems very very optimistic.
>>
>>
>>> The draft claims incentives can be aligned by an implementation being
>>> arranged to ensure that NQB traffic benefits from NQB marking and QB
>>> traffic benefits from QB marking. Incentives are hard to guess, so that may or
>>> may not be true. However, I don't think we (the tsvwg/IETF) can state
>>> categorically that it is not true.
>>>
>>> 	[SM] I would have thought it would be on those claiming a laxer
>>> enforcement model to proof that that model is sufficient?
>>>
>>> The draft makes the point that, even if incentives are aligned, queue-
>>> building traffic could be mismarked as NQB, either accidentally or maliciously.
>>> That's a sound reason for an implementer to include traffic protection, but I
>>> don't think it's a good reason for us (the tsvwg/IETF) to require them to.
>>> While there is no operational experience of NQB deployments, I think the
>>> market (i.e. most early-adopting operators) will want the warm feeling of
>>> some form of traffic protection. But as we get more experience we might
>>> find incentives really are aligned. And we might find accidents and malice are
>>> not a significant problem.
>>>
>>> 	[SM] @ the chairs, how hard would it be to retroactively change from
>>> a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be
>>> nice, but if it is hard I would vote for a MUST as that seems to be the safer
>>> option.
>>>
>>> So I think the current 'SHOULD' is the right call. It could be beefed up with
>> warnings on the risks of not providing protection - not least the risk no early
>> adopter will want to use such an implementation.
>>>
>>>
>>> As an analogy, when TCP congestion control was first developed, it was
>>> known that end-systems could run a subverted TCP algorithm or just use
>>> unresponsive UDP. At that time, the view could have been taken that per-
>>> flow scheduling would have to be a 'MUST' requirement for all Internet
>>> bottlenecks.
>>> As it has turned out, the Internet does have /per-user/ scheduling at
>>> bottlenecks,
>>>
>>> 	[SM] Question is this universally true? I am not 100% sure. My access
>>> link is limited by my contracted rate, but due to oversubscription there is no
>>> guarantee that on a congested link between my home and my ISP's traffic-
>>> shaper/the wider internet I get a share that reflects my "per-user" fraction of
>>> the shared capacity.
>>> 	I fail to see any scheduler beyond my ISPs traffic shaper that has a
>>> wholistic-enough view to classify traffic based on "user" (think NATed IPv4
>>> address, but variable length IPv6 prefixes, plus a router's ipv6/64), could you
>>> elaborate please?
>>>
>>> but there has been little need for /per-flow/ scheduling for capacity sharing
>>> (yes, FQ exists, but it's not needed for capacity sharing).
>>> In the TCP case, it turned out that a delicate balance of incentives proved
>>> sufficient to allow most Internet equipment to be simpler and cheaper.
>>> There is a poorly understood balance of incentives in the NQB case. So let's
>>> not require equipment to be more complex than it might need to be, at least
>>> not yet.
>>>
>>> 	[SM] I believe for malicious actors NQB will be a attractive DSCP as it
>>> promised to allow doing harm even at low bandwidth (just send a low
>>> average rate in lumpy bursts and I would expect an NQB-honoring L4S
>>> scheduler to get into trouble*, without requiring a large offensive traffic
>>> load, keeping it cheap and hard to detect yet sufficiently disruptive to rob the
>>> low latency queue of its intended functionality).
>>>
>>>
>>> *) This is a hypothesis I have not confirmed in any way, but it seems to be in
>>> line with how I understand dual queue aqm to work (so it might be more of a
>>> reflection of my level of understanding and not so much of dual queue aqm).
>>> Please correct me if this seems wrong.
>>>
>>> u
>>>
>>> Bob
>>>
>>>
>>>> Thanks, --David (TSVWG co-chair, will be shepherd for NQB draft).
>>>> ----------------------------------------------------------------
>>>> David L. Black, Senior Distinguished Engineer
>>>> Dell EMC, 176 South St., Hopkinton, MA  01748
>>>> [phone] New    Mobile: [phone]
>>>> [e-mail]
>>>> ----------------------------------------------------------------
>>>>
>>> --
>>>
>> __________________________________________________________
>> ______
>>> Bob Briscoe
>>> http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/