Re: [tsvwg] Traffic protection as a hard requirement for NQB

[Bob Briscoe <[e-mail]>]

Jonathan,

On 05/09/2019 20:52, Jonathan Morton wrote:
>> On 5 Sep, 2019, at 9:23 pm, Bob Briscoe <[e-mail]> wrote:
>>
>> Config:
>> 	• Scheduler:
>> 		• WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least 60Mb/s for NQB flows.
>> 		• Scheduler quantum: 1500B.
>> 	• Buffering:
>> 		• The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
>> 		• The QB buffer is deeper (say 200ms) with an AQM target delay of say 10ms.
> This seems like a reasonable implementation, on the face of it.
>
>> Now, why do you think "the resulting latency is (at least initially) lower" for a QB flow that marks itself NQB? Why do you think incentives are misaligned (i.e. a tragedy of the commons)?
> Okay, let's extend your traffic scenario a bit by assuming there's a significant amount of QB traffic, perhaps from a number of 50GB game updates in progress to various PCs and consoles in the household.  This could be as many as a couple of dozen bulk, saturating flows at a time, with the load being more-or-less continuous for an hour or so.  This traffic originates from gaming-related companies, so they know better than to risk interfering with actual gaming traffic carrying the NQB marking.
>
> The above PHB template does a good job of isolating the NQB traffic from the BE traffic in this case, in which everyone behaves nicely.
>
> But now let's introduce an adversary; let's call them Netflix' Unscrupulous Rival (NUR).  They are not in the gaming (interactive entertainment) business, but Video On Demand (passive entertainment).  Their USP is that they get the complete video file onto the subscriber's PC in the minimum possible time; this incidentally also removes the load from their servers as early as possible.  In short, they have chugged the entire cask of Flow Completion Time koolaid, and their flows are multiple gigabytes each.
>
> In service to this goal, NUR have selected BBRv2 as their CC algorithm because, lacking the traditional TCP sawtooth, it achieves higher throughput than most, and without needing to open multiple connections (which means reduced server load and client software complexity).  And, because they're unscrupulous and literally don't care about competing traffic - how important can it be if they're busy watching our video? - they've increased the packet loss threshold to 10% from the default of 1%; they don't mind retransmitting a few packets if it gets the job done faster.  Since we may assume that the AQM in this PHB implements RFC-3168 ECN, this tweak doesn't actually have much effect on this particular case, but it does elsewhere, on the many networks still using dumb FIFOs.
>
> What NUR now notices is that, at some times and with some particular subscribers, their throughput is maybe a tenth of what they expect.  This is unacceptable, so they investigate.  And then they switch on NQB marking to see what happens.  After all, BBRv2 is advertised as not building a queue, so it's allowed, right?
>
> This gives them almost uncontended access to the 60Mbps reserved for NQB traffic, instead of having to share the 120Mbps total pipe with a couple of dozen other bulk flows.  NUR is ecstatic and rolls it out across their entire system.
>
> And there's your incentive to mis-mark as NQB.
>
> Now, what effect does this have on the actual NQB traffic?  Well, BBRv2 spends most of its time pacing at just below the detected path capacity, but periodically probes for additional bandwidth by pacing at a higher rate for an RTT or so.  Usually this will exceed the capacity allocated to NQB and start queuing, which imposes some delay both on itself and on other NQB traffic sharing the same queue.  This will cap out at 6ms (30 packets at 60Mbps) - ten times the worst-case delay you calculated for NQB traffic alone - at which point packet loss will begin to occur.
[BB]: Picky point: I said NQB has 3ms of physical buffer.
> Because the NUR flow is very long and not application-limited, tail loss is not a factor for most of its lifetime, and they have configured BBRv2 to be exceptionally tolerant of loss before initiating congestive backoff.
>
> In short, NQB traffic may periodically experience an additional 6ms delay and 10% packet loss due to this mis-marked NUR flow, depending on some other factors.
>
> As for the "tragedy of the commons", NUR then proudly publishes an investor report and a white paper explaining how they quintupled their throughput with this one weird trick.  Other unscrupulous Internet companies take note of this and try it themselves, usually with less competence and without measuring the actual extent to which they can expect improvements in practice.  After all, there's no law against it, and it worked for NUR didn't it?  They're a big famous Internet company so they must be right!
>
> Consequently the NQB queue becomes increasingly full of QB traffic.  The latency advantage of the NQB marking is thus eroded, with 6ms delays and significant packet loss becoming the rule rather than the exception, not all that much better than the 10ms target delays on the other side of the scheduler.
>
> Do you now see?
Yes, this is a possible attack, altho some picky details are 
questionable, e.g. I think BBRv2 will run significantly under 60Mb/s 
because of the interaction between its bandwidth probes and the very 
shallow buffer (that can be tried experimentally). But in general I 
admit there will be a gain to the attacker here.

However, you cannot put this solely down to incentive misalignment. It 
is an active attack, because NUR has tweaked BBRv2's loss tolerance from 
1% to 10%. That in itself would give a QB flow significantly higher 
throughput than other competing QB flows (whether Cubic or BBRv2).

And you're basing NUR's attack on BBRv2's exploitation of a min loss 
threshold (for whatever value of threshold is chosen), as if BBRv2 
cannot be the cause of the tragedy of the commons because Google asserts 
it is not, then applying it to an NQB queue allows you to blame NQB for 
the tragedy. Rather a disingenuous argument isn't it?

Wouldn't it be sufficient to use a Cubic flow or even Reno with NQB, 
which would get about 75% of 60Mb/s, which would be higher than getting 
1/24 of 120Mb/s with QB. It would even be higher than 33% of 120Mb/s if 
there were only two long-running flows in the QB queue.

Nonetheless, you have not considered the question of how often NUR's 
tweak will make throughput worse by using NQB, in all the cases where 
the QB queue has 0 or 1 long-running flow in it (likely the more 
prevalent cases for most users). Wouldn't users report this in the 
forums which would put others off trying it out?

More generally, attempts to take more throughput at the expense of other 
applications of the same customer, have come and gone over the years, 
and so far none has taken hold. Again, probably because users report the 
negative effects on your own other apps in the forums.

I'm not saying none will ever take hold. Just as I'm not trying to 
downplay the risks of attacks against NQB.

My argument is that implementers can decide whether traffic protection 
is worthwhile, and it's not the IETF's place to tell them to.



Bob

>   - Jonathan Morton

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/