Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-19.txt - BBR

[[e-mail]]

Hi Roland,

thanks for the references given. As you also mention, Neil pointed out to be aware that no queue pressure and low queue pressure are significantly different things, which I respect.

I'm also glad that you confirm that 8*60[ms]@100Mbit/s may be an unusually deep buffer.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Bless, Roland (TM) <[e-mail]> 
Gesendet: Mittwoch, 2. August 2023 20:36
An: Geib, Rüdiger <[e-mail]>; [e-mail]
Cc: [e-mail]
Betreff: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-19.txt - BBR

Hi,

On 02.08.23 at 11:19 [e-mail] wrote:
> thanks. Some questions:
> - is BBR "work in progress" and statements referring to it best should mention a version (e.g., BBRv2 and earlier versions...)?
> - Roland, could you provide a reference for insertion in draft NQB?

For BBRv1 I would shamelessly ;-) recommend our own paper M. Hock, R. Bless and M. Zitterbart, "Experimental evaluation of BBR congestion control," 2017 IEEE 25th International Conference on Network Protocols (ICNP), Toronto, ON, Canada, 2017, pp. 1-10, doi: 
10.1109/ICNP.2017.8117540.
https://ieeexplore.ieee.org/document/8117540
or
https://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
as it has several plots showing the overshoot during BBRv1's startup phase, especially in large buffers. There is also D. Scholz, B. Jaeger, L. Schwaighofer, D. Raumer, F. Geyer and G. Carle, "Towards a Deeper Understanding of TCP BBR Congestion Control," 2018 IFIP Networking Conference (IFIP Networking) and Workshops, Zurich, Switzerland, 2018, pp. 1-9, doi: 10.23919/IFIPNetworking.2018.8696830.
https://dl.ifip.org/db/conf/networking/networking2018/2B1-1570416152.pdf


For BBRv2
Y. -J. Song, G. -H. Kim, I. Mahmud, W. -K. Seo and Y. -Z. Cho, "Understanding of BBRv2: Evaluation and Comparison With BBRv1 Congestion Control Algorithm," in IEEE Access, vol. 9, pp. 37131-37145, 2021, doi: 
10.1109/ACCESS.2021.3061696. https://ieeexplore.ieee.org/document/9361674
has some nice plots, too.

> I found https://datatracker.ietf.org/meeting/117/materials/slides-117-ccwg-bbrv3-algorithm-bug-fixes-and-public-internet-deployment-00, contains:
> - So most test results for BBRv2 will not apply to BBRv3
> - Primary impact of these changes:
>    Lower queuing delays and packet loss rates during and shortly after 
> STARTUP
> 
> I don't claim that to be true and don't know, whether that has yet been validated independently. If I'm to have trust in Greg's work, I'd have to have trust into this presentation too.

Neal answered already quite clearly that for BBRv3, especially multiple flows "cause repeated/sustained queue pressure" and that the queue is "in the neighborhood of 1.5*BDP", which is sustained queue pressure that we analytically derived for BBRv1 already.

>  From all that I learned about networking, in the absence of admission control no sender is able to predict whether its initial traffic sent to a destination may cause congestion at the bottleneck. This, I think, holds independent of a DSCP chosen to mark that traffic. It is more important to me to understand, whether a BBR(v3) flow repeatedly causes queue-pressure. NQB, as Greg emphasized, provides low jitter and loss on a statistical basis *only* (and it *doesn't* provide throughput guarantees). A single, brief period of congestion every such and such minutes won't significantly impact jitter and loss statistics, if these were measured by a number of probes allowing stat-calculation with high confidence.
> 
> Roland, I agree, bottleneck buffer size and the number of flows impact buffer dimensioning and the buffer depth impacts the performance experienced by the flows. You mention an 8 BDP buffer, is that 8*60[ms]*100 Mbit/s? If yes, I'm astonished. Is that a standard Internet config - or ,e.g., WiFi specific? 8*20ms@100Mbit/s would be closer to my expectations.

Yes, we used this as an academic extreme at one end of the spectrum. 
However, since minRTTs may vary one flow's BDP may be different from another flow's BDP and a huge looking buffer may become small for flows with larger minRTTs.

Regards,
  Roland

> -----Ursprüngliche Nachricht-----
> Von: Bless, Roland (TM) <[e-mail]>
> Gesendet: Mittwoch, 2. August 2023 10:22
> An: Geib, Rüdiger <[e-mail]>; [e-mail]
> Cc: [e-mail]
> Betreff: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-19.txt - BBR
> 
> Hi Rüdiger,
> 
> On 02.08.23 at 08:25 [e-mail] wrote:
>> thanks. My point is, draft-NQB claims BBR to "generally overshoot the available capacity, causing a queue to build up at the bottleneck link". This contradicts the BBR design aim, as formulated by the BBR authors "..without causing excessive queue pressure", and it seems to contradict a statement to be found within an RFC co-authored by Greg too. In the NQB draft, Greg puts down a negative evaluation of BBR, but does neither provide a reference, nor measurement results backing his assessment. A non-informed reader may take that as whatever BBR claims, they failed. I'd expect a reference for statements like that at least. Or removal.
>>
> 
> It is correct that BBRv1 and BBRv2 may overshoot considerably during their startup phase as other congestion controls would also do during their slow start phase. Although BBRv3 has a changed startup phase, I guess that the behavior is still somewhat similar. In general, BBR's behavior also depends on the bottleneck buffer size and the number of flows present. In larger buffers BBRv2 still caused considerable queueing delays that would not fit NQB's goals (e.g., we measured two flows 20ms and 60ms RTT, 100 Mbit/s bottleneck capacity, 8 BDP buffer caused 40–60ms queuing delay). This was usually better in smaller buffers as inflight_hi was set early due to packet loss >%2.
> However, newly starting BBR flows would probably regularly exceed NQB's buffer capacity and thus causing jitter and loss.
> 
> Regards,
>    Roland
> 
>> -----Ursprüngliche Nachricht-----
>> Von: Bless, Roland (TM) <[e-mail]>
>> Gesendet: Dienstag, 1. August 2023 16:39
>> An: Geib, Rüdiger <[e-mail]>; [e-mail]
>> Cc: [e-mail]
>> Betreff: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-19.txt - BBR
>>
>> Hi Rüdiger,
>>
>> I brought this issue up on 2019/09/13 already (see "Comments on draft-white-tsvwg-nqb-02"
>> https://mailarchive.ietf.org/arch/msg/tsvwg/OY6yFWNUGxukDOwvpMP1Fuoki
>> y E/ for reference). In general, capacity seeking does not 
>> automatically mean that it is queue building (e.g., TCP LoLa tries 
>> limit its queue to 5ms, but it is capacity seeking). BBR cannot always avoid to build a queue, but tries to limit its length, too.
>> As far as I understand the NQB design and draft, NQB flows are application-limited in nature (and thus not capacity seeking), so neither BBR nor LoLa flows are intended to use the NQB PHB.
>>
>> Regards,
>>     Roland
>>
>> On 01.08.23 at 16:09 [e-mail] wrote:
>>> Hi Greg,
>>>
>>> Your draft NQB characterizes BBR as a capacity seeking protocol...causing a queue to build up at the bottleneck link (see below). This is seemingly in contradiction with the design aim of BBR (see below). Further, RFC9330 lists BBR as a "scalable congestion control" based protocol (see below). I note that the latter is co-authored by you. I take the statement of RFC9330 to contradict the standards track NQB judgement of BBR. Could you provide a reference or text explaining to readers, why your latest standards track RFC judges BBR as Queue Building? Alternatively, you could remove BBR from the statement in draft NQB.
>>>
>>> Regards,
>>>
>>> Ruediger
>>>
>>>
>>> https://www.ietf.org/archive/id/draft-ietf-tsvwg-nqb-19.html#name-in
>>> t r oduction Many of the commonly-deployed congestion control 
>>> algorithms, such as ..... BBR, are designed to seek the available 
>>> capacity of the end-to-end path (which can frequently be the access network link capacity), and in doing so generally overshoot the available capacity, causing a queue to build up at the bottleneck link.
>>>
>>> https://datatracker.ietf.org/doc/html/draft-cardwell-iccrg-bbr-conge
>>> s
>>> t
>>> ion-control#name-introduction-7 BBR uses recent measurements of a 
>>> transport connection's delivery rate, round-trip time, and packet loss rate to build an explicit model of the network path, including its estimated available bandwidth, bandwidth-delay product, and the maximum volume of data that the connection can place in-flight in the network without causing excessive queue pressure.
>>>
>>> https://www.rfc-editor.org/rfc/rfc9330.html#name-l4s-architecture-ov
>>> e r view Indeed, between the present document being drafted and 
>>> published, the following Scalable congestion controls were
>>> implemented: ...., and the L4S ECN part of Bottleneck Bandwidth and Round-trip propagation time (BBRv2) [BBRv2] intended for TCP and QUIC transports.
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: tsvwg <[e-mail]> Im Auftrag von 
>>> [e-mail]
>>> Gesendet: Mittwoch, 26. Juli 2023 19:08
>>> An: [e-mail]
>>> Cc: [e-mail]
>>> Betreff: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-19.txt
>>>
>>>
>>> A New Internet-Draft is available from the on-line Internet-Drafts directories. This Internet-Draft is a work item of the Transport Area Working Group (TSVWG) WG of the IETF.
>>>
>>>       Title           : A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services
>>>       Authors         : Greg White
>>>                         Thomas Fossati
>>>       Filename        : draft-ietf-tsvwg-nqb-19.txt
>>>       Pages           : 29
>>>       Date            : 2023-07-26
>>>
>>> Abstract:
>>>       This document specifies properties and characteristics of a Non-
>>>       Queue-Building Per-Hop Behavior (NQB PHB).  The NQB PHB provides a
>>>       shallow-buffered, best-effort service as a complement to a Default
>>>       deep-buffered best-effort service for Internet services.  The purpose
>>>       of this NQB PHB is to provide a separate queue that enables smooth
>>>       (i.e. non-bursty), low-data-rate, application-limited traffic
>>>       microflows, which would ordinarily share a queue with bursty and
>>>       capacity-seeking traffic, to avoid the latency, latency variation and
>>>       loss caused by such traffic.  This PHB is implemented without
>>>       prioritization and can be implemented without rate policing, making
>>>       it suitable for environments where the use of these features is
>>>       restricted.  The NQB PHB has been developed primarily for use by
>>>       access network segments, where queuing delays and queuing loss caused
>>>       by Queue-Building protocols are manifested, but its use is not
>>>       limited to such segments.  In particular, applications to cable
>>>       broadband links, Wi-Fi links, and mobile network radio and core
>>>       segments are discussed.  This document recommends a specific
>>>       Differentiated Services Code Point (DSCP) to identify Non-Queue-
>>>       Building microflows.
>>>
>>>       [NOTE (to be removed by RFC-Editor): This document references an ISE
>>>       submission draft (I-D.briscoe-docsis-q-protection) that is approved
>>>       for publication as an RFC.  This draft should be held for publication
>>>       until the queue protection RFC can be referenced.]
>>>
>>> The IETF datatracker status page for this Internet-Draft is:
>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/
>>>
>>> There is also an HTML version available at:
>>> https://www.ietf.org/archive/id/draft-ietf-tsvwg-nqb-19.html
>>>
>>> A diff from the previous version is available at:
>>> https://author-tools.ietf.org/iddiff?url2=draft-ietf-tsvwg-nqb-19
>>>
>>> Internet-Drafts are also available by rsync at 
>>> rsync.ietf.org::internet-drafts
>>>
>>>
>>
>