[EToSat] Congestion control beyond initial BDP

[Christian Huitema <[e-mail]>]

I have been reviewing and tuning the implementation of the QUIC/BDP 
draft in picoquic, adding a bunch of tests based on simulations. The 
tests are finding some interesting issues. My first fixes were to look 
at teh way the BDP is "plumbed" at the beginning of the connection. I 
ended up doing the following:

1) For new Reno and Cubic, just setting the CWIN was not sufficient. If 
the CWIN is set to the BDP and slow start or high start continue, the 
window will wildly overshoot after 1 RTT, causing lots of losses and a 
degradation in performance. To fix that, I had to set both CWIN and 
SSTHRESH, and exit slow-start immediately.

2) For BBR, the same overshoot issue happens, and is solved by setting 
the bottlenect bandwidth from BDP and then moving to the "probe 
bandwidth" state.

3) In all cases, we have to be careful to not just copy the value of the 
CWIN of the previous connection in the BDP, because CWIN is sometimes 
way too big, for example at the end of slow-start. Instead, I checked 
the estimated bandwidth, obtained from the arrival of ACKs, and set the 
BDP to estimated bandwidth times RTT.

With that, yes, remembering BDP from previous connections works as 
expected in the initial phase of the connection, but no, this is not 
sufficient to properly handle high BDP links. Consider the following:

1) When using New Reno, the window is set at the BDP value, and then 
appears to never change. It does in fact change, but very slowly: 
increase by one packet every RTT. The tests last just a few seconds, so 
this means increase by just a few packets, which is hardly noticeable. 
This also means that if the initial BDP is too small, it is going to 
remain small for the duration of the session.

2) When using BBR, the bottleneck bandwidth is set from the BDP value. 
There were a few tests in which that BDP value was too high. It took me 
some time to understand why it remained too high for a long time, but 
that can be traced to the "10 RTT filter" implemented in BBR: the pacing 
rate is set to the highest bandwidth observed in the last 10 RTT. So, it 
takes at least 6 seconds for BBR to adjust from an overoptimistic 
initial setting. It will also take a long time to adjust from a low 
initial setting, because BBR uses RTT-length epochs, and only tests for 
additional 25% bandwidth in some of these epochs. Still, that would be 
better than New Reno.

3) When using Cubic, the initial ssthresh is set from the BDP, and then 
Cubic moves to the congestion avoidance mode. I observe a series of 
epoch each ending with high number of losses. That may be due to a bug 
in my implementation, but there is also the epoch effect: Cubic epochs 
are set as a function of the RTT, and thus Cubic reacts less well when 
the RTT is large.

Bottom line, fixing the BDP helps but is not sufficient. New Reno is 
probably hopeless -- Sally Floyd's work of years ago still applies. 
Cubic currently does not work well, which means we need specific 
investigations, and then specific recommendations. BBR sort of works, 
but some additional tuning would help. It would probably be a good idea 
to provide specific "implementation recommendations" for the popular 
congestion control algorithms.

-- Christian Huitema