[EToSat] Picoquic, satellites and BBR

[Christian Huitema <[e-mail]>]

I spent some time updating the Picoquic code so it performs decently on
satellite links. I added a set of tests mimicking the 250/3 "clean" and
250/3 lossy scenarios. I did not implement the proposed extension yet,
but here is are the first results, running on a simulator:

1) Upload 100MB over 250 Mbps link with 3 Mbps return link, no
transmission errors: completes in about 6.4 seconds.

2) Upload 100MB over 250 Mbps link with 3 Mbps return list, 1.6%
transmission errors: completes in about 9.8 seconds.

The transmission buffers were set to 1 BDP on each side.

The tests use upload instead of download because of some limitations of
my setup: it was easier for me to instrument the client rather than the
server. Download tests would last the same time.

In both cases, the congestion control algorithm is set to BBR, with a
couple of small modifications: time stamps in ACKs, and Hystart style
replacement of start-up. BBR does not reduce data rate on transient
losses like those found in the lossy scenario. Picoquic implements ACK
coalescing, with 10 packets per ACK when transmitting at high rate.

For the "clean" test, the 6.4 seconds include 0.6 second for setting the
connection, 0.6 for requesting/confirming the file, and 0.6 for closing
the connection. The actual transmission lasts 5.2 seconds, running at
153 Mbps on average, 61% efficiency. The main overhead is the ramp-up
time, from the initial window to the nominal data rate. In the tests
that lasts about 3.6 seconds. After that, the transmission rate
stabilizes close to full utilization. I did a couple of tests
downloading 1GB instead of 100MB, and the last 900MB are transmitted at
or close to line rate.

For the "lossy" test, there is significant additional delay at the end
of the transmission. At full speed, the BDP is 18.75 MB, i.e. close to
13000 packets. Out of these, about 200 will need to be resent once, and
3 of those will have to be resent twice. That means adding a couple of
RTOs at the end of the transmission. That explains about 90% of the
difference with the "clean" test. I suppose that I could work on the
retransmission code and reduce these delays a bit. Most of the work I
did so far dealt with performance issues caused by long ACK lists and
frequent "holes", not on actual efficiency of error correction. But even
with frequent losses, the transmission rate stabilizes at or close to
line rate.

I added time stamps to the ACK frames when testing the asymmetry. I
wanted to measure how much of the delays were due to transmit side
queues, and how much to ACK queues. Asymmetry actually ceased to be a
problem after ACK coalescing was properly debugged. In the latest tests,
all the delay variations are due to the transmit side queues, and I
would get the same results without the timestamps. Still, time stamps in
ACKs are a good idea in general. It is an insurance against ACK
coalescing in ACK queueing in the network, enabling proper bandwidth
measurements even when the ACK link is congested. It seems very valuable
when using BBR or delay-based congestion control like FAST or LEDBAT.

I had an issue with the "startup" algorithm of BBR. Early testing showed
that BBR startup phase requires several more RTT than the Hystart
process used in modern versions of Reno or Cubic. BBR only ramps up the
data rate after the first bandwidth measurement is available, 2*RTT
after start, while Reno or Cubic start ramping up after just 1 RTT. BBR
only exits startup if three consecutive RTT pass without significant BW
measurement increase, which not only adds delay but also creates big
queues as data is sent at 2.89 times the bottleneck rate for an addition
2 RTT. This is a tradeoff: longer search for bandwidth in slow start is
less likely to stop too early because of transient issues, but on high
bandwidth and long delay links this translates to long delays and a big
batch of packet losses. The BBR implementation in Picoquic addresses
these issues by switching to Hystart instead of BBR-startup if the RTT
is above 100 ms.

This batch of "satellite" fixes is implemented in the latest version of
Picoquic.

Of course, I am still waiting for the BDP option, which could solve the
"ramp up" issue. I just want to make sure that we are not opening a big
DOS mechanism.

-- Christian Huitema