Re: [EToSat] Picoquic, satellites and BBR

["Su, Chi-Jiun" <[e-mail]>]

Hi Christian, 

Thank you for doing a detailed analysis of quic performance over satellite:
- Startup performance
- Impact of Ack and fixes
- performance under packet loss.

Also thanks for implementing satellite fixes.
Picoquic can be used to compare the performance with other enhancement proposals such as BDP options.

Currently, operational configuration of production level software seems to limit the steady state throughput of QUIC over a high BDP link.

The following are the parameters configured in gQUIC (chromium), IIRC, 

- maximum congestion window is 2000 packets (~2.6MByte, 35Mbps for 600msec RTT)

Flow control limit
- maximum receive window sizes for QUIC streams is 6MB
- maximum receive window sizes for QUIC sessions is 15 MB

High BDP link requires bigger values than those.

thanks.
cj

-----Original Message-----
From: EToSat <[e-mail]> On Behalf Of Christian Huitema
Sent: Thursday, January 2, 2020 10:40 PM
To: [e-mail]
Subject: [EToSat] Picoquic, satellites and BBR

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I spent some time updating the Picoquic code so it performs decently on

satellite links. I added a set of tests mimicking the 250/3 "clean" and

250/3 lossy scenarios. I did not implement the proposed extension yet,

but here is are the first results, running on a simulator:



1) Upload 100MB over 250 Mbps link with 3 Mbps return link, no

transmission errors: completes in about 6.4 seconds.



2) Upload 100MB over 250 Mbps link with 3 Mbps return list, 1.6%

transmission errors: completes in about 9.8 seconds.



The transmission buffers were set to 1 BDP on each side.



The tests use upload instead of download because of some limitations of

my setup: it was easier for me to instrument the client rather than the

server. Download tests would last the same time.



In both cases, the congestion control algorithm is set to BBR, with a

couple of small modifications: time stamps in ACKs, and Hystart style

replacement of start-up. BBR does not reduce data rate on transient

losses like those found in the lossy scenario. Picoquic implements ACK

coalescing, with 10 packets per ACK when transmitting at high rate.



For the "clean" test, the 6.4 seconds include 0.6 second for setting the

connection, 0.6 for requesting/confirming the file, and 0.6 for closing

the connection. The actual transmission lasts 5.2 seconds, running at

153 Mbps on average, 61% efficiency. The main overhead is the ramp-up

time, from the initial window to the nominal data rate. In the tests

that lasts about 3.6 seconds. After that, the transmission rate

stabilizes close to full utilization. I did a couple of tests

downloading 1GB instead of 100MB, and the last 900MB are transmitted at

or close to line rate.



For the "lossy" test, there is significant additional delay at the end

of the transmission. At full speed, the BDP is 18.75 MB, i.e. close to

13000 packets. Out of these, about 200 will need to be resent once, and

3 of those will have to be resent twice. That means adding a couple of

RTOs at the end of the transmission. That explains about 90% of the

difference with the "clean" test. I suppose that I could work on the

retransmission code and reduce these delays a bit. Most of the work I

did so far dealt with performance issues caused by long ACK lists and

frequent "holes", not on actual efficiency of error correction. But even

with frequent losses, the transmission rate stabilizes at or close to

line rate.



I added time stamps to the ACK frames when testing the asymmetry. I

wanted to measure how much of the delays were due to transmit side

queues, and how much to ACK queues. Asymmetry actually ceased to be a

problem after ACK coalescing was properly debugged. In the latest tests,

all the delay variations are due to the transmit side queues, and I

would get the same results without the timestamps. Still, time stamps in

ACKs are a good idea in general. It is an insurance against ACK

coalescing in ACK queueing in the network, enabling proper bandwidth

measurements even when the ACK link is congested. It seems very valuable

when using BBR or delay-based congestion control like FAST or LEDBAT.



I had an issue with the "startup" algorithm of BBR. Early testing showed

that BBR startup phase requires several more RTT than the Hystart

process used in modern versions of Reno or Cubic. BBR only ramps up the

data rate after the first bandwidth measurement is available, 2*RTT

after start, while Reno or Cubic start ramping up after just 1 RTT. BBR

only exits startup if three consecutive RTT pass without significant BW

measurement increase, which not only adds delay but also creates big

queues as data is sent at 2.89 times the bottleneck rate for an addition

2 RTT. This is a tradeoff: longer search for bandwidth in slow start is

less likely to stop too early because of transient issues, but on high

bandwidth and long delay links this translates to long delays and a big

batch of packet losses. The BBR implementation in Picoquic addresses

these issues by switching to Hystart instead of BBR-startup if the RTT

is above 100 ms.



This batch of "satellite" fixes is implemented in the latest version of

Picoquic.



Of course, I am still waiting for the BDP option, which could solve the

"ramp up" issue. I just want to make sure that we are not opening a big

DOS mechanism.



-- Christian Huitema









_______________________________________________

EToSat mailing list

[e-mail]

https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!los_YyTDsxcqaA5g26kUP5CfnLjARb897F1irGgXDEnAwFRyLzV57jzkcNe-ePr7Rw$