Re: [EToSat] Interop runner with satellite links

[Christian Huitema <[e-mail]>]

Thanks for the kind words, CJ.

A lot of the satellite innovation implemented in Picoquic results from 
the efforts of Nicolas Kuhn, Gorry Fairhurst, John Border, Stephan Emile 
and others in the ETOSAT community. In my mind, the most interesting 
contribution of that community was to define series of scenarios and 
expected results 
(https://datatracker.ietf.org/doc/draft-kuhn-quic-4-sat/). These 
scenarios act as a kind of "benchmark" for implementers. If implementers 
care about the scenarios, they will program the corresponding test cases 
in their test suites, and if needed ship the corresponding fixes. In 
order to work in these scenarios, Picoquic did the following:

1) Allow flow control windows to grow to values larger than the BDP. I 
believe that many bad results seen in Sebastian's tests are due to 
implementations capping the maximum flow control window to a low value.

2) Use Cubic or BBR rather than Reno. Both Cubic and BBR would give OK 
results in the absence of losses. BBR works well by default. Cubic works 
OK if using a filter to ignore isolated losses.

3) Perform ACK coalescing and avoid congesting a lower capacity return 
link. Picoquic implements the ACK frequency draft 
(https://datatracker.ietf.org/doc/draft-ietf-quic-ack-frequency/) and 
tries to not send too many acks.

4) Remember the bandwidth and delay observed in previous connections. 
Sebastian's benchmark did not test for that, but local tests show that 
implementing the 0RTT BDP draft 
(https://datatracker.ietf.org/doc/draft-kuhn-quic-0rtt-bdp/) results in 
much better performance when resuming sessions over GEO satellite links.

These four points are all pretty much obvious, and I would expect to see 
them over time in most implementations. The only controversial one is 
the "flow control" issue. Some implementers may feel that opening flow 
control too much carries risks of memory exhaustion. On the other hand, 
some implementations are doing adaptive flow control already, i.e., 
"opening flow control enough but not too much". So there is hope.

On top of the four fairly standard points above, Picoquic implements two 
simple features:

5) Fast bandwidth estimate during slow start, in order to accelerate the 
slow start phase on high BDP paths 
(https://huitema.wordpress.com/2020/04/21/faster-slow-start-for-satellite-links/). 
Classic slow start needs log2(BDP/IW) RTTs. Fast estimate allows initial 
CWIN estimation in 4 or 5 RTT, instead of 10 RTT for a 250 Mbps GEO 
path, or 8 for a 20 Mbps GEO path.

6) Preemptive repeat of unacknowledged data at the end of a file 
transfer, in order to avoid waiting 1 or 2 extra RTT for complete 
reception of a file in presence of packet losses.

I don't know if there is any interest in standardizing these two.

-- Christian Huitema


On 2/28/2022 10:43 AM, Su, Chi-Jiun wrote:
> Hi Christian,
>
> Thank you very much for coming up with various innovative approaches to improve QUIC over satellite links.
> As the result shows, picoquic seems to perform very well over the GEO satellites.
> We should continue this effort.
>
>    *   It is a great challenge to improve user's application performance due to lack of corporation between end points and network
>       *   picoquic does preemptive retransmission to speed up the data transfer
>       *   Link layer may employ ARQ or packet level FEC in local links
>       *   Neither end points nor local link is aware of optimization done by the other party.
>       *   As a result, the performance may end up worse than without the optimization.
>       *   Especially for upload cases, where return link in wireless/sat is more limited and resource allocation is more involved.
>    *   Repetition code is simplest error correcting code with least complexity.
>       *   Other FEC will offer better bandwidth efficiency.
>       *   Your idea of using some form of FEC will be useful.
>
> thanks.
> cj
>
>    *
>       *
>
> ________________________________
> From: EToSat <[e-mail]> on behalf of Christian Huitema <[e-mail]>
> Sent: Friday, February 25, 2022 9:39 PM
> To: Su, Chi-Jiun <Chi-Jiun.Su=[e-mail]>; Sebastian Endres <[e-mail]>; [e-mail] <[e-mail]>; [e-mail] <[e-mail]>
> Cc: [e-mail] <[e-mail]>
> Subject: Re: [EToSat] Interop runner with satellite links
>
> **EXTERNAL EMAIL**
>
>
> On 2/21/2022 10:16 AM, Su, Chi-Jiun wrote:
>> Hi Sebastian,
>>
>> Thanks for the great work.
>> Some comments/questions.
>>
>>     *   Sec. IV,D, 5: Interesting to know "picoquic" does speculative retransmission. As you argue, this may not always help. Did you confirm with the author?
> This is indeed supported in picoquic. There are APIs that allows the
> implementation to turn the feature on or off:
> `picoquic_set_preemptive_repeat_policy(quic_context, do_repeat)` is used
> to set the policy per Quic context, i.e., for all new connections;
> `picoquic_set_preemptive_repeat_per_cnx(connection_context, do_repeat)`
> is used to set the policy for a specific connection.
>
> The speculative retransmission happens at the lowest priority, i.e.,
> only happens if there is nothing else to send. It is subject to
> congestion control and rate limiting. The "nothing to send" rule means
> that it will only kick after all data of a stream and the FIN mark have
> been sent. The selection of data for speculative retransmission is based
> on stream level acknowledgements. The code deduces the acknowledged
> parts of a data stream from the packet acknowledgements. It will look at
> data that has been sent, but not yet acknowledged.
>
> As you say, this does not always help. If the packet loss rate is low,
> most of the preemptively repeated packets will be useless. On the other
> hand, when sending a file over a lossy link, the application may wait a
> long time for retransmission of packets lost in the last RTT. If loss
> rate and data rates are high enough, some of these packets will have to
> be repeated twice, or maybe three times. So we have a tradeoff: waste
> bandwidth, or waste time. The API allows the application to consider
> that tradeoff and decide whether it is useful or not.
>
> I would very much like to replace the current implementation of
> preemptive repeat by some version of FEC. FEC in general is a poor fit
> for transmission of big files, because it is always less bandwidth
> efficient than just repeating the packets that are lost. But if the
> application knows that the file transmission is almost complete, because
> there is just one CWIN worth of data left, it could turn own FEC for the
> the duration of the last RTT. It will "waste" a modest number of FEC
> packets, while drastically reducing the impact of packet losses on the
> duration of the file transfer. We could even imagine only sending the
> FEC packets after the FIN mark has been sent, making sure that FEC does
> not increase the duration of transfer if there were no errors.
>
> -- Christian Huitema
>
>
>>     *    The results show loss-based CC does not perform well compared to BBR
>>     *   Production software performs well more like picoquic or not ?
>>        *   how big difference is between production vs these implementations in the test?
>>     *   Any Time-Offset graphs for EUTELSAT case ?
>>     *   Research overview page: additional column on emulated or real Sat link will be helpful
>>
>> Good useful work.
>> thanks.
>> cj
>> ________________________________
>> From: QUIC <[e-mail]> on behalf of Sebastian Endres <[e-mail]>
>> Sent: Friday, February 18, 2022 7:02 AM
>> To: [e-mail] <[e-mail]>; [e-mail] <[e-mail]>
>> Cc: [e-mail] <[e-mail]>
>> Subject: Re: [EToSat] Interop runner with satellite links
>>
>> **EXTERNAL EMAIL**
>>
>> Dear all,
>>
>> we've published a pre-print of our paper in which we present the QUIC-Interop-runner extended to include satellite scenarios and our measurement results using numerous publicly available QUIC implementations:
>>
>> https://urldefense.com/v3/__https://arxiv.org/abs/2202.08228__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkW-JRj-Kg$
>>
>> Best regards,
>>
>> Sebastian
>>
>> On Mittwoch, 29. September 2021 21:38:05 CET Sebastian Endres wrote:
>>> Dear all,
>>>
>>> for my master's thesis we ran measurements of all publicly available QUIC implementations over an emulated satellite link. The results are available online: https://urldefense.com/v3/__https://interop.sedrubal.de/__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ$
>>>
>>> A click on the results also shows time-offset plots, but are not available for every combination.
>>>
>>> In general, the performance of QUIC over high latency (e.g., geostationary satellites) is rather poor, especially if there is packet loss.
>>>
>>> Would it make sense to add such tests with challenging link characteristics to the official QUIC interop runner?
>>>
>>> Best regards,
>>>
>>> Sebastian
>>>
>>>
>>> _______________________________________________
>>> EToSat mailing list
>>> [e-mail]
>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkXTZO8rhQ$
>>>
>>
>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$
>
>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/etosat