Re: [EToSat] Interop runner with satellite links

[Emmanuel Lochin <[e-mail]>]

Hi all,

This is an interesting discussion. Nicolas and I get involved in a study 
on the performance of Picoquic/BBR and TCP/CUBIC within a Sliding Window 
FEC tunnel over SATCOM and clearly, BBR alone beats FEC coded CUBIC 
flows over SATCOM. We wrote a technical paper to present all our results 
here : https://hal.archives-ouvertes.fr/hal-03590651 and to confirm 
point 2) from Christian below and give some insights to previous CJ's email.

Emmanuel & Nicolas


Le 28/02/2022 à 20:54, Christian Huitema a écrit :
> Thanks for the kind words, CJ.
>
> A lot of the satellite innovation implemented in Picoquic results from 
> the efforts of Nicolas Kuhn, Gorry Fairhurst, John Border, Stephan 
> Emile and others in the ETOSAT community. In my mind, the most 
> interesting contribution of that community was to define series of 
> scenarios and expected results 
> (https://datatracker.ietf.org/doc/draft-kuhn-quic-4-sat/). These 
> scenarios act as a kind of "benchmark" for implementers. If 
> implementers care about the scenarios, they will program the 
> corresponding test cases in their test suites, and if needed ship the 
> corresponding fixes. In order to work in these scenarios, Picoquic did 
> the following:
>
> 1) Allow flow control windows to grow to values larger than the BDP. I 
> believe that many bad results seen in Sebastian's tests are due to 
> implementations capping the maximum flow control window to a low value.
>
> 2) Use Cubic or BBR rather than Reno. Both Cubic and BBR would give OK 
> results in the absence of losses. BBR works well by default. Cubic 
> works OK if using a filter to ignore isolated losses.
>
> 3) Perform ACK coalescing and avoid congesting a lower capacity return 
> link. Picoquic implements the ACK frequency draft 
> (https://datatracker.ietf.org/doc/draft-ietf-quic-ack-frequency/) and 
> tries to not send too many acks.
>
> 4) Remember the bandwidth and delay observed in previous connections. 
> Sebastian's benchmark did not test for that, but local tests show that 
> implementing the 0RTT BDP draft 
> (https://datatracker.ietf.org/doc/draft-kuhn-quic-0rtt-bdp/) results 
> in much better performance when resuming sessions over GEO satellite 
> links.
>
> These four points are all pretty much obvious, and I would expect to 
> see them over time in most implementations. The only controversial one 
> is the "flow control" issue. Some implementers may feel that opening 
> flow control too much carries risks of memory exhaustion. On the other 
> hand, some implementations are doing adaptive flow control already, 
> i.e., "opening flow control enough but not too much". So there is hope.
>
> On top of the four fairly standard points above, Picoquic implements 
> two simple features:
>
> 5) Fast bandwidth estimate during slow start, in order to accelerate 
> the slow start phase on high BDP paths 
> (https://huitema.wordpress.com/2020/04/21/faster-slow-start-for-satellite-links/). 
> Classic slow start needs log2(BDP/IW) RTTs. Fast estimate allows 
> initial CWIN estimation in 4 or 5 RTT, instead of 10 RTT for a 250 
> Mbps GEO path, or 8 for a 20 Mbps GEO path.
>
> 6) Preemptive repeat of unacknowledged data at the end of a file 
> transfer, in order to avoid waiting 1 or 2 extra RTT for complete 
> reception of a file in presence of packet losses.
>
> I don't know if there is any interest in standardizing these two.
>
> -- Christian Huitema
>
>
> On 2/28/2022 10:43 AM, Su, Chi-Jiun wrote:
>> Hi Christian,
>>
>> Thank you very much for coming up with various innovative approaches 
>> to improve QUIC over satellite links.
>> As the result shows, picoquic seems to perform very well over the GEO 
>> satellites.
>> We should continue this effort.
>>
>>    *   It is a great challenge to improve user's application 
>> performance due to lack of corporation between end points and network
>>       *   picoquic does preemptive retransmission to speed up the 
>> data transfer
>>       *   Link layer may employ ARQ or packet level FEC in local links
>>       *   Neither end points nor local link is aware of optimization 
>> done by the other party.
>>       *   As a result, the performance may end up worse than without 
>> the optimization.
>>       *   Especially for upload cases, where return link in 
>> wireless/sat is more limited and resource allocation is more involved.
>>    *   Repetition code is simplest error correcting code with least 
>> complexity.
>>       *   Other FEC will offer better bandwidth efficiency.
>>       *   Your idea of using some form of FEC will be useful.
>>
>> thanks.
>> cj
>>
>>    *
>>       *
>>
>> ________________________________
>> From: EToSat <[e-mail]> on behalf of Christian Huitema 
>> <[e-mail]>
>> Sent: Friday, February 25, 2022 9:39 PM
>> To: Su, Chi-Jiun <Chi-Jiun.Su=[e-mail]>; Sebastian 
>> Endres <[e-mail]>; [e-mail] <[e-mail]>; 
>> [e-mail] <[e-mail]>
>> Cc: [e-mail] <[e-mail]>
>> Subject: Re: [EToSat] Interop runner with satellite links
>>
>> **EXTERNAL EMAIL**
>>
>>
>> On 2/21/2022 10:16 AM, Su, Chi-Jiun wrote:
>>> Hi Sebastian,
>>>
>>> Thanks for the great work.
>>> Some comments/questions.
>>>
>>>     *   Sec. IV,D, 5: Interesting to know "picoquic" does 
>>> speculative retransmission. As you argue, this may not always help. 
>>> Did you confirm with the author?
>> This is indeed supported in picoquic. There are APIs that allows the
>> implementation to turn the feature on or off:
>> `picoquic_set_preemptive_repeat_policy(quic_context, do_repeat)` is used
>> to set the policy per Quic context, i.e., for all new connections;
>> `picoquic_set_preemptive_repeat_per_cnx(connection_context, do_repeat)`
>> is used to set the policy for a specific connection.
>>
>> The speculative retransmission happens at the lowest priority, i.e.,
>> only happens if there is nothing else to send. It is subject to
>> congestion control and rate limiting. The "nothing to send" rule means
>> that it will only kick after all data of a stream and the FIN mark have
>> been sent. The selection of data for speculative retransmission is based
>> on stream level acknowledgements. The code deduces the acknowledged
>> parts of a data stream from the packet acknowledgements. It will look at
>> data that has been sent, but not yet acknowledged.
>>
>> As you say, this does not always help. If the packet loss rate is low,
>> most of the preemptively repeated packets will be useless. On the other
>> hand, when sending a file over a lossy link, the application may wait a
>> long time for retransmission of packets lost in the last RTT. If loss
>> rate and data rates are high enough, some of these packets will have to
>> be repeated twice, or maybe three times. So we have a tradeoff: waste
>> bandwidth, or waste time. The API allows the application to consider
>> that tradeoff and decide whether it is useful or not.
>>
>> I would very much like to replace the current implementation of
>> preemptive repeat by some version of FEC. FEC in general is a poor fit
>> for transmission of big files, because it is always less bandwidth
>> efficient than just repeating the packets that are lost. But if the
>> application knows that the file transmission is almost complete, because
>> there is just one CWIN worth of data left, it could turn own FEC for the
>> the duration of the last RTT. It will "waste" a modest number of FEC
>> packets, while drastically reducing the impact of packet losses on the
>> duration of the file transfer. We could even imagine only sending the
>> FEC packets after the FIN mark has been sent, making sure that FEC does
>> not increase the duration of transfer if there were no errors.
>>
>> -- Christian Huitema
>>
>>
>>>     *    The results show loss-based CC does not perform well 
>>> compared to BBR
>>>     *   Production software performs well more like picoquic or not ?
>>>        *   how big difference is between production vs these 
>>> implementations in the test?
>>>     *   Any Time-Offset graphs for EUTELSAT case ?
>>>     *   Research overview page: additional column on emulated or 
>>> real Sat link will be helpful
>>>
>>> Good useful work.
>>> thanks.
>>> cj
>>> ________________________________
>>> From: QUIC <[e-mail]> on behalf of Sebastian Endres 
>>> <[e-mail]>
>>> Sent: Friday, February 18, 2022 7:02 AM
>>> To: [e-mail] <[e-mail]>; [e-mail] <[e-mail]>
>>> Cc: [e-mail] <[e-mail]>
>>> Subject: Re: [EToSat] Interop runner with satellite links
>>>
>>> **EXTERNAL EMAIL**
>>>
>>> Dear all,
>>>
>>> we've published a pre-print of our paper in which we present the 
>>> QUIC-Interop-runner extended to include satellite scenarios and our 
>>> measurement results using numerous publicly available QUIC 
>>> implementations:
>>>
>>> https://urldefense.com/v3/__https://arxiv.org/abs/2202.08228__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkW-JRj-Kg$ 
>>>
>>>
>>> Best regards,
>>>
>>> Sebastian
>>>
>>> On Mittwoch, 29. September 2021 21:38:05 CET Sebastian Endres wrote:
>>>> Dear all,
>>>>
>>>> for my master's thesis we ran measurements of all publicly 
>>>> available QUIC implementations over an emulated satellite link. The 
>>>> results are available online: 
>>>> https://urldefense.com/v3/__https://interop.sedrubal.de/__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ$
>>>>
>>>> A click on the results also shows time-offset plots, but are not 
>>>> available for every combination.
>>>>
>>>> In general, the performance of QUIC over high latency (e.g., 
>>>> geostationary satellites) is rather poor, especially if there is 
>>>> packet loss.
>>>>
>>>> Would it make sense to add such tests with challenging link 
>>>> characteristics to the official QUIC interop runner?
>>>>
>>>> Best regards,
>>>>
>>>> Sebastian
>>>>
>>>>
>>>> _______________________________________________
>>>> EToSat mailing list
>>>> [e-mail]
>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkXTZO8rhQ$ 
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> EToSat mailing list
>>> [e-mail]
>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$ 
>>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$ 
>>
>>
>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/etosat
>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/etosat

-- 
Emmanuel LOCHIN
ENAC - Ecole Nationale de l'Aviation Civile
7, avenue Edouard Belin CS 54005, 31055 Toulouse Cedex 4 France
http://www.enac.fr
https://elochin.github.io/