Re: [EToSat] Interop runner with satellite links

[François Michel <[e-mail]>]

Hello,

Thank you for the reference @Emmanuel, the study is interesting and we have similar results right here.
This illustrates quite well our point of view concerning the use of FEC within the transport
(in your article, you used it below the transport if I understand well): it seems better to use a
dedicated congestion control than trying to hide the losses with FEC
(it is never good to hide a signal IMO... :-) ). And your article demonstrates it well !

@everyone, and especially @Christian: I think sending FEC at the right moment, depending on the
use-case, can help a lot more than sending a fixed rate of redundancy, and what @Christian is doing
with the speculative retransmissions is IMO exactly sending redundancy at the right
moment for bulk transfer.

We worked quite a bit with FEC and QUIC during the
last months and years. We did it with Pluginized QUIC (PQUIC), but while PQUIC offers nice
features, the architecture of PQUIC made it hard to have the best results due to some bad
CPU with the PQUIC implem.
It think it becomes the moment where we start implementing FEC again in a clean QUIC implem,
and picoquic might be one of them. We did it several times now, so we are definitely willing to
look at it in the near future and it should not take much time. We now have cool use-cases and
diverse network accesses, including satellites, to experiment on. However, I think it will be
faster and cleaner if we look at it conjointly, so let me know if you're interested, we could
start a new FEC branch on picoquic and iterate on that.

It might also me a good moment to do a new pass on the design of quic-fec. After all this time using it,
I'm convinced we can improve it significantly. :-) So feel free to contact me for any of this.

Regards,

François


________________________________
From: EToSat <[e-mail]> on behalf of Emmanuel Lochin <[e-mail]>
Sent: 01 March 2022 14:58
To: [e-mail] <[e-mail]>
Cc: Nicolas Kuhn <[e-mail]>
Subject: Re: [EToSat] Interop runner with satellite links

Hi all,

This is an interesting discussion. Nicolas and I get involved in a study
on the performance of Picoquic/BBR and TCP/CUBIC within a Sliding Window
FEC tunnel over SATCOM and clearly, BBR alone beats FEC coded CUBIC
flows over SATCOM. We wrote a technical paper to present all our results
here : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhal.archives-ouvertes.fr%2Fhal-03590651&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=u7xI%2Bl4hQCxgMck6vyawzxbKnP30P4rRrLPeV0Nxf1M%3D&amp;reserved=0 and to confirm
point 2) from Christian below and give some insights to previous CJ's email.

Emmanuel & Nicolas


Le 28/02/2022 à 20:54, Christian Huitema a écrit :
> Thanks for the kind words, CJ.
>
> A lot of the satellite innovation implemented in Picoquic results from
> the efforts of Nicolas Kuhn, Gorry Fairhurst, John Border, Stephan
> Emile and others in the ETOSAT community. In my mind, the most
> interesting contribution of that community was to define series of
> scenarios and expected results
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-kuhn-quic-4-sat%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=JlS7QpDgEM1i2ao6viQ2vTzFVPwqx7RrABGT2nVBDsA%3D&amp;reserved=0). These
> scenarios act as a kind of "benchmark" for implementers. If
> implementers care about the scenarios, they will program the
> corresponding test cases in their test suites, and if needed ship the
> corresponding fixes. In order to work in these scenarios, Picoquic did
> the following:
>
> 1) Allow flow control windows to grow to values larger than the BDP. I
> believe that many bad results seen in Sebastian's tests are due to
> implementations capping the maximum flow control window to a low value.
>
> 2) Use Cubic or BBR rather than Reno. Both Cubic and BBR would give OK
> results in the absence of losses. BBR works well by default. Cubic
> works OK if using a filter to ignore isolated losses.
>
> 3) Perform ACK coalescing and avoid congesting a lower capacity return
> link. Picoquic implements the ACK frequency draft
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=DQ%2Bx4m4d8DYlPX8jkoqr%2B5KRaZvqS2h7mTekmg4gx4A%3D&amp;reserved=0) and
> tries to not send too many acks.
>
> 4) Remember the bandwidth and delay observed in previous connections.
> Sebastian's benchmark did not test for that, but local tests show that
> implementing the 0RTT BDP draft
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-kuhn-quic-0rtt-bdp%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=lSfamWAkxRlrHO7bWhpeG5jys%2BNNEOK1nhBQyCuLxg8%3D&amp;reserved=0) results
> in much better performance when resuming sessions over GEO satellite
> links.
>
> These four points are all pretty much obvious, and I would expect to
> see them over time in most implementations. The only controversial one
> is the "flow control" issue. Some implementers may feel that opening
> flow control too much carries risks of memory exhaustion. On the other
> hand, some implementations are doing adaptive flow control already,
> i.e., "opening flow control enough but not too much". So there is hope.
>
> On top of the four fairly standard points above, Picoquic implements
> two simple features:
>
> 5) Fast bandwidth estimate during slow start, in order to accelerate
> the slow start phase on high BDP paths
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuitema.wordpress.com%2F2020%2F04%2F21%2Ffaster-slow-start-for-satellite-links%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=WpM735c%2FAmhNzm7KEX84FUeI8zoz%2BEKruNsaT3Svy%2Fg%3D&amp;reserved=0).
> Classic slow start needs log2(BDP/IW) RTTs. Fast estimate allows
> initial CWIN estimation in 4 or 5 RTT, instead of 10 RTT for a 250
> Mbps GEO path, or 8 for a 20 Mbps GEO path.
>
> 6) Preemptive repeat of unacknowledged data at the end of a file
> transfer, in order to avoid waiting 1 or 2 extra RTT for complete
> reception of a file in presence of packet losses.
>
> I don't know if there is any interest in standardizing these two.
>
> -- Christian Huitema
>
>
> On 2/28/2022 10:43 AM, Su, Chi-Jiun wrote:
>> Hi Christian,
>>
>> Thank you very much for coming up with various innovative approaches
>> to improve QUIC over satellite links.
>> As the result shows, picoquic seems to perform very well over the GEO
>> satellites.
>> We should continue this effort.
>>
>>    *   It is a great challenge to improve user's application
>> performance due to lack of corporation between end points and network
>>       *   picoquic does preemptive retransmission to speed up the
>> data transfer
>>       *   Link layer may employ ARQ or packet level FEC in local links
>>       *   Neither end points nor local link is aware of optimization
>> done by the other party.
>>       *   As a result, the performance may end up worse than without
>> the optimization.
>>       *   Especially for upload cases, where return link in
>> wireless/sat is more limited and resource allocation is more involved.
>>    *   Repetition code is simplest error correcting code with least
>> complexity.
>>       *   Other FEC will offer better bandwidth efficiency.
>>       *   Your idea of using some form of FEC will be useful.
>>
>> thanks.
>> cj
>>
>>    *
>>       *
>>
>> ________________________________
>> From: EToSat <[e-mail]> on behalf of Christian Huitema
>> <[e-mail]>
>> Sent: Friday, February 25, 2022 9:39 PM
>> To: Su, Chi-Jiun <Chi-Jiun.Su=[e-mail]>; Sebastian
>> Endres <[e-mail]>; [e-mail] <[e-mail]>;
>> [e-mail] <[e-mail]>
>> Cc: [e-mail] <[e-mail]>
>> Subject: Re: [EToSat] Interop runner with satellite links
>>
>> **EXTERNAL EMAIL**
>>
>>
>> On 2/21/2022 10:16 AM, Su, Chi-Jiun wrote:
>>> Hi Sebastian,
>>>
>>> Thanks for the great work.
>>> Some comments/questions.
>>>
>>>     *   Sec. IV,D, 5: Interesting to know "picoquic" does
>>> speculative retransmission. As you argue, this may not always help.
>>> Did you confirm with the author?
>> This is indeed supported in picoquic. There are APIs that allows the
>> implementation to turn the feature on or off:
>> `picoquic_set_preemptive_repeat_policy(quic_context, do_repeat)` is used
>> to set the policy per Quic context, i.e., for all new connections;
>> `picoquic_set_preemptive_repeat_per_cnx(connection_context, do_repeat)`
>> is used to set the policy for a specific connection.
>>
>> The speculative retransmission happens at the lowest priority, i.e.,
>> only happens if there is nothing else to send. It is subject to
>> congestion control and rate limiting. The "nothing to send" rule means
>> that it will only kick after all data of a stream and the FIN mark have
>> been sent. The selection of data for speculative retransmission is based
>> on stream level acknowledgements. The code deduces the acknowledged
>> parts of a data stream from the packet acknowledgements. It will look at
>> data that has been sent, but not yet acknowledged.
>>
>> As you say, this does not always help. If the packet loss rate is low,
>> most of the preemptively repeated packets will be useless. On the other
>> hand, when sending a file over a lossy link, the application may wait a
>> long time for retransmission of packets lost in the last RTT. If loss
>> rate and data rates are high enough, some of these packets will have to
>> be repeated twice, or maybe three times. So we have a tradeoff: waste
>> bandwidth, or waste time. The API allows the application to consider
>> that tradeoff and decide whether it is useful or not.
>>
>> I would very much like to replace the current implementation of
>> preemptive repeat by some version of FEC. FEC in general is a poor fit
>> for transmission of big files, because it is always less bandwidth
>> efficient than just repeating the packets that are lost. But if the
>> application knows that the file transmission is almost complete, because
>> there is just one CWIN worth of data left, it could turn own FEC for the
>> the duration of the last RTT. It will "waste" a modest number of FEC
>> packets, while drastically reducing the impact of packet losses on the
>> duration of the file transfer. We could even imagine only sending the
>> FEC packets after the FIN mark has been sent, making sure that FEC does
>> not increase the duration of transfer if there were no errors.
>>
>> -- Christian Huitema
>>
>>
>>>     *    The results show loss-based CC does not perform well
>>> compared to BBR
>>>     *   Production software performs well more like picoquic or not ?
>>>        *   how big difference is between production vs these
>>> implementations in the test?
>>>     *   Any Time-Offset graphs for EUTELSAT case ?
>>>     *   Research overview page: additional column on emulated or
>>> real Sat link will be helpful
>>>
>>> Good useful work.
>>> thanks.
>>> cj
>>> ________________________________
>>> From: QUIC <[e-mail]> on behalf of Sebastian Endres
>>> <[e-mail]>
>>> Sent: Friday, February 18, 2022 7:02 AM
>>> To: [e-mail] <[e-mail]>; [e-mail] <[e-mail]>
>>> Cc: [e-mail] <[e-mail]>
>>> Subject: Re: [EToSat] Interop runner with satellite links
>>>
>>> **EXTERNAL EMAIL**
>>>
>>> Dear all,
>>>
>>> we've published a pre-print of our paper in which we present the
>>> QUIC-Interop-runner extended to include satellite scenarios and our
>>> measurement results using numerous publicly available QUIC
>>> implementations:
>>>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Farxiv.org%2Fabs%2F2202.08228__%3B!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkW-JRj-Kg%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=Rq4pbHLGeeXqDSf1q3mfKQ7CZP33YivZ6VeZRknI3B0%3D&amp;reserved=0
>>>
>>>
>>> Best regards,
>>>
>>> Sebastian
>>>
>>> On Mittwoch, 29. September 2021 21:38:05 CET Sebastian Endres wrote:
>>>> Dear all,
>>>>
>>>> for my master's thesis we ran measurements of all publicly
>>>> available QUIC implementations over an emulated satellite link. The
>>>> results are available online:
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Finterop.sedrubal.de%2F__%3B!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=qUxmio7Yi6NDQ8cW4qOmAn4hPZA7y45Vzb%2BI8x2e53E%3D&amp;reserved=0
>>>>
>>>> A click on the results also shows time-offset plots, but are not
>>>> available for every combination.
>>>>
>>>> In general, the performance of QUIC over high latency (e.g.,
>>>> geostationary satellites) is rather poor, especially if there is
>>>> packet loss.
>>>>
>>>> Would it make sense to add such tests with challenging link
>>>> characteristics to the official QUIC interop runner?
>>>>
>>>> Best regards,
>>>>
>>>> Sebastian
>>>>
>>>>
>>>> _______________________________________________
>>>> EToSat mailing list
>>>> [e-mail]
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat__%3B!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkXTZO8rhQ%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=z%2FL6ggN6uCRKUI6d0Na0n%2FAs4WtCQX1tKmxH%2F7gFLVc%3D&amp;reserved=0
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> EToSat mailing list
>>> [e-mail]
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat__%3B!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=v5TH%2BJGED7M9jWJYCFuLYXT0Jywo21%2BHxj4t5f0gZPg%3D&amp;reserved=0
>>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat__%3B!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=v5TH%2BJGED7M9jWJYCFuLYXT0Jywo21%2BHxj4t5f0gZPg%3D&amp;reserved=0
>>
>>
>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=%2FQAyUMsNxAM%2BnqS25RJL6vjBymeHQti0NaseC4LHm5c%3D&amp;reserved=0
>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=%2FQAyUMsNxAM%2BnqS25RJL6vjBymeHQti0NaseC4LHm5c%3D&amp;reserved=0

--
Emmanuel LOCHIN
ENAC - Ecole Nationale de l'Aviation Civile
7, avenue Edouard Belin CS 54005, 31055 Toulouse Cedex 4 France
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.enac.fr%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=bY7XjNinIEUbGNxKWmyTQpBKAyNocNsqkSawuMeWkTI%3D&amp;reserved=0
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Felochin.github.io%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=UITTEYc2bMivTD3C%2B%2F64Gi7H6SgGqIqI1VttceO6oBk%3D&amp;reserved=0

_______________________________________________
EToSat mailing list
[e-mail]
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=%2FQAyUMsNxAM%2BnqS25RJL6vjBymeHQti0NaseC4LHm5c%3D&amp;reserved=0