Re: [EToSat] Download Regression Tests

[Kuhn Nicolas <[e-mail]>]

I have updated the document with a proposal  
- the burst model includes congestion and propagation losses
- the uniform random loss includes only wi-fi losses 
That may be a starting point for discussion. 

If we want to include the congestion at the wifi router (without AQM?), which is an interesting use-case, I guess we have to go more into the details of the architecture. 
A solution would be to include multiple machines and emulated SATCOM system with satellite terminal. 
We can surely add this kind of regression test, but I am afraid that it may be too complexed as a first step. 
Anyway, we can propose an optional more complexed scenario if you think his is relevant. 

Cheers, 

Nico

-----Message d'origine-----
De : Christian Huitema <[e-mail]> 
Envoyé : jeudi 5 décembre 2019 21:27
À : Kuhn Nicolas <[e-mail]>; 'François Michel' <[e-mail]>; Border, John <[e-mail]>
Cc : [e-mail]; Emmanuel Lochin <[e-mail]>
Objet : Re: [EToSat] Download Regression Tests

I think the spirit is to use the loss scenario #2 in the "Wi-Fi + satellite" simulations. That sounds like a good test case.

We can start with that, but I am still concerned about modelling congestion losses as part of the loss pattern, especially Wi-Fi losses.
I can imagine situations in which the Wi-Fi link is the congestion bottleneck, instead of the satellite link. In these situations, the end to end goal would be to pick a transmission rate that does not saturate the Wi-Fi link, and ensure a good flow of information end to end.

I am concerned that observations based on end-of-end transport flows mix congestion losses due to sending faster than the Wi-Fi link can transmit, and radio losses due to poor Wi-Fi transmission conditions. I wonder whether we can do experiments that separate the two.

-- Christian Huitema

On 12/5/2019 5:39 PM, Kuhn Nicolas wrote:
> This surely helps ! 
> Thanks a lot. 
>
> I have updated the document with two possible loss patterns. 
> https://github.com/NicoKos/QUIC_HIGH_BDP/blob/master/ietf-document/dra
> ft-quic-4-sat/draft-kuhn-quic-4-sat-03.txt
> If you have any views, please let us know. 
>
> I guess there are still some open questions :
> - should we keep all four scenarios ?
> - should we propose two loss models or only one ?
>
> Nico
>
>
> -----Message d'origine-----
> De : François Michel <[e-mail]> Envoyé : mercredi 
> 4 décembre 2019 15:13 À : Border, John <[e-mail]>; Kuhn 
> Nicolas <[e-mail]> Cc : [e-mail]; Emmanuel Lochin 
> <[e-mail]> Objet : Re: [EToSat] Download 
> Regression Tests
>
> Hello,
>
> We also performed some experiments with PQUIC/Picoquic over a satellite link with Nicolas and Emmanuel. We analyzed the loss patterns and deduced some two-state Gilbert parameters from network traces for two different scenarios. This model is less general than the Gilbert-Elliott model but is allows us to represent bursty loss patterns: in the GOOD state, all the packets are successfully received. In the bad state, the packets are simply dropped.
>
> Here are the two studied scenarios :
>
> 1) The client is connected to a wired network and its traffic towards the server is then routed through a satellite link.
> 2) The client is connected to a WiFi access point instead of a wired network. Its traffic towards the server is then still routed through a satellite link.
>
> What differs between these two scenarios is that in 2) we can also encounter losses due to the WiFi communication.
>
> We count the received packets at the receiver, knowing that the sender 
> is configured to send packets with a contiguously increasing packet
> number: although it is feasible in QUIC, our Picoquic sender will never send packet n if it has not sent packet n-1. We provide the packet counts so that you can also do your own computations if needed.
>
> Here are the packets count and estimated probability of transitions of the Gilbert model we deduced from our experiments :
>
>
> For the scenario 1:
> -------------------
>
> from/to GOOD    BAD
> GOOD    80709   1370
> BAD     1370    86
>
> This means that 80709 packets have been received while the previous 
> packet was received (GOOD->GOOD), 86 packets have been lost while the 
> previous packet was also lost (BAD->BAD), 1370 packets have been 
> received while the previous packet was lost (BAD->GOOD) and 1370 
> packets have been lost while the previous packet was received (GOOD->BAD).
>
>
> The average burst in the BAD state is quite small, but in our traces 
> we still found one loss burst of 15 packets.
>
> The estimated transitions probabilities of the Gilbert model:
> P(GOOD->GOOD) = 0.9833087635083273
> P(GOOD->BAD) = 0.016691236491672656
> P(BAD->GOOD) = 0.9409340659340659
> P(BAD->BAD) = 0.059065934065934064
>
>
> For the scenario 2:
> -------------------
>
> from/to GOOD    BAD
> GOOD    38700   700
> BAD     699     385
>
> P(GOOD->GOOD) = 0.9822335025380711
> P(GOOD->BAD) = 0.017766497461928935
> P(BAD->GOOD) = 0.6448339483394834
> P(BAD->BAD) = 0.3551660516605166
>
>
> Also quite isolated losses in general but we encountered more frequent 
> bursts, and we encountered larger loss bursts such as one burst of 
> size
> 33 and one burst of size 30.
> Keep in mind that these experiments are quite small and could be 
> influenced by the period of the day during which they have been performed.
>
>
> We hope this helps.
>
>
> François
>
>
>
> Le 3/12/19 à 19:24, Border, John a écrit :
>> The following are the variables we play with:
>>
>>   * Delay - *600 milliseconds*, 1000 milliseconds
>>   * File Size - *1 MB*, 10 MB, 100 MB, *1000 MB*
>>   * Packet Error Rate - *0.0%*, 0.1%, *1%*, 5%, 10%
>>   * Plan Rate:
>>       o Download (Internet to host)  - 10 Mbps, 25 Mbps, 100 Mbps, *250
>>         Mbps*
>>       o Upload (host to Internet) - *3 Mbps*, 10 Mbps, 25 Mbps
>>
>> The ones in bold are the ones we focus on since all of the above 
>> result in a lot of combinations.  If I had to pick just two, right 
>> now I would go with:
>>
>>   * 600 milliseconds, 1000 MB, 0.0%, 250 Mbps, 3 Mbps
>>   * 600 milliseconds, 1000 MB, 1.0%, 250 Mbps, 3 Mbps
>>
>> 600 milliseconds is the baseline delay.  Longer delays definitely 
>> happen during traffic peaks.  Longer than one second delays happen at 
>> traffic peak for traffic we classify as lower priority.
>>
>> 1 MB tests are web page object size tests.  1 GB tests are download 
>> tests.  (We would use 100 MB for upload tests but we have not really 
>> done any yet.)  We really have only been focusing large file sizes 
>> since we have been focusing on what we lose without TCP spoofing.
>>
>> For error rates, I included 5% an 10% because they represent WiFi 
>> extremes (and we did try them).  But, those values barely work at all.
>>   We mainly focus on 1%.  There are actually two variants of error 
>> rate.  One is loss between the client and the satellite terminal.  
>> (This is the loss TCP spoofing helps with.)  The other is loss 
>> between the satellite terminal and the satellite gateway which 
>> connects to the Internet.  We focus mostly on client side loss.
>>
>> We mostly have been testing with a 250 Mbps download plan rate so we 
>> can see how things work without being artificially limited..  We are 
>> easily able to fill this with TCP spoofing and ultimately we want 
>> QUIC to be on par with that.  (I think the best we have seen Google 
>> QUIC do in recent testing is ~30 Mbps but a few years ago it was only 
>> 3 Mbps so things are moving in a good direction.)   But, you cannot 
>> get plan rates like that outside of a lab right now.  In today's real 
>> world, we run with 25 Mbps download plans for customers.  Some 
>> providers use ~10 Mbps download plan rates.  We look at the higher 
>> rates anyway because we are trying to look ahead and satellite 
>> technology improves significantly over time.  We envision 100 Mbps download plans someday.
>>
>> We have not done much upload testing to date.  So, the upload plan 
>> rate has mattered mostly for purposes of how much ACK traffic it can carry.
>> ACK reduction strategies are important for satellite but we have not 
>> done much with this yet other than to observe that 30 Mbps of Google 
>> QUIC download traffic fills the upload pipe.
>>
>> Not sure if this is what you were looking for but.
>>
>> John
>>
>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>> .ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=02%7C01%7C%7Cc47c9ee
>> 8fadc4bc05bb908d7781e1768%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%
>> 7C637109942949480777&amp;sdata=nbhVVrQcHaOLIUvQOuW5CA8UjXQCZXkz7SFyxo
>> reGR0%3D&amp;reserved=0
>>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/etosat