Re: [EToSat] Download Regression Tests

[Christian Huitema <[e-mail]>]

I think the spirit is to use the loss scenario #2 in the "Wi-Fi +
satellite" simulations. That sounds like a good test case.

We can start with that, but I am still concerned about modelling
congestion losses as part of the loss pattern, especially Wi-Fi losses.
I can imagine situations in which the Wi-Fi link is the congestion
bottleneck, instead of the satellite link. In these situations, the end
to end goal would be to pick a transmission rate that does not saturate
the Wi-Fi link, and ensure a good flow of information end to end.

I am concerned that observations based on end-of-end transport flows mix
congestion losses due to sending faster than the Wi-Fi link can
transmit, and radio losses due to poor Wi-Fi transmission conditions. I
wonder whether we can do experiments that separate the two.

-- Christian Huitema

On 12/5/2019 5:39 PM, Kuhn Nicolas wrote:
> This surely helps ! 
> Thanks a lot. 
>
> I have updated the document with two possible loss patterns. 
> https://github.com/NicoKos/QUIC_HIGH_BDP/blob/master/ietf-document/draft-quic-4-sat/draft-kuhn-quic-4-sat-03.txt 
> If you have any views, please let us know. 
>
> I guess there are still some open questions :
> - should we keep all four scenarios ?
> - should we propose two loss models or only one ?
>
> Nico
>
>
> -----Message d'origine-----
> De : François Michel <[e-mail]> 
> Envoyé : mercredi 4 décembre 2019 15:13
> À : Border, John <[e-mail]>; Kuhn Nicolas <[e-mail]>
> Cc : [e-mail]; Emmanuel Lochin <[e-mail]>
> Objet : Re: [EToSat] Download Regression Tests
>
> Hello,
>
> We also performed some experiments with PQUIC/Picoquic over a satellite link with Nicolas and Emmanuel. We analyzed the loss patterns and deduced some two-state Gilbert parameters from network traces for two different scenarios. This model is less general than the Gilbert-Elliott model but is allows us to represent bursty loss patterns: in the GOOD state, all the packets are successfully received. In the bad state, the packets are simply dropped.
>
> Here are the two studied scenarios :
>
> 1) The client is connected to a wired network and its traffic towards the server is then routed through a satellite link.
> 2) The client is connected to a WiFi access point instead of a wired network. Its traffic towards the server is then still routed through a satellite link.
>
> What differs between these two scenarios is that in 2) we can also encounter losses due to the WiFi communication.
>
> We count the received packets at the receiver, knowing that the sender is configured to send packets with a contiguously increasing packet
> number: although it is feasible in QUIC, our Picoquic sender will never send packet n if it has not sent packet n-1. We provide the packet counts so that you can also do your own computations if needed.
>
> Here are the packets count and estimated probability of transitions of the Gilbert model we deduced from our experiments :
>
>
> For the scenario 1:
> -------------------
>
> from/to GOOD    BAD
> GOOD    80709   1370
> BAD     1370    86
>
> This means that 80709 packets have been received while the previous 
> packet was received (GOOD->GOOD), 86 packets have been lost while the 
> previous packet was also lost (BAD->BAD), 1370 packets have been 
> received while the previous packet was lost (BAD->GOOD) and 1370 packets 
> have been lost while the previous packet was received (GOOD->BAD).
>
>
> The average burst in the BAD state is quite small, but in our traces we 
> still found one loss burst of 15 packets.
>
> The estimated transitions probabilities of the Gilbert model:
> P(GOOD->GOOD) = 0.9833087635083273
> P(GOOD->BAD) = 0.016691236491672656
> P(BAD->GOOD) = 0.9409340659340659
> P(BAD->BAD) = 0.059065934065934064
>
>
> For the scenario 2:
> -------------------
>
> from/to GOOD    BAD
> GOOD    38700   700
> BAD     699     385
>
> P(GOOD->GOOD) = 0.9822335025380711
> P(GOOD->BAD) = 0.017766497461928935
> P(BAD->GOOD) = 0.6448339483394834
> P(BAD->BAD) = 0.3551660516605166
>
>
> Also quite isolated losses in general but we encountered more frequent 
> bursts, and we encountered larger loss bursts such as one burst of size 
> 33 and one burst of size 30.
> Keep in mind that these experiments are quite small and could be 
> influenced by the period of the day during which they have been performed.
>
>
> We hope this helps.
>
>
> François
>
>
>
> Le 3/12/19 à 19:24, Border, John a écrit :
>> The following are the variables we play with:
>>
>>   * Delay - *600 milliseconds*, 1000 milliseconds
>>   * File Size - *1 MB*, 10 MB, 100 MB, *1000 MB*
>>   * Packet Error Rate - *0.0%*, 0.1%, *1%*, 5%, 10%
>>   * Plan Rate:
>>       o Download (Internet to host)  - 10 Mbps, 25 Mbps, 100 Mbps, *250
>>         Mbps*
>>       o Upload (host to Internet) - *3 Mbps*, 10 Mbps, 25 Mbps
>>
>> The ones in bold are the ones we focus on since all of the above result 
>> in a lot of combinations.  If I had to pick just two, right now I would 
>> go with:
>>
>>   * 600 milliseconds, 1000 MB, 0.0%, 250 Mbps, 3 Mbps
>>   * 600 milliseconds, 1000 MB, 1.0%, 250 Mbps, 3 Mbps
>>
>> 600 milliseconds is the baseline delay.  Longer delays definitely happen 
>> during traffic peaks.  Longer than one second delays happen at traffic 
>> peak for traffic we classify as lower priority.
>>
>> 1 MB tests are web page object size tests.  1 GB tests are download 
>> tests.  (We would use 100 MB for upload tests but we have not really 
>> done any yet.)  We really have only been focusing large file sizes since 
>> we have been focusing on what we lose without TCP spoofing.
>>
>> For error rates, I included 5% an 10% because they represent WiFi 
>> extremes (and we did try them).  But, those values barely work at all. 
>>   We mainly focus on 1%.  There are actually two variants of error 
>> rate.  One is loss between the client and the satellite terminal.  (This 
>> is the loss TCP spoofing helps with.)  The other is loss between the 
>> satellite terminal and the satellite gateway which connects to the 
>> Internet.  We focus mostly on client side loss.
>>
>> We mostly have been testing with a 250 Mbps download plan rate so we can 
>> see how things work without being artificially limited..  We are easily 
>> able to fill this with TCP spoofing and ultimately we want QUIC to be on 
>> par with that.  (I think the best we have seen Google QUIC do in recent 
>> testing is ~30 Mbps but a few years ago it was only 3 Mbps so things are 
>> moving in a good direction.)   But, you cannot get plan rates like that 
>> outside of a lab right now.  In today's real world, we run with 25 Mbps 
>> download plans for customers.  Some providers use ~10 Mbps download plan 
>> rates.  We look at the higher rates anyway because we are trying to look 
>> ahead and satellite technology improves significantly over time.  We 
>> envision 100 Mbps download plans someday.
>>
>> We have not done much upload testing to date.  So, the upload plan rate 
>> has mattered mostly for purposes of how much ACK traffic it can carry.  
>> ACK reduction strategies are important for satellite but we have not 
>> done much with this yet other than to observe that 30 Mbps of Google 
>> QUIC download traffic fills the upload pipe.
>>
>> Not sure if this is what you were looking for but.
>>
>> John
>>
>>
>> _______________________________________________
>> EToSat mailing list
>> [e-mail]
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=02%7C01%7C%7Cc47c9ee8fadc4bc05bb908d7781e1768%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637109942949480777&amp;sdata=nbhVVrQcHaOLIUvQOuW5CA8UjXQCZXkz7SFyxoreGR0%3D&amp;reserved=0
>>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/etosat