Re: [EToSat] Download Regression Tests

[Gorry Fairhurst <[e-mail]>]

On 04/12/2019, 08:14, Kuhn Nicolas wrote:
>
> Hi,
>
> We had initiated a regression test description in the QUIC 4 SATCOM 
> document.
>
> https://github.com/NicoKos/QUIC_HIGH_BDP/tree/master/ietf-document/draft-quic-4-sat 
>
>
> An updated version in a more readable format than XML is attached and 
> in the GITHUB repo.
>
> The scenario that you propose in more challenging than the ones we had 
> discussed with Gorry at Singapore.
>
> I have added your scenario in the document – but you may not want to 
> keep them all.
>
> What do you think ?
>
> I guess we can at least remove the 10 Mbps one.
>
[GF] I'd keep this for the moment. It's a simple enough test, and if you 
can fill 10 Mbps with a single stream, then that's pretty good.
>
> However, keeping the 50 Mbps / 10 Mbps shows a different asymmetry and 
> it may be interesting to keep it.
>
> More comments inline.
>
> Cheers,
>
> Nico
>
Gorry
>
> *De :* Christian Huitema <[e-mail]>
> *Envoyé :* mardi 3 décembre 2019 23:28
> *À :* Border, John <[e-mail]>; Kuhn Nicolas 
> <[e-mail]>
> *Cc :* [e-mail]
> *Objet :* Re: [EToSat] Download Regression Tests
>
> Thanks, that's very helpful.
>
> I was doing test so far by simulating 600 ms delay, 100 Mbps download 
> and 100 Mbps upload (symmetric). I will switch the simulated 
> configuration to 250 Mbps upload and 3 Mbps download.
>
> I am still working on the asymmetric link part, but I just did a test 
> with 1GB with 250 Mbps symmetric link. With 0 error, it completes in a 
> bit less than 36 sec on the Picoquic simulator -- effective data rate 
> would be 222 Mbps. Not a problem for a one-off test, but that's a bit 
> onerous for a non-regression test that shall run on every check-in.
>
> I also tested 100 MB and 10 MB. The 100 MB test completes on the 
> Picoquic simulator in 8.24 sec, average rate of 97 Mbps, and the 10MB 
> test completes in 5.22 sec, average of 15 Mbps. The 10MB test is 
> dominated by connection establishment and rate ramping issues (slow 
> start, flow control, etc.) The 100 MB test has that, plus about 3 
> seconds of full speed transmission. I think 100MB is enough for the 
> "non regression" function. It provides a good mix of establishment and 
> data rate issues.
>
> */[NK] That size seems a good trade-off to me/*
>
> I guess the file size is a simple parameter. I have two additional 
> questions for setting up the simulations:
>
> 1) What is the maximum size of the buffer on the download path?
>
> [NK] We propose to set the buffersize to the BDP – to begin with
>
> 2) When simulating the 1% loss rate, is there a pattern to the losses, 
> or are they just randomly distributed with independent probability for 
> each packet?
>
> */[NK] The 1% loss rate is not really random since both congestion and 
> random losses occur in a bursty way. We will try to share more details 
> very soon/*
>
> -- Christian Huitema
>
> On 12/4/2019 2:24 AM, Border, John wrote:
>
>     The following are the variables we play with:
>
>       * Delay – *600 milliseconds*, 1000 milliseconds
>       * File Size – *1 MB*, 10 MB, 100 MB, *1000 MB*
>       * Packet Error Rate – *0.0%*, 0.1%, *1%*, 5%, 10%
>       * Plan Rate:
>
>           o Download (Internet to host) - 10 Mbps, 25 Mbps, 100 Mbps,
>             *250 Mbps*
>           o Upload (host to Internet) – *3 Mbps*, 10 Mbps, 25 Mbps
>
>     The ones in bold are the ones we focus on since all of the above
>     result in a lot of combinations. If I had to pick just two, right
>     now I would go with:
>
>       * 600 milliseconds, 1000 MB, 0.0%, 250 Mbps, 3 Mbps
>       * 600 milliseconds, 1000 MB, 1.0%, 250 Mbps, 3 Mbps
>
>     600 milliseconds is the baseline delay. Longer delays definitely
>     happen during traffic peaks. Longer than one second delays happen
>     at traffic peak for traffic we classify as lower priority…
>
>     1 MB tests are web page object size tests. 1 GB tests are download
>     tests. (We would use 100 MB for upload tests but we have not
>     really done any yet.) We really have only been focusing large file
>     sizes since we have been focusing on what we lose without TCP
>     spoofing.
>
>     For error rates, I included 5% an 10% because they represent WiFi
>     extremes (and we did try them). But, those values barely work at
>     all. We mainly focus on 1%. There are actually two variants of
>     error rate. One is loss between the client and the satellite
>     terminal. (This is the loss TCP spoofing helps with.) The other is
>     loss between the satellite terminal and the satellite gateway
>     which connects to the Internet. We focus mostly on client side loss.
>
>     We mostly have been testing with a 250 Mbps download plan rate so
>     we can see how things work without being artificially limited...
>     We are easily able to fill this with TCP spoofing and ultimately
>     we want QUIC to be on par with that. (I think the best we have
>     seen Google QUIC do in recent testing is ~30 Mbps but a few years
>     ago it was only 3 Mbps so things are moving in a good direction.)
>     But, you cannot get plan rates like that outside of a lab right
>     now. In today’s real world, we run with 25 Mbps download plans for
>     customers. Some providers use ~10 Mbps download plan rates. We
>     look at the higher rates anyway because we are trying to look
>     ahead and satellite technology improves significantly over time.
>     We envision 100 Mbps download plans someday.
>
>     We have not done much upload testing to date. So, the upload plan
>     rate has mattered mostly for purposes of how much ACK traffic it
>     can carry. ACK reduction strategies are important for satellite
>     but we have not done much with this yet other than to observe that
>     30 Mbps of Google QUIC download traffic fills the upload pipe.
>
>     Not sure if this is what you were looking for but…
>
>     John
>
>
>
>     _______________________________________________
>
>     EToSat mailing list
>
>     [e-mail]  <mailto:[e-mail]>
>
>     https://www.ietf.org/mailman/listinfo/etosat
>
>
>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/etosat