Re: [EToSat] Download Regression Tests

[Kuhn Nicolas <[e-mail]>]

Hi,

We had initiated a regression test description in the QUIC 4 SATCOM document.
https://github.com/NicoKos/QUIC_HIGH_BDP/tree/master/ietf-document/draft-quic-4-sat

An updated version in a more readable format than XML is attached and in the GITHUB repo.
The scenario that you propose in more challenging than the ones we had discussed with Gorry at Singapore.
I have added your scenario in the document - but you may not want to keep them all.

What do you think ?

I guess we can at least remove the 10 Mbps one.
However, keeping the 50 Mbps / 10 Mbps shows a different asymmetry and it may be interesting to keep it.

More comments inline.

Cheers,

Nico

De : Christian Huitema <[e-mail]>
Envoyé : mardi 3 décembre 2019 23:28
À : Border, John <[e-mail]>; Kuhn Nicolas <[e-mail]>
Cc : [e-mail]
Objet : Re: [EToSat] Download Regression Tests


Thanks, that's very helpful.

I was doing test so far by simulating 600 ms delay, 100 Mbps download and 100 Mbps upload (symmetric). I will switch the simulated configuration to 250 Mbps upload and 3 Mbps download.

I am still working on the asymmetric link part, but I just did a test with 1GB with 250 Mbps symmetric link. With 0 error, it completes in a bit less than 36 sec on the Picoquic simulator -- effective data rate would be 222 Mbps. Not a problem for a one-off test, but that's a bit onerous for a non-regression test that shall run on every check-in.

I also tested 100 MB and 10 MB. The 100 MB test completes on the Picoquic simulator in 8.24 sec, average rate of 97 Mbps, and the 10MB test completes in 5.22 sec, average of 15 Mbps. The 10MB test is dominated by connection establishment and rate ramping issues (slow start, flow control, etc.) The 100 MB test has that, plus about 3 seconds of full speed transmission. I think 100MB is enough for the "non regression" function. It provides a good mix of establishment and data rate issues.

[NK] That size seems a good trade-off to me

I guess the file size is a simple parameter. I have two additional questions for setting up the simulations:

1) What is the maximum size of the buffer on the download path?

[NK] We propose to set the buffersize to the BDP - to begin with

2) When simulating the 1% loss rate, is there a pattern to the losses, or are they just randomly distributed with independent probability for each packet?

[NK] The 1% loss rate is not really random since both congestion and random losses occur in a bursty way. We will try to share more details very soon

-- Christian Huitema
On 12/4/2019 2:24 AM, Border, John wrote:

The following are the variables we play with:


  *   Delay - 600 milliseconds, 1000 milliseconds
  *   File Size - 1 MB, 10 MB, 100 MB, 1000 MB
  *   Packet Error Rate - 0.0%, 0.1%, 1%, 5%, 10%
  *   Plan Rate:

     *   Download (Internet to host)  - 10 Mbps, 25 Mbps, 100 Mbps, 250 Mbps
     *   Upload (host to Internet) - 3 Mbps, 10 Mbps, 25 Mbps

The ones in bold are the ones we focus on since all of the above result in a lot of combinations.  If I had to pick just two, right now I would go with:


  *   600 milliseconds, 1000 MB, 0.0%, 250 Mbps, 3 Mbps
  *   600 milliseconds, 1000 MB, 1.0%, 250 Mbps, 3 Mbps

600 milliseconds is the baseline delay.  Longer delays definitely happen during traffic peaks.  Longer than one second delays happen at traffic peak for traffic we classify as lower priority...

1 MB tests are web page object size tests.  1 GB tests are download tests.  (We would use 100 MB for upload tests but we have not really done any yet.)  We really have only been focusing large file sizes since we have been focusing on what we lose without TCP spoofing.

For error rates, I included 5% an 10% because they represent WiFi extremes (and we did try them).  But, those values barely work at all.  We mainly focus on 1%.  There are actually two variants of error rate.  One is loss between the client and the satellite terminal.  (This is the loss TCP spoofing helps with.)  The other is loss between the satellite terminal and the satellite gateway which connects to the Internet.  We focus mostly on client side loss.

We mostly have been testing with a 250 Mbps download plan rate so we can see how things work without being artificially limited..  We are easily able to fill this with TCP spoofing and ultimately we want QUIC to be on par with that.  (I think the best we have seen Google QUIC do in recent testing is ~30 Mbps but a few years ago it was only 3 Mbps so things are moving in a good direction.)   But, you cannot get plan rates like that outside of a lab right now.  In today's real world, we run with 25 Mbps download plans for customers.  Some providers use ~10 Mbps download plan rates.  We look at the higher rates anyway because we are trying to look ahead and satellite technology improves significantly over time.  We envision 100 Mbps download plans someday.

We have not done much upload testing to date.  So, the upload plan rate has mattered mostly for purposes of how much ACK traffic it can carry.  ACK reduction strategies are important for satellite but we have not done much with this yet other than to observe that 30 Mbps of Google QUIC download traffic fills the upload pipe.

Not sure if this is what you were looking for but...

John



_______________________________________________

EToSat mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/etosat