Re: [EToSat] Download Regression Tests

[Christian Huitema <[e-mail]>]

Thanks, that's very helpful.

I was doing test so far by simulating 600 ms delay, 100 Mbps download
and 100 Mbps upload (symmetric). I will switch the simulated
configuration to 250 Mbps upload and 3 Mbps download.

I am still working on the asymmetric link part, but I just did a test
with 1GB with 250 Mbps symmetric link. With 0 error, it completes in a
bit less than 36 sec on the Picoquic simulator -- effective data rate
would be 222 Mbps. Not a problem for a one-off test, but that's a bit
onerous for a non-regression test that shall run on every check-in.

I also tested 100 MB and 10 MB. The 100 MB test completes on the
Picoquic simulator in 8.24 sec, average rate of 97 Mbps, and the 10MB
test completes in 5.22 sec, average of 15 Mbps. The 10MB test is
dominated by connection establishment and rate ramping issues (slow
start, flow control, etc.) The 100 MB test has that, plus about 3
seconds of full speed transmission. I think 100MB is enough for the "non
regression" function. It provides a good mix of establishment and data
rate issues.

I guess the file size is a simple parameter. I have two additional
questions for setting up the simulations:

1) What is the maximum size of the buffer on the download path?

2) When simulating the 1% loss rate, is there a pattern to the losses,
or are they just randomly distributed with independent probability for
each packet?

-- Christian Huitema

On 12/4/2019 2:24 AM, Border, John wrote:
>
>  
>
> The following are the variables we play with:
>
>  
>
>   * Delay – *600 milliseconds*, 1000 milliseconds
>   * File Size – *1 MB*, 10 MB, 100 MB, *1000 MB*
>   * Packet Error Rate – *0.0%*, 0.1%, *1%*, 5%, 10%
>   * Plan Rate:
>       o Download (Internet to host)  - 10 Mbps, 25 Mbps, 100 Mbps,
>         *250 Mbps*
>       o Upload (host to Internet) – *3 Mbps*, 10 Mbps, 25 Mbps
>
>  
>
> The ones in bold are the ones we focus on since all of the above
> result in a lot of combinations.  If I had to pick just two, right now
> I would go with:
>
>  
>
>   * 600 milliseconds, 1000 MB, 0.0%, 250 Mbps, 3 Mbps
>   * 600 milliseconds, 1000 MB, 1.0%, 250 Mbps, 3 Mbps
>
>  
>
> 600 milliseconds is the baseline delay.  Longer delays definitely
> happen during traffic peaks.  Longer than one second delays happen at
> traffic peak for traffic we classify as lower priority…
>
>  
>
> 1 MB tests are web page object size tests.  1 GB tests are download
> tests.  (We would use 100 MB for upload tests but we have not really
> done any yet.)  We really have only been focusing large file sizes
> since we have been focusing on what we lose without TCP spoofing.
>
>  
>
> For error rates, I included 5% an 10% because they represent WiFi
> extremes (and we did try them).  But, those values barely work at all.
>  We mainly focus on 1%.  There are actually two variants of error
> rate.  One is loss between the client and the satellite terminal. 
> (This is the loss TCP spoofing helps with.)  The other is loss between
> the satellite terminal and the satellite gateway which connects to the
> Internet.  We focus mostly on client side loss.
>
>  
>
> We mostly have been testing with a 250 Mbps download plan rate so we
> can see how things work without being artificially limited..  We are
> easily able to fill this with TCP spoofing and ultimately we want QUIC
> to be on par with that.  (I think the best we have seen Google QUIC do
> in recent testing is ~30 Mbps but a few years ago it was only 3 Mbps
> so things are moving in a good direction.)   But, you cannot get plan
> rates like that outside of a lab right now.  In today’s real world, we
> run with 25 Mbps download plans for customers.  Some providers use ~10
> Mbps download plan rates.  We look at the higher rates anyway because
> we are trying to look ahead and satellite technology improves
> significantly over time.  We envision 100 Mbps download plans someday.
>
>  
>
> We have not done much upload testing to date.  So, the upload plan
> rate has mattered mostly for purposes of how much ACK traffic it can
> carry.  ACK reduction strategies are important for satellite but we
> have not done much with this yet other than to observe that 30 Mbps of
> Google QUIC download traffic fills the upload pipe.
>
>  
>
> Not sure if this is what you were looking for but…
>
>  
>
> John
>
>
> _______________________________________________
> EToSat mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/etosat