Re: [bmwg] Mean vs Median

[Marius Georgescu <[e-mail]>]

Hi Paul,


> On Nov 11, 2015, at 23:11, Paul Emmerich <[e-mail]> wrote:
> 
> Hi,
> 
> On 10.11.15 02:18, Marius Georgescu wrote:
> > Can you give us more context (test setup; physical/virtualized tester/DUT; one tester/sender_receiver tester ... ) on these measurements?
> 
> Example 1: forwarding UDP packets with a userspace application at 20% of the maximum possible load between two ports (unidirectional, MoonGen as packet generator externally)
> 
> Median latency: 42 us, 99th perc: 45 us, 99.9th perc: 71 us
> 
> I then moved the same setup into a KVM VM connected via Open vSwitch and a VirtIO vNIC and the latency distribution changed (at 20% of the maximum load of the new setup). The traffic still came from an external host.
> 
> Median latency: 205 us, 99th perc: 341 us, 99.9th perc: 3030 us
> Example 2: see my other email from earlier today. The long tail there isn't quite as long because it uses a "proper" packet forwarder instead of a userspace application. But the only reason it isn't as bad there is because there was no other load on the system.
> 

Thanks for sharing the results. They are indeed quite relevant. I would like to see the rest of the draft, if possible. 

> Example 3: interesting things happen once other stuff runs on the same hypervisor. Note that this is a typical scenario for fancy NFV setups where you just deploy your VM whereever. The other VMs compete for resources and this can lead to excessive delays. There is an interesting paper by Whiteaker et al. which looks at latencies with Xen and Linux-VServer virtualization [1].
> 
> 
> > I think we should keep practicality in mind here. If we follow RFC2544.latency measurement, the frame stream has to be 2 min long. 2000 min ~ 33h  of testing for just one test sounds unreasonable to me. I would agree to have a lower bound for the sample size as RFC2544 actually recommends (n > 20).
> 
> Is there a good reason why this can't be changed to something more modern for a new standard? For example, I usually aim for 1000 timestamped packets per second in my tests which works just fine.

I don’t think it would be a problem to have n>1000, as long as the test time would be 120s (or something more reasonable than 33h).  

> 
> 
> Paul
> 
> [1] http://ccr.sigcomm.org/online/files/p39-v41n1f2-whiteakerPS.pdf