Re: [bmwg] Mean vs Median

[Paul Emmerich <[e-mail]>]

Hi,

On 10.11.15 02:18, Marius Georgescu wrote:
 > Can you give us more context (test setup; physical/virtualized 
tester/DUT; one tester/sender_receiver tester ... ) on these measurements?

Example 1: forwarding UDP packets with a userspace application at 20% of 
the maximum possible load between two ports (unidirectional, MoonGen as 
packet generator externally)

Median latency: 42 us, 99th perc: 45 us, 99.9th perc: 71 us

I then moved the same setup into a KVM VM connected via Open vSwitch and 
a VirtIO vNIC and the latency distribution changed (at 20% of the 
maximum load of the new setup). The traffic still came from an external 
host.

Median latency: 205 us, 99th perc: 341 us, 99.9th perc: 3030 us

Example 2: see my other email from earlier today. The long tail there 
isn't quite as long because it uses a "proper" packet forwarder instead 
of a userspace application. But the only reason it isn't as bad there is 
because there was no other load on the system.

Example 3: interesting things happen once other stuff runs on the same 
hypervisor. Note that this is a typical scenario for fancy NFV setups 
where you just deploy your VM whereever. The other VMs compete for 
resources and this can lead to excessive delays. There is an interesting 
paper by Whiteaker et al. which looks at latencies with Xen and 
Linux-VServer virtualization [1].


 > I think we should keep practicality in mind here. If we follow 
RFC2544.latency measurement, the frame stream has to be 2 min long. 2000 
min ~ 33h  of testing for just one test sounds unreasonable to me. I 
would agree to have a lower bound for the sample size as RFC2544 
actually recommends (n > 20).

Is there a good reason why this can't be changed to something more 
modern for a new standard? For example, I usually aim for 1000 
timestamped packets per second in my tests which works just fine.


Paul

[1] http://ccr.sigcomm.org/online/files/p39-v41n1f2-whiteakerPS.pdf