Re: [bmwg] Mean vs Median

[Stenio Fernandes <[e-mail]>]

Very interesting results, Paul. I'd like to have a look at the results in
the journal paper.

In the past, statisticians struggled with a shortage of samples to do their
stuff. This is not the case for the computer networking people as we are
constantly flooded with samples.

The discussions so far have led me to conclude that in the context here,
there is no need to make any assumptions on the sample set. Any specific
measure of centrality or dispersion might not be precise enough in some
cases. As stated by others, mean/median would work for well-behaved (e.g.,
normally distributed) data, but would not work for multi-modal or
heavy-tailed ones. Recall that heavy-tailed distributions are usually
characterized by the shape and location parameters instead of mean and
variance. Regarding the number of samples, it is really tough to
characterize heavy-tailed or multi-modal distributions with a few samples,
even using advanced algorithms for maximum-likelihood estimation.

 Stenio

On Wed, Nov 11, 2015 at 12:11 PM, Paul Emmerich <[e-mail]>
wrote:

> Hi,
>
> On 10.11.15 02:18, Marius Georgescu wrote:
> > Can you give us more context (test setup; physical/virtualized
> tester/DUT; one tester/sender_receiver tester ... ) on these measurements?
>
> Example 1: forwarding UDP packets with a userspace application at 20% of
> the maximum possible load between two ports (unidirectional, MoonGen as
> packet generator externally)
>
> Median latency: 42 us, 99th perc: 45 us, 99.9th perc: 71 us
>
> I then moved the same setup into a KVM VM connected via Open vSwitch and a
> VirtIO vNIC and the latency distribution changed (at 20% of the maximum
> load of the new setup). The traffic still came from an external host.
>
> Median latency: 205 us, 99th perc: 341 us, 99.9th perc: 3030 us
>
> Example 2: see my other email from earlier today. The long tail there
> isn't quite as long because it uses a "proper" packet forwarder instead of
> a userspace application. But the only reason it isn't as bad there is
> because there was no other load on the system.
>
> Example 3: interesting things happen once other stuff runs on the same
> hypervisor. Note that this is a typical scenario for fancy NFV setups where
> you just deploy your VM whereever. The other VMs compete for resources and
> this can lead to excessive delays. There is an interesting paper by
> Whiteaker et al. which looks at latencies with Xen and Linux-VServer
> virtualization [1].
>
>
> > I think we should keep practicality in mind here. If we follow
> RFC2544.latency measurement, the frame stream has to be 2 min long. 2000
> min ~ 33h  of testing for just one test sounds unreasonable to me. I would
> agree to have a lower bound for the sample size as RFC2544 actually
> recommends (n > 20).
>
> Is there a good reason why this can't be changed to something more modern
> for a new standard? For example, I usually aim for 1000 timestamped packets
> per second in my tests which works just fine.
>
>
> Paul
>
> [1] http://ccr.sigcomm.org/online/files/p39-v41n1f2-whiteakerPS.pdf
>
>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg
>



-- 
Prof. Stenio Fernandes
CIn/UFPE
http://www.steniofernandes.com