Re: [bmwg] Mean vs Median

Hi Paul,

Please see my answers inline.

> On Nov 10, 2015, at 02:40, Paul Emmerich <emmericp@net.in.tum.de> wrote:
> 
> Hi,
> 
> On 03.11.15 09:45, Stenio Fernandes wrote:
>> a word of caution here... a number of phenomena in computer networks
>> follows a heavy-tailed probability distribution function, which means
>> that there is a non-negligible probability that a random variable will
>> take huge values. these values might be erroneously considered as outliers.
> 
> this is a really important point. I have benchmarked software where the 99th percentile of the latency is twice the average/median and the 99.9th percentile ten times the average/median.

Can you give us more context (test setup; physical/virtualized tester/DUT; one tester/sender_receiver tester ... ) on these measurements?

> This is an important performance characteristic for latency-sensitive applications that isn't captured by taking just 20 measurements. So I'd really like to see a standard that calls for thousands of latency measurements to capture this properly.
> 

I think we should keep practicality in mind here. If we follow RFC2544.latency measurement, the frame stream has to be 2 min long. 2000 min ~ 33h  of testing for just one test sounds unreasonable to me. I would agree to have a lower bound for the sample size as RFC2544 actually recommends (n > 20). 

> You can also get interesting insights into a black-box device by looking at histograms/probability density functions. For example, you can figure out if the device processes packets in batches, estimate the batch size, figure out at which rates interrupt moderation algorithms change etc. (This is, of course, not really a performance metric, just an interesting insight.)
> 

I agree this is an interesting insight. It can also be the base for a decision between summarizing functions. However, in the light of consistency and simplicity of the methodology, I think we would need to recommend one function. We could do that depending on the metric/DUT characteristics, previous testing behavior …

> 
> Paul
> 
> -- 
> Paul Emmerich
> Technical University of Munich (TUM)
> Department of Informatics
> Chair for Network Architectures and Services
> 
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg