Re: [bmwg] question regarding draft-dcn-bmwg-containerized-infra-10

[Gábor LENCSE <[e-mail]>]

Dear Minh-Ngoc,

Thank you very much for your quick reply. Please see my comments inline.

3/27/2023 8:46 AM keltezéssel, Minh Ngoc Tran írta:
> Hi Gábor,
>
> Thanks for your interest in the draft,
> We performed each test at least 5 times. And the average result of 
> them will be recorded as the final measurement value.

OK. I think 5 can be acceptable if your results are stable enough, but I 
usually use 10 or 20.

> Because of slide submission before hackathon time, the figures inside 
> the current slide is only the results of previous hackathon. This ietf 
> hackathon 116 we perform verification of these result with newer and 
> recent version of software (dpdk, ovs,...), so the example result  
> from ietf 116 hackathon I mentioned here will be a little bit different.

Yes, it seemed to be strange that results "From Hackathon 116" appeared 
in the slides. So they were from the previous one. OK.

I am curious about your new results. It would be nice if you could 
present them tomorrow!

> These are the example results
> - OVS-DPDK zero-packet loss (<0,1%) throughput in Gbps (1st to 5th 
> test) - 1518 bytes frame size: 26.17, 26.34, 26.14, 26.44, 26.07. 
> Standard deviation: 0.15
They are quite stable.
> - OVS-DPDK zero-packet loss (<0,1%) throughput in Gbps (1st to 5th 
> test) - 128 bytes frame size: 7.25, 7.41, 7.60, 7.30, 7.64. Standard 
> deviation: 0.17
They seem to be also acceptable, although I usually consider it as a 
kind of a warning sign if the difference of the minimum and maximum 
(7.64-7.25=0.39) is higher than than 5% of the average or median.

To be honest,  I expected much more scattered results due to 
Hyper-Threading enabled. So it was a nice surprise to me. :-)

BTW, it would be a great help for the readers to complementthe column 
diagrams with error bars showing either the standard deviation or 
minimum and maximum values. They give useful information about the 
stability of the results.

See you tomorrow at the BMWG meeting!

Gábor

> Thanks,
> Minh-Ngoc.
>
> On Mon, Mar 27, 2023 at 3:24 AM Gábor LENCSE <[e-mail]> wrote:
>
>     Dear Authors,
>
>     I have found your topic interesting and I have looked into your
>     slides (
>     https://datatracker.ietf.org/meeting/116/materials/slides-116-bmwg-considerations-for-benchmarking-network-performance-in-containerized-infrastructures
>     ) in advance.
>
>     I have a question, and I thought that perhaps it is better if I
>     ask it in advance and not during the meeting.
>
>     I have seen that you have some nice results "From Hackathon 116".
>     There are several graphs that show throughput (measured in Gbps)
>     as a function of frame size and some different other parameters
>     (e.g. number of CPU cores, etc.)
>
>     I wonder how many times the tests were repeated, and what
>     summarizing function was used (e.g. average, median) to express
>     the results of (hopefully) multiple tests only with a single number.
>     I would be interested in the standard deviation and/or minimum,
>     maximum, etc., (whatever you chose) of the results so that I can
>     see that stable or scattered nature of the results.
>
>     I ask this, because I have noticed among the BIOS settings:
>     "Intel(R) Hyper-Threading Tech Enabled", which I always disable to
>     avoid unstable results.
>
>     Best regards,
>
>     Gábor
>     _______________________________________________
>     bmwg mailing list
>     [e-mail]
>     https://www.ietf.org/mailman/listinfo/bmwg
>
>
>
> -- 
>
> __________________________________
>
> *Minh-Ngoc Tran*
>
> *Distributed Cloud and Network lab*
>
> *Soongsil University *
>
> *511 Sangdo-dong Dongjak-gu*
>
> *Seoul, 156-743 Korea*
>
> *TEL : [phone] *
>