[bmwg] 回复： Comments on draft- huang-bmwg-virtual-network-performance-02

Hi, Hickman
Wish the following could be helpful:
1. In the real network, DUT must be running with VMs for different applications. Virtual tester is a kind of application, so sharing the same resource seems make sense. Of course virtual tester will somehow impact the performance of DUT. That's why we should calibrate virtual tester beforehand to find its own performance under different CPU/MEM allocations. Then we can retain enough resources for DUT.
2. Throughput Test: I agree with you. When packets loss happens, we should decrease the traffic instead of increasing vCPU for tester.
3. Latency Test: Yes, we can use Model A or B if all testers could become time synchronized.
4. Test consideration: You are right. Firstly find the baseline of virtual tester, then use it in a holiday. Thanks a lot!
--------------------LuHuang
China Mobile Research Institute
Mobile: +86 13810820540

    "Hickman, Brooks" <Brooks.Hickman@spirent.com> 于 2017年3月17日, 星期五, 下午 9:29 写道：

Given that there have been some issues raised on the draft- huang-bmwg-virtual-network-performance-02, I have reviewed the draft and have identified the following issues/problems that I believe need to be addressed:    Most importantly, I have a concern with the “virtual tester” sharing the same resources as the DUT(vSwitch) that is being tested and its effect on testing results. Specifically, by allocating host server resources(i.e. – vCPU’s/Memory) to the virtual tester, one virtual tester using test model A and two virtual testers using test model B, those resources cannot by assigned to the DUT. This might reduce the DUT performance results, such as lower throughput or higher latencies that might otherwise be attained if the aforementioned  resources were assigned to the DUT.    Throughput Test(Section 6.1)    In the throughput test process described in section 6.1.4(step 2) it notes to:    "Increase the number of vCPU's in the tester until the traffic has no packet loss"                                                                                 If the packet loss is a performance limitation of the DUT(vSwitch), then it seems to me that assigning more vCPU's to the virtual tester will have no effect on the packet loss and that the additional vCPU resources would need to be increased on the DUT(vSwitch) side, if available. If, in fact, increasing  the vCPU’s on the virtual tester reduces or eliminates the packet loss, it seems to me to be an indication that there is a performance issue with the virtual tester in its ability to process incoming test traffic.    Latency Test(Section 6.5)    In section 6.5, it discusses use of model A setup and substituting the virtual tester with an echo server. Then, assuming the "one-way delay" is half of the round trip. I don’t believe it’s a safe assumption that the one-way delay is half of the round trip time. There could be significant delta's between the two directions.  In addition, the setup for performing the latency test is not the same as the setup for the throughput test. Specifically, replacing the virtual tester with an echo server. While I understand the intent is to provide for a more accurate latency measurement, my concern is that the performance of the echo server is not known. Traffic is offered at the throughput determined in section 6.1, but will the echo server be able to handle the throughput rate? If not, then the latency measurement would include buffering by the echo server or worse, the packet(s) that is tagged for the latency measurement gets dropped altogether.    Also, is the reported latency the round trip measurement or the delta between with and without the DUT?    Test Consideration(Section 3)    Not saying there are issues here, but clarification may be helpful as I am unclear on the calibration requirements described in this section. Specifically, the section notes:    “Just like the DUT, tester running as software will have its own performance peak under various conditions.    And    “Tester's calibration is essential in benchmarking testing in a virtual environment. Furthermore, to reduce the enormous combination of various conditions, tester must be calibrated with the exact same combination and parameter settings the user wants to measure against the DUT.”    This, to me, infers that baseline testing of the virtual testers may be required, prior to testing the DUT(vSwitch) . In addition, those same parameters used in the baseline testing must remain in force once the DUT(vSwitch) is introduced. Is this correct?

Spirent Communications e-mail confidentiality.
------------------------------------------------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US, 

Spirent Communications,
27349 Agoura Road, Calabasas, CA, 91301, USA.
Tel No. 1-818-676- 2300 
_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www.ietf.org/mailman/listinfo/bmwg