[bmwg] Comments on draft- huang-bmwg-virtual-network-performance-02

"Hickman, Brooks" <Brooks.Hickman@spirent.com> Fri, 17 March 2017 04:11 UTC

Return-Path: <Brooks.Hickman@spirent.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 95E37129BE9 for <bmwg@ietfa.amsl.com>; Thu, 16 Mar 2017 21:11:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.921
X-Spam-Level:
X-Spam-Status: No, score=-1.921 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=spirent1.onmicrosoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vWa8b1v2n4TT for <bmwg@ietfa.amsl.com>; Thu, 16 Mar 2017 21:11:03 -0700 (PDT)
Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0137.outbound.protection.outlook.com [104.47.38.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6708E129BDB for <bmwg@ietf.org>; Thu, 16 Mar 2017 21:11:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spirent1.onmicrosoft.com; s=selector1-spirent-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ZSzB3hpt84ZVptZVVEdXsWFYBhL1Ri1FMoKH0rz3pWI=; b=m9Ubw7xwpOl8pcJgL52/RVehi8nkTGVWtkBVbcLVSHj+LQb1Un2GwrGGCycWlWA9F85+nLO4BpgNeYAGfFoOuNJ7kPrKBxMa4Jov55iGNLyuTXbSiaKkunoG+cIcRJwr5ocH2uicQyKLf9yquiTA3G+gExzCIp30AbhP3Mpz+v0=
Received: from DM5PR10MB1610.namprd10.prod.outlook.com (10.172.34.143) by DM5PR10MB1611.namprd10.prod.outlook.com (10.172.34.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.977.11; Fri, 17 Mar 2017 04:11:01 +0000
Received: from DM5PR10MB1610.namprd10.prod.outlook.com ([10.172.34.143]) by DM5PR10MB1610.namprd10.prod.outlook.com ([10.172.34.143]) with mapi id 15.01.0947.020; Fri, 17 Mar 2017 04:11:01 +0000
From: "Hickman, Brooks" <Brooks.Hickman@spirent.com>
To: "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: Comments on draft- huang-bmwg-virtual-network-performance-02
Thread-Index: AdKezzXDxqxQIjZASLG0u/Tj67MSUw==
Date: Fri, 17 Mar 2017 04:11:01 +0000
Message-ID: <DM5PR10MB161005B77BD2E308253A08B49F390@DM5PR10MB1610.namprd10.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: ietf.org; dkim=none (message not signed) header.d=none;ietf.org; dmarc=none action=none header.from=spirent.com;
x-originating-ip: [108.192.130.1]
x-ms-office365-filtering-correlation-id: 5f628e81-9605-4f24-eabb-08d46ceb9fa2
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DM5PR10MB1611;
x-microsoft-exchange-diagnostics: 1; DM5PR10MB1611; 7:e8CP+udM5JGT3F6yw6EVRhzH6TWTujutwJY7OLVQk5oCBmA7+Aqt/VscltjVDMtSIoeaY8grNefT6nIDJQS/Dgm5o08QHLE82HH7Cv7e1qxofHx+RwFFue7bhq/UfplNUVwRd7+QPrcqMAu9AmWrD5fg3zMuhdx7PytezRZgUbTEKEDM3FqwafbefSUuV5d4/v7uV92ShghJx5j+55c1Dhp66kXvWg/cz/vMS8KXPb29S0ftU26yfpbMfc73X0r8eEU/WcXN39pO1rMiNXJfvLHloDz6cUzBKH/3u9BmV24zS30G1hfF8lL3vsKfSFbXH1vLu3FUwtqIRY4b3Hq06A==
x-disclaimer: Yes
x-microsoft-antispam-prvs: <DM5PR10MB16117B6433200F631C7F45A19F390@DM5PR10MB1611.namprd10.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(158342451672863)(21748063052155);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123560025)(20161123564025)(20161123555025)(20161123562025)(20161123558025)(6072148); SRVR:DM5PR10MB1611; BCL:0; PCL:0; RULEID:; SRVR:DM5PR10MB1611;
x-forefront-prvs: 0249EFCB0B
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(979002)(6009001)(39840400002)(39450400003)(39410400002)(85714005)(9686003)(5660300001)(53936002)(2900100001)(105586002)(230783001)(6306002)(81166006)(54896002)(8676002)(1730700003)(66066001)(54356999)(2501003)(33656002)(38730400002)(7736002)(2351001)(110136004)(8936002)(6916009)(50986999)(3280700002)(2906002)(25786008)(3660700001)(189998001)(790700001)(86362001)(7696004)(122556002)(6506006)(6116002)(5640700003)(6436002)(102836003)(99286003)(3846002)(74316002)(55016002)(77096006)(5630700001)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1102; SCL:1; SRVR:DM5PR10MB1611; H:DM5PR10MB1610.namprd10.prod.outlook.com; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; LANG:en;
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative; boundary="_000_DM5PR10MB161005B77BD2E308253A08B49F390DM5PR10MB1610namp_"
MIME-Version: 1.0
X-OriginatorOrg: spirent.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2017 04:11:01.4368 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: eb68cad0-6bd5-483f-a802-0f72f974373f
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1611
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/CxIo6jJbGJKQofxSRDc6uOCOK1g>
Subject: [bmwg] Comments on draft- huang-bmwg-virtual-network-performance-02
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 17 Mar 2017 13:28:52 -0000

Given that there have been some issues raised on the draft- huang-bmwg-virtual-network-performance-02, I have reviewed the draft and have identified the following issues/problems that I believe need to be addressed:

Most importantly, I have a concern with the "virtual tester" sharing the same resources as the DUT(vSwitch) that is being tested and its effect on testing results. Specifically, by allocating host server resources(i.e. - vCPU's/Memory) to the virtual tester, one virtual tester using test model A and two virtual testers using test model B, those resources cannot by assigned to the DUT. This might reduce the DUT performance results, such as lower throughput or higher latencies that might otherwise be attained if the aforementioned  resources were assigned to the DUT.

Throughput Test(Section 6.1)



In the throughput test process described in section 6.1.4(step 2) it notes to:



"Increase the number of vCPU's in the tester until the traffic has no packet loss"



If the packet loss is a performance limitation of the DUT(vSwitch), then it seems to me that assigning more vCPU's to the virtual tester will have no effect on the packet loss and that the additional vCPU resources would need to be increased on the DUT(vSwitch) side, if available. If, in fact, increasing  the vCPU's on the virtual tester reduces or eliminates the packet loss, it seems to me to be an indication that there is a performance issue with the virtual tester in its ability to process incoming test traffic.


Latency Test(Section 6.5)



In section 6.5, it discusses use of model A setup and substituting the virtual tester with an echo server. Then, assuming the "one-way delay" is half of the round trip. I don't believe it's a safe assumption that the one-way delay is half of the round trip time. There could be significant delta's between the two directions.

In addition, the setup for performing the latency test is not the same as the setup for the throughput test. Specifically, replacing the virtual tester with an echo server. While I understand the intent is to provide for a more accurate latency measurement, my concern is that the performance of the echo server is not known. Traffic is offered at the throughput determined in section 6.1, but will the echo server be able to handle the throughput rate? If not, then the latency measurement would include buffering by the echo server or worse, the packet(s) that is tagged for the latency measurement gets dropped altogether.



Also, is the reported latency the round trip measurement or the delta between with and without the DUT?



Test Consideration(Section 3)


Not saying there are issues here, but clarification may be helpful as I am unclear on the calibration requirements described in this section. Specifically, the section notes:


"Just like the DUT, tester running as software will have its own performance peak under various conditions.

And


"Tester's calibration is essential in benchmarking testing in a virtual environment. Furthermore, to reduce the enormous combination

of various conditions, tester must be calibrated with the exact same combination and parameter settings the user wants to measure against the DUT."

This, to me, infers that baseline testing of the virtual testers may be required, prior to testing the DUT(vSwitch) . In addition, those same parameters used in the baseline testing must remain in force once the DUT(vSwitch) is introduced. Is this correct?





Spirent Communications e-mail confidentiality.
------------------------------------------------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US,

Spirent Communications,
27349 Agoura Road, Calabasas, CA, 91301, USA.
Tel No. 1-818-676- 2300