Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opnfv
"MORTON, ALFRED C (AL)" <acmorton@att.com> Tue, 25 April 2017 11:44 UTC
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1D39C12EC2D; Tue, 25 Apr 2017 04:44:07 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.4
X-Spam-Level:
X-Spam-Status: No, score=-5.4 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-2.8, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZypTGAFNjDKJ; Tue, 25 Apr 2017 04:44:05 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C65FC12EC2F; Tue, 25 Apr 2017 04:44:04 -0700 (PDT)
Received: from pps.filterd (m0049463.ppops.net [127.0.0.1]) by m0049463.ppops.net-00191d01. (8.16.0.17/8.16.0.17) with SMTP id v3PBZW6k012878; Tue, 25 Apr 2017 07:44:01 -0400
Received: from alpi155.enaf.aldc.att.com (sbcsmtp7.sbc.com [144.160.229.24]) by m0049463.ppops.net-00191d01. with ESMTP id 2a25as13nd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 Apr 2017 07:44:00 -0400
Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id v3PBi0ZT015785; Tue, 25 Apr 2017 07:44:00 -0400
Received: from mlpi407.sfdc.sbc.com (mlpi407.sfdc.sbc.com [130.9.128.239]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id v3PBho4f015716 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 25 Apr 2017 07:43:52 -0400
Received: from clpi183.sldc.sbc.com (clpi183.sldc.sbc.com [135.41.1.46]) by mlpi407.sfdc.sbc.com (RSA Interceptor); Tue, 25 Apr 2017 11:43:32 GMT
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id v3PBhWdr032158; Tue, 25 Apr 2017 06:43:32 -0500
Received: from mail-azure.research.att.com (mail-azure.research.att.com [135.207.255.18]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id v3PBhOtl031783; Tue, 25 Apr 2017 06:43:24 -0500
Received: from exchange.research.att.com (njmtcas2.research.att.com [135.207.255.47]) by mail-azure.research.att.com (Postfix) with ESMTP id ACBE3E022F; Tue, 25 Apr 2017 07:43:23 -0400 (EDT)
Received: from njmtexg4.research.att.com ([fe80::8cd:baa3:219e:5bd4]) by njmtcas2.research.att.com ([fe80::d550:ec84:f872:cad9%15]) with mapi id 14.03.0319.002; Tue, 25 Apr 2017 07:43:23 -0400
From: "MORTON, ALFRED C (AL)" <acmorton@att.com>
To: Warren Kumari <warren@kumari.net>, "draft-ietf-bmwg-vswitch-opnfv@ietf.org" <draft-ietf-bmwg-vswitch-opnfv@ietf.org>, "sbanks@encrypted.net" <sbanks@encrypted.net>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: AD Eval of draft-ietf-bmwg-vswitch-opnfv
Thread-Index: AQHSvD+m1+lMKFsOy0SwDArzS5XAjKHV+UWw
Date: Tue, 25 Apr 2017 11:43:22 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF25F89CE8@njmtexg4.research.att.com>
References: <CAHw9_iLhzXyV42a9cMVLMCcxNFDgFUODqXLyFnsRrq1XpXiQvA@mail.gmail.com>
In-Reply-To: <CAHw9_iLhzXyV42a9cMVLMCcxNFDgFUODqXLyFnsRrq1XpXiQvA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [192.144.77.36]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-RSA-Inspected: yes
X-RSA-Classifications: public
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-04-25_07:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704250209
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/GulDAPxfbYzcGD7rpdAMsPoTDK8>
Subject: Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opnfv
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 25 Apr 2017 11:44:07 -0000
Thanks for your review & comments, Warren; the edits will take a few days to reach the top of the ToDo list. Al > -----Original Message----- > From: Warren Kumari [mailto:warren@kumari.net] > Sent: Sunday, April 23, 2017 10:40 AM > To: draft-ietf-bmwg-vswitch-opnfv@ietf.org; sbanks@encrypted.net; > bmwg@ietf.org > Subject: AD Eval of draft-ietf-bmwg-vswitch-opnfv > > Hi there, > > I've just done the AD eval of draft-ietf-bmwg-vswitch-opnfv - thank > you for writing this, I think that it is a useful document. > > I had a few questions and comments which I think it would be useful > to address before starting IETF LC. I'm not a benchmarking (or NFV / > vswitch person), so some of these questions / comments may be very > basic. > > I've noted my questions with [WK: ...]. My questions / comments look > long, because I kept much of the text to make readability easier. > > --------- > > > 2. Scope > > The primary purpose and scope of the memo is to inform the industry > of work-in-progress that builds on the body of extensive BMWG > literature and experience, and describe the extensions needed for > benchmarking virtual switches. Inital feedback indicates that many > > [WK: s/Inital/Initial/] > > of these extensions may be applicable beyond the current scope (to > hardware switches in the NFV Infrastructure and to virtual routers, > for example). Additionally, this memo serves as a vehicle to include > more detail and commentary from BMWG and other Open Source > communities, under BMWG's chartered work to characterize the NFV > > Infrastructure (a virtual switch is an important aspect of that > infrastructure). > > The benchmarking covered in this memo should be applicable to many > types of vswitches, and remain vswitch-agnostic to great degree. > > [WK: Please expand / define vswitch - perhaps in the parenthesis above.] > > There has been no attempt to track and test all features of any > specific vswitch implementation. > > ... > > 3.1. Comparison with Physical Network Functions > > To compare the performance of virtual designs and implementations > with their physical counterparts, identical benchmarks are needed. > BMWG has developed specifications for many network functions this > memo re-uses existing benchmarks through references, and expands them > during development of new methods. > > [ WK: This sentence makes no sense / is missing some words / should be > 2 sentences / something....] > > A key configuration aspect is the > number of parallel cores required to achieve comparable performance > with a given physical device, or whether some limit of scale was > reached before the cores could achieve the comparable level. > > It's unlikely that the virtual switch will be the only application > running on the SUT, so CPU utilization, Cache utilization, and Memory > footprint should also be recorded for the virtual implementations of > internetworking functions. > > ... > > 3.3. New Configuration Parameters > > A key consideration when conducting any sort of benchmark is trying > to ensure the consistency and repeatability of test results. When > benchmarking the performance of a vSwitch > > [ WK: Please choose a single term (vswitch / vSwitch) ] > > there are many factors that > can affect the consistency of results, one key factor is matching the > various hardware and software details of the SUT. This section lists > some of the many new parameters which this project believes are > critical to report in order to achieve repeatability. > > Hardware details including: > > o Platform details > > o Processor details > > o Memory information (type and size) > > [ WK: I would also think that the memory layout is important - if not, > ignore. Oh, it is mentioned below ("Memory DIMM configurations"). This > would be easier to read if you grouped memory together, processor / > cores together, etc.] > > > o Number of enabled cores > > o Number of cores used for the test > > o Number of physical NICs, as well as their details (manufacturer, > versions, type and the PCI slot they are plugged into) > > o NIC interrupt configuration > > [WK: What about other NIC features (DDP, checksum magic, > scatter-gather, etc)? Or are these not applicable? (if not, ignore) ] > > o BIOS version, release date and any configurations that were > modified > > o CPU microcode level > > o Memory DIMM configurations (quad rank performance may not be the > same as dual rank) in size, freq and slot locations > > o PCI configuration parameters (payload size, early ack option...) > > o Power management at all levels (ACPI sleep states, processor > package, OS...) > > Software details including: > > o OS parameters and behavior (text vs graphical no one typing at the > console on one system) > > o OS version (for host and VNF) > > o Kernel version (for host and VNF) > > o GRUB boot parameters (for host and VNF) > > o Hypervisor details (Type and version) > > o Selected vSwitch, version number or commit id used > > o vSwitch launch command line if it has been parameterised > > o Memory allocation to the vSwitch > > o which NUMA node it is using, and how many memory channels > > o DPDK or any other SW dependency version number or commit id used > > o Memory allocation to a VM - if it's from Hugpages/elsewhere > > [WK: Hugpages? It this the emo version of hugepages? :-) ] > > > o VM storage type: snapshot/independent persistent/independent non- > persistent > > o Number of VMs > > o Number of Virtual NICs (vNICs), versions, type and driver > > > [ WK: What about IO virtualization stuff like SR-IOV and similar. Or > not applicable?] > > o Number of virtual CPUs and their core affinity on the host > > o Number vNIC interrupt configuration > > o Thread affinitization for the applications (including the vSwitch > itself) on the host > > o Details of Resource isolation, such as CPUs designated for Host/ > Kernel (isolcpu) and CPUs designated for specific processes > (taskset). - Test duration. - Number of flows. > > Test Traffic Information: > > o Traffic type - UDP, TCP, IMIX / Other > > > [ WK: I don't think that IMIX is a well known acronym. I think either > define it (or ref: RFC6985?), or drop it (the section is "some of the > many new parameters"...)] > > o Packet Sizes > > o Deployment Scenario > > 3.4. Flow classification > > Virtual switches group packets into flows by processing and matching > particular packet or frame header information, or by matching packets > based on the input ports. Thus a flow can be thought of a sequence > of packets that have the same set of header field values (5-tuple) or > have arrived on the same port. Performance results can vary based on > the parameters the vSwitch uses to match for a flow. The recommended > flow classification parameters for any vSwitch performance tests are: > the input port, the source IP address, the destination IP address and > the Ethernet protocol type field. > > [ WK: Is it useful to say "5-tuple" above? To me 5-tuple is src ip, > src port, dst ip, dst port, ip protocol. This says for vswitch it is > input port, src ip, dst ip and eth protocol. The two close together > may confuse readers - I think just drop "(5-tuple)". Also, this says > "input port" - is that actually input interface / should it be called > "physical port"? (when hearing IP and then port, people think L3 > port)] > > > o Scalability Tests to understand how the virtual switch performs as > the number of flows, active ports, complexity of the forwarding > logic's configuration... it has to deal with increases. > > [ WK: Looks like got sidetracked in this sentence. ] > > > --------- > > Thanks again, > W > > -- > I don't think the execution is relevant when it was obviously a bad > idea in the first place. > This is like putting rabid weasels in your pants, and later expressing > regret at having chosen those particular rabid weasels and that pair > of pants. > ---maf
- [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opnfv Warren Kumari
- Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opn… MORTON, ALFRED C (AL)
- Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opn… Warren Kumari
- Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opn… MORTON, ALFRED C (AL)