Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opnfv
"MORTON, ALFRED C (AL)" <acmorton@att.com> Mon, 01 May 2017 19:00 UTC
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 271CE12EB27; Mon, 1 May 2017 12:00:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.4
X-Spam-Level:
X-Spam-Status: No, score=-5.4 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-2.8, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tTIYVlo8LrMx; Mon, 1 May 2017 12:00:30 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0a-00191d01.pphosted.com [67.231.149.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1611F129C60; Mon, 1 May 2017 11:57:26 -0700 (PDT)
Received: from pps.filterd (m0048589.ppops.net [127.0.0.1]) by m0048589.ppops.net-00191d01. (8.16.0.17/8.16.0.17) with SMTP id v41GjdOG020929; Mon, 1 May 2017 13:09:06 -0400
Received: from alpi155.enaf.aldc.att.com (sbcsmtp7.sbc.com [144.160.229.24]) by m0048589.ppops.net-00191d01. with ESMTP id 2a64yuy569-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 01 May 2017 13:09:06 -0400
Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id v41H94Rf024179; Mon, 1 May 2017 13:09:05 -0400
Received: from mlpi407.sfdc.sbc.com (mlpi407.sfdc.sbc.com [130.9.128.239]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id v41H8w4I024007 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 1 May 2017 13:09:00 -0400
Received: from clpi183.sldc.sbc.com (clpi183.sldc.sbc.com [135.41.1.46]) by mlpi407.sfdc.sbc.com (RSA Interceptor); Mon, 1 May 2017 17:08:33 GMT
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id v41H8XLO020240; Mon, 1 May 2017 12:08:33 -0500
Received: from mail-azure.research.att.com (mail-azure.research.att.com [135.207.255.18]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id v41H8R9W019462; Mon, 1 May 2017 12:08:28 -0500
Received: from exchange.research.att.com (njmtcas2.research.att.com [135.207.255.47]) by mail-azure.research.att.com (Postfix) with ESMTP id 133DEE03D5; Mon, 1 May 2017 13:08:27 -0400 (EDT)
Received: from njmtexg4.research.att.com ([fe80::8cd:baa3:219e:5bd4]) by njmtcas2.research.att.com ([fe80::d550:ec84:f872:cad9%15]) with mapi id 14.03.0319.002; Mon, 1 May 2017 13:08:26 -0400
From: "MORTON, ALFRED C (AL)" <acmorton@att.com>
To: Warren Kumari <warren@kumari.net>, "draft-ietf-bmwg-vswitch-opnfv@ietf.org" <draft-ietf-bmwg-vswitch-opnfv@ietf.org>, "sbanks@encrypted.net" <sbanks@encrypted.net>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: AD Eval of draft-ietf-bmwg-vswitch-opnfv
Thread-Index: AQHSvD+m1+lMKFsOy0SwDArzS5XAjKHV+UWwgAnHf1A=
Date: Mon, 01 May 2017 17:08:25 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF25F9080A@njmtexg4.research.att.com>
References: <CAHw9_iLhzXyV42a9cMVLMCcxNFDgFUODqXLyFnsRrq1XpXiQvA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [135.70.214.123]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-RSA-Inspected: yes
X-RSA-Classifications: public
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-05-01_12:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705010109
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/V0t7diyXa5HrD48KlwSHrRdR_sA>
Subject: Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opnfv
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 01 May 2017 19:00:33 -0000
Hi Warren, The latest version of this draft: https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/ addresses your comments plus some other editorial clean-up, and includes an abbreviations section (which should help with further reviews). thanks again for your review, Al (for the co-authors) > -----Original Message----- > From: MORTON, ALFRED C (AL) > Sent: Tuesday, April 25, 2017 7:43 AM > To: 'Warren Kumari'; draft-ietf-bmwg-vswitch-opnfv@ietf.org; > sbanks@encrypted.net; bmwg@ietf.org > Subject: RE: AD Eval of draft-ietf-bmwg-vswitch-opnfv > > Thanks for your review & comments, Warren; the edits will > take a few days to reach the top of the ToDo list. > Al > > > -----Original Message----- > > From: Warren Kumari [mailto:warren@kumari.net] > > Sent: Sunday, April 23, 2017 10:40 AM > > To: draft-ietf-bmwg-vswitch-opnfv@ietf.org; sbanks@encrypted.net; > > bmwg@ietf.org > > Subject: AD Eval of draft-ietf-bmwg-vswitch-opnfv > > > > Hi there, > > > > I've just done the AD eval of draft-ietf-bmwg-vswitch-opnfv - thank > > you for writing this, I think that it is a useful document. > > > > I had a few questions and comments which I think it would be useful > > to address before starting IETF LC. I'm not a benchmarking (or NFV / > > vswitch person), so some of these questions / comments may be very > > basic. > > > > I've noted my questions with [WK: ...]. My questions / comments look > > long, because I kept much of the text to make readability easier. > > > > --------- > > > > > > 2. Scope > > > > The primary purpose and scope of the memo is to inform the industry > > of work-in-progress that builds on the body of extensive BMWG > > literature and experience, and describe the extensions needed for > > benchmarking virtual switches. Inital feedback indicates that many > > > > [WK: s/Inital/Initial/] > > > > of these extensions may be applicable beyond the current scope (to > > hardware switches in the NFV Infrastructure and to virtual routers, > > for example). Additionally, this memo serves as a vehicle to > include > > more detail and commentary from BMWG and other Open Source > > communities, under BMWG's chartered work to characterize the NFV > > > > Infrastructure (a virtual switch is an important aspect of that > > infrastructure). > > > > The benchmarking covered in this memo should be applicable to many > > types of vswitches, and remain vswitch-agnostic to great degree. > > > > [WK: Please expand / define vswitch - perhaps in the parenthesis > above.] > > > > There has been no attempt to track and test all features of any > > specific vswitch implementation. > > > > ... > > > > 3.1. Comparison with Physical Network Functions > > > > To compare the performance of virtual designs and implementations > > with their physical counterparts, identical benchmarks are needed. > > BMWG has developed specifications for many network functions this > > memo re-uses existing benchmarks through references, and expands > them > > during development of new methods. > > > > [ WK: This sentence makes no sense / is missing some words / should be > > 2 sentences / something....] > > > > A key configuration aspect is the > > number of parallel cores required to achieve comparable performance > > with a given physical device, or whether some limit of scale was > > reached before the cores could achieve the comparable level. > > > > It's unlikely that the virtual switch will be the only application > > running on the SUT, so CPU utilization, Cache utilization, and > Memory > > footprint should also be recorded for the virtual implementations > of > > internetworking functions. > > > > ... > > > > 3.3. New Configuration Parameters > > > > A key consideration when conducting any sort of benchmark is trying > > to ensure the consistency and repeatability of test results. When > > benchmarking the performance of a vSwitch > > > > [ WK: Please choose a single term (vswitch / vSwitch) ] > > > > there are many factors that > > can affect the consistency of results, one key factor is matching > the > > various hardware and software details of the SUT. This section > lists > > some of the many new parameters which this project believes are > > critical to report in order to achieve repeatability. > > > > Hardware details including: > > > > o Platform details > > > > o Processor details > > > > o Memory information (type and size) > > > > [ WK: I would also think that the memory layout is important - if not, > > ignore. Oh, it is mentioned below ("Memory DIMM configurations"). This > > would be easier to read if you grouped memory together, processor / > > cores together, etc.] > > > > > > o Number of enabled cores > > > > o Number of cores used for the test > > > > o Number of physical NICs, as well as their details (manufacturer, > > versions, type and the PCI slot they are plugged into) > > > > o NIC interrupt configuration > > > > [WK: What about other NIC features (DDP, checksum magic, > > scatter-gather, etc)? Or are these not applicable? (if not, ignore) ] > > > > o BIOS version, release date and any configurations that were > > modified > > > > o CPU microcode level > > > > o Memory DIMM configurations (quad rank performance may not be the > > same as dual rank) in size, freq and slot locations > > > > o PCI configuration parameters (payload size, early ack option...) > > > > o Power management at all levels (ACPI sleep states, processor > > package, OS...) > > > > Software details including: > > > > o OS parameters and behavior (text vs graphical no one typing at > the > > console on one system) > > > > o OS version (for host and VNF) > > > > o Kernel version (for host and VNF) > > > > o GRUB boot parameters (for host and VNF) > > > > o Hypervisor details (Type and version) > > > > o Selected vSwitch, version number or commit id used > > > > o vSwitch launch command line if it has been parameterised > > > > o Memory allocation to the vSwitch > > > > o which NUMA node it is using, and how many memory channels > > > > o DPDK or any other SW dependency version number or commit id used > > > > o Memory allocation to a VM - if it's from Hugpages/elsewhere > > > > [WK: Hugpages? It this the emo version of hugepages? :-) ] > > > > > > o VM storage type: snapshot/independent persistent/independent > non- > > persistent > > > > o Number of VMs > > > > o Number of Virtual NICs (vNICs), versions, type and driver > > > > > > [ WK: What about IO virtualization stuff like SR-IOV and similar. Or > > not applicable?] > > > > o Number of virtual CPUs and their core affinity on the host > > > > o Number vNIC interrupt configuration > > > > o Thread affinitization for the applications (including the > vSwitch > > itself) on the host > > > > o Details of Resource isolation, such as CPUs designated for Host/ > > Kernel (isolcpu) and CPUs designated for specific processes > > (taskset). - Test duration. - Number of flows. > > > > Test Traffic Information: > > > > o Traffic type - UDP, TCP, IMIX / Other > > > > > > [ WK: I don't think that IMIX is a well known acronym. I think either > > define it (or ref: RFC6985?), or drop it (the section is "some of the > > many new parameters"...)] > > > > o Packet Sizes > > > > o Deployment Scenario > > > > 3.4. Flow classification > > > > Virtual switches group packets into flows by processing and > matching > > particular packet or frame header information, or by matching > packets > > based on the input ports. Thus a flow can be thought of a sequence > > of packets that have the same set of header field values (5-tuple) > or > > have arrived on the same port. Performance results can vary based > on > > the parameters the vSwitch uses to match for a flow. The > recommended > > flow classification parameters for any vSwitch performance tests > are: > > the input port, the source IP address, the destination IP address > and > > the Ethernet protocol type field. > > > > [ WK: Is it useful to say "5-tuple" above? To me 5-tuple is src ip, > > src port, dst ip, dst port, ip protocol. This says for vswitch it is > > input port, src ip, dst ip and eth protocol. The two close together > > may confuse readers - I think just drop "(5-tuple)". Also, this says > > "input port" - is that actually input interface / should it be called > > "physical port"? (when hearing IP and then port, people think L3 > > port)] > > > > > > o Scalability Tests to understand how the virtual switch performs > as > > the number of flows, active ports, complexity of the forwarding > > logic's configuration... it has to deal with increases. > > > > [ WK: Looks like got sidetracked in this sentence. ] > > > > > > --------- > > > > Thanks again, > > W > > > > -- > > I don't think the execution is relevant when it was obviously a bad > > idea in the first place. > > This is like putting rabid weasels in your pants, and later expressing > > regret at having chosen those particular rabid weasels and that pair > > of pants. > > ---maf
- [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opnfv Warren Kumari
- Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opn… MORTON, ALFRED C (AL)
- Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opn… Warren Kumari
- Re: [bmwg] AD Eval of draft-ietf-bmwg-vswitch-opn… MORTON, ALFRED C (AL)