Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

"MORTON, ALFRED C (AL)" <acm@research.att.com> Sat, 23 May 2020 17:40 UTC

Return-Path: <acm@research.att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7A3123A0D7E for <bmwg@ietfa.amsl.com>; Sat, 23 May 2020 10:40:37 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level:
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DCRxvJF5a-Ek for <bmwg@ietfa.amsl.com>; Sat, 23 May 2020 10:40:33 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 702493A0D8B for <bmwg@ietf.org>; Sat, 23 May 2020 10:40:33 -0700 (PDT)
Received: from pps.filterd (m0049459.ppops.net [127.0.0.1]) by m0049459.ppops.net-00191d01. (8.16.0.42/8.16.0.42) with SMTP id 04NHVuds044408; Sat, 23 May 2020 13:40:20 -0400
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com [144.160.112.28]) by m0049459.ppops.net-00191d01. with ESMTP id 3177ed0qg8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 23 May 2020 13:40:20 -0400
Received: from enaf.dadc.sbc.com (localhost [127.0.0.1]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id 04NHeJKw068571; Sat, 23 May 2020 12:40:19 -0500
Received: from zlp30496.vci.att.com (zlp30496.vci.att.com [135.46.181.157]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id 04NHeDWN068405 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 23 May 2020 12:40:14 -0500
Received: from zlp30496.vci.att.com (zlp30496.vci.att.com [127.0.0.1]) by zlp30496.vci.att.com (Service) with ESMTP id 4B029401B0B5; Sat, 23 May 2020 17:40:13 +0000 (GMT)
Received: from clph811.sldc.sbc.com (unknown [135.41.107.12]) by zlp30496.vci.att.com (Service) with ESMTP id 1ACAC4014CA0; Sat, 23 May 2020 17:40:13 +0000 (GMT)
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clph811.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id 04NHeCT1082149; Sat, 23 May 2020 12:40:12 -0500
Received: from mail-blue.research.att.com (mail-blue.research.att.com [135.207.178.11]) by clph811.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id 04NHe6lA080558; Sat, 23 May 2020 12:40:06 -0500
Received: from exchange.research.att.com (njbdcas1.research.att.com [135.197.255.61]) by mail-blue.research.att.com (Postfix) with ESMTPS id 7745310A301B; Sat, 23 May 2020 13:40:05 -0400 (EDT)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0468.000; Sat, 23 May 2020 13:40:05 -0400
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: =?utf-8?B?TGVuY3NlIEfDoWJvcg==?= <lencse@hit.bme.hu>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt
Thread-Index: AQHWLnmtoOX0Lok0+0aBCQF6UrSJLai0fxFAgAFiYQD///yb0A==
Date: Sat, 23 May 2020 17:40:04 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF0108A5BC52@njmtexg5.research.att.com>
References: <158995996438.13925.2934780472900149847@ietfa.amsl.com> <14002442-9713-d474-8012-bca5dcd6976c@hit.bme.hu> <4D7F4AD313D3FC43A053B309F97543CF0108A5BA22@njmtexg5.research.att.com> <598e85fd-cf9b-1cdd-61c0-3a76623145f9@hit.bme.hu>
In-Reply-To: <598e85fd-cf9b-1cdd-61c0-3a76623145f9@hit.bme.hu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [135.75.114.12]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-23_09:2020-05-22, 2020-05-23 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 suspectscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 lowpriorityscore=0 spamscore=0 impostorscore=0 cotscore=-2147483648 mlxscore=0 adultscore=0 priorityscore=1501 bulkscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2005230146
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/A3ccte20lFdszno74UjoEz16jJA>
Subject: Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 23 May 2020 17:40:45 -0000

Hi Gábor,

please see my replies below,
Al


> -----Original Message-----
> From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Lencse Gábor
> Sent: Saturday, May 23, 2020 8:29 AM
> To: bmwg@ietf.org
> Subject: Re: [bmwg] An Upgrade to Benchmarking Methodology for Network
> Interconnect Devices -- Fwd: New Version Notification for draft-lencse-
> bmwg-rfc2544-bis-00.txt
> 
> Dear Al Morton,
> 
> Thank you very much for your reply and for all the references! I was not
> aware of them and now I see that they have already solved several of the
> issues we wanted to handle. (Even some issues, which we did not mention
> in the draft, but I had in my mind, like using a high number of
> different source port numbers for the tests.)
[acm] 
Yes, starting out with 1 flow will probably reap the best performance, 
but this can be a distinguishing factor, as seen with virtual switches
(OVS, OVS+DPDK and VPP).

> 
> I am sorry for incorrectly using the word "bis" in the filename without
> proper understanding its meaning. I wanted to say something like
> "upgrade" / "update" / "amend", but not something like "replace" /
> "rewrite" / "reissue", which seems to be the meaning of "bis" according
> to https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__stackoverflow.com_questions_9105639_httpbis-2Dwhat-2Ddoes-2Dbis-
> 2Dmean&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=CuEaOl6rwRXBk_oZVCfrGd2JVKBcSeaUmdnefEFqhG0&e=
> We will not use the word "bis" in the consecutive versions of our draft.
[acm] 
Thanks and no need for apology - we are all learning from each other!

> 
> As for the non-overlapping areas of our draft and the documents you
> cited, I have not found anything in them about our suggestion for
> "Improved Throughput and Frame Loss Rate Measurement Procedures using
> Individual Frame Timeout".
> 
> If you think this one joins well into your efforts to update RFC 2544,
> then we could focus on this one first, and deal with some others later
> one by one in (in different documents).
> 
> What do you think of it?
[acm] 
Using a constant frame timeout for declaration of Loss (and delay) is
much like the IPPM WG metrics and methods (see RFC 7679 and RFC 7680)
for the production network measurements (Tmax). A constant waiting time for 
frames to arrive at the receiver simply excludes frames on an on-going 
basis. RFC 2544 has a waiting time at the end of the trial, where the
tester must wait for 2 seconds for buffers to clear but the first frame
sent has the entire trial duration+2sec to arrive. 

Perhaps the biggest potential change to the Throughput definition is 
whether or not we demand frame correspondence between sender and receiver,
so that we can calculate one-way delay. RFC 2544 and ETSI NFV TST009 only 
require equal send and receive frame counts to satisfy the zero loss
criteria in the Throughput benchmark definition. But in ETSI NFV TST009,
the Capacity at X% Loss Ratio metric-variant definition begins to infer 
frame correspondence. Later, the definitions of (one-way) Latency and 
Delay Variation and Loss allow a Sample Filter, which could be a constant 
maximum time-out for individual frames, as you suggest. Note that a Sample
Filter could be applied in post-measurement processing, assuming all the 
delays are available.

But the real question in my mind now, after looking into the possibilities
to take frame correspondence with a fixed timeout into account, is whether
we can find some examples where adding timeout criteria makes a significant 
difference for Throughput measurements that would not be accounted for by 
revised/modernized Latency and new Delay Variation Benchmarks and 
metric variants? 

Others should feel free to chime-in on this discussion, too!


> 
> Best regards,
> 
> Gábor
> 
> 
> 
> 
> 2020.05.22. 23:50 keltezéssel, MORTON, ALFRED C (AL) írta:
> > Hi Gábor Lencse,
> >
> > Thanks for your message and the draft you wrote with Keiichi Shima.
> > I remember you from the draft that Marius Georgescu led in BMWG,
> > resulting in RFC 8219.
> >
> > The BMWG has been updating areas of RFC 2544 as we determine the need
> > for better procedures. This practice began with RFC 4814 [0] in 2007
> > (Hash and Stuffing), and continued with RFC 5180 [1] for IPv6. Many
> > others have followed. RFC 8219 is yet another example.
> >
> > BMWG has had some help keeping RFC 2544 up to date. The ETSI ISG on
> > Network Function Virtualization has a Testing and Open Source Working
> > Group. This WG has prepared and frequently updated their specification
> > containing many of the topics you propose, but not all. The
> Specification
> > is TST009 [2], and it directly deals with your proposed topic of:
> > "An Optional Non-zero Frame Loss Acceptance Criterion for the
> > Throughput Measurement  Procedure". The TST009 benchmark "Throughput" is
> > equivalent to the RFC2544 Throughput (zero loss), and the TST009 (clause
> 8)
> > Metric Variant of "Capacity with X% Loss Ratio" covers the non-zero
> > loss case.  Besides Section 26.1 of RFC 2544 (Throughput),
> > TST009 clauses 9 and 10 go on to expand Section 26.2 (Latency),
> > and RFC 2544 26.3 (Frame Loss Rate) expanded in TST009 clause 11 (Loss).
> > Each TST009 clause contains one Benchmark and several Metric Variants.
> >
> > Getting back to BMWG's own updates, RFC 2544 Section 26.6 (System
> Resets)
> > was updated by RFC 6201. It also seems to me that RFC 2544
> > Section 26.5 (System Recovery) was treated more completely in a
> > recent effort, but reference escapes me at the moment.
> >
> > The astute reader has noticed that I skipped over RFC 2544 Section 26.4
> > (Back-to-back Frame Benchmark) until now. BMWG has work in progress
> > to update Section 26.4 [3], and we have discussed this draft at our
> > May 15 Interim meeting and again on the list this week.
> >
> > Updates for another RFC 2544 Section, Section 24 on Trial Duration
> > are included in ETSI NFV TST009 [2] clause 12.3, with other work
> > in progress under consideration in the BMWG: Multiple Loss Ratio Search
> [4]
> > and Probabilistic Loss Ratio Search [5].
> >
> > This was, by no means, an exhaustive roadmap to benchmarking best
> practices.
> > However, after revisiting all the RFC 2544 Section updates above,
> > I conclude:
> >
> > 1. that there is benefit in some of the work you propose, specifically
> > "Requirement of Statistically Relevant Number of Tests", and perhaps
> > some of "the Novelties of RFC8219" could be generalized, if not covered
> > by the work discussed and cited above.
> >
> > 2. some of the work you proposed joins with current efforts to update
> > sections of RFC 2544, but it does not constitute a "RFC 2544-bis"
> > by any means (as the file name indicates; I suggest to change the file
> > name in the next release to make this more clear and maybe separate
> > the draft in two or more specific topics).
> >
> > So, in summary, I encourage your work in non-overlapping areas.
> >
> > best regards,
> > Al
> > (as a participant)
> >
> >
> > [0] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_rfc4814_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=ad8oEp_p2xbcefSuK0ATByFwoNJOWhvoK7g5ztnvQUU&e=
> >
> > [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_rfc5180_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=754TbBQZqzjt2kHOuE5ibNFJw8PG_oRoh90_dg2Mc3o&e=
> >
> > [2] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.etsi.org_deliver_etsi-5Fgs_NFV-2DTST_001-5F099_009_03.02.01-
> 5F60_gs-5FNFV-2DTST009v030201p.pdf&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=zQVfOPRjpDVQBmSYZRrvOREkUAXK_lbXzISU1X8U7Yw&e=
> >
> > [3] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Db2b-2Dframe-
> 2D02&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=g1q55ezrR-4DCH1EY7dF5wUKmX2uLRmLeDAjNToksGI&e=
> >
> > [4] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dvpolak-2Dmkonstan-2Dbmwg-2Dmlrsearch-
> 2D03&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=sfE3wNSDZzIkfpYxuTT_xYe3PlD5Et495BTDaUoUcnM&e=
> >
> > [5] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dvpolak-2Dbmwg-2Dplrsearch-
> 2D03&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=59_-tqKC7laikb6Xte72W_6EuEaXkbWwhXX-XLaFC_8&e=
> >
> >
> >
> > From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Lencse Gábor
> > Sent: Wednesday, May 20, 2020 3:38 AM
> > To: bmwg@ietf.org
> > Subject: [bmwg] An Upgrade to Benchmarking Methodology for Network
> Interconnect Devices -- Fwd: New Version Notification for draft-lencse-
> bmwg-rfc2544-bis-00.txt
> >
> > Dear BMWG List Members!
> >
> > On the basis of our experience with RFC 8219, we have made four
> recommendations to upgrade RFC 2544 in our new draft "An Upgrade to
> Benchmarking Methodology for Network Interconnect Devices".
> >
> > Could you please read and comment it?
> >
> > It is really short: we tried to summarize the essence of our
> recommendations. We are happy to work out the details, if there is
> interest / support in BMWG.
> >
> > Any feedback is welcome!
> >
> > Best regards,
> >
> > Gábor Lencse
> >
> >
> > -------- Továbbított üzenet --------
> > Tárgy:
> > New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt
> > Dátum:
> > Wed, 20 May 2020 00:32:44 -0700
> > Feladó:
> > internet-drafts@ietf.org
> > Címzett:
> > Gabor Lencse <lencse@hit.bme.hu>hu>, Keiichi Shima <keiichi@iijlab.net>
> >
> >
> >
> > A new version of I-D, draft-lencse-bmwg-rfc2544-bis-00.txt
> > has been successfully submitted by Gabor Lencse and posted to the
> > IETF repository.
> >
> > Name: draft-lencse-bmwg-rfc2544-bis
> > Revision: 00
> > Title: An Upgrade to Benchmarking Methodology for Network Interconnect
> Devices
> > Document date: 2020-05-20
> > Group: Individual Submission
> > Pages: 9
> > URL: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_internet-2Ddrafts_draft-2Dlencse-2Dbmwg-2Drfc2544-2Dbis-
> 2D00.txt&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=DtKMUhM23geo2ithMSmG2RCoZHRgG4SoGqf-rsXEpxA&e=
> > Status: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_draft-2Dlencse-2Dbmwg-2Drfc2544-
> 2Dbis_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=-P3kCHa_yDm9OgWKze6huEyj152Qa_AHV6aE0va47V8&e=
> > Htmlized: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dlencse-2Dbmwg-2Drfc2544-2Dbis-
> 2D00&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=xOXkKe35i5dpHKbU4_kVcwFhxVgBYWYKYHNfNYnbybU&e=
> > Htmlized: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_html_draft-2Dlencse-2Dbmwg-2Drfc2544-
> 2Dbis&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=avTmfbV8OdkQxHRKhlQekhva_qdfNHzV4mHWbwNhCGY&e=
> >
> >
> > Abstract:
> > RFC 2544 has defined a benchmarking methodology for network
> > interconnect devices. We recommend a few upgrades to it for
> > producing more reasonable results. The recommended upgrades can be
> > classified into two categories: the application of the novelties of
> > RFC 8219 for the legacy RFC 2544 use cases and the following new
> > ones. Checking a reasonably small timeout individually for every
> > single frame in the throughput and frame loss rate benchmarking
> > procedures. Performing a statistically relevant number of tests for
> > all benchmarking procedures. Addition of an optional non-zero frame
> > loss acceptance criterion for the throughput measurement procedure
> > and defining its reporting format.
> >
> >
> >
> > Please note that it may take a couple of minutes from the time of
> submission
> > until the htmlized version and diff are available at tools.ietf.org.
> >
> > The IETF Secretariat
> >
> >
> 
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=KiUcsN1oXFmEApWDA9lU7WE3Vgruh
> BloEnlNdS9Qjmc&s=DgZPv_UxTelPpDBYn7p-byeUgGcmUI5yDr9ypLKtOSE&e=