Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

"MORTON, ALFRED C (AL)" <acm@research.att.com> Fri, 22 May 2020 21:51 UTC

Return-Path: <acm@research.att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1123D3A0D84 for <bmwg@ietfa.amsl.com>; Fri, 22 May 2020 14:51:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level:
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YZHPpSpPQfyp for <bmwg@ietfa.amsl.com>; Fri, 22 May 2020 14:50:57 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 717443A0BE0 for <bmwg@ietf.org>; Fri, 22 May 2020 14:50:57 -0700 (PDT)
Received: from pps.filterd (m0049459.ppops.net [127.0.0.1]) by m0049459.ppops.net-00191d01. (8.16.0.42/8.16.0.42) with SMTP id 04MLfC9b025236; Fri, 22 May 2020 17:50:41 -0400
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com [144.160.112.28]) by m0049459.ppops.net-00191d01. with ESMTP id 316n0wkd86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 22 May 2020 17:50:41 -0400
Received: from enaf.dadc.sbc.com (localhost [127.0.0.1]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id 04MLoeTq022910; Fri, 22 May 2020 16:50:40 -0500
Received: from zlp30493.vci.att.com (zlp30493.vci.att.com [135.46.181.176]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id 04MLoa9m022826 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 22 May 2020 16:50:36 -0500
Received: from zlp30493.vci.att.com (zlp30493.vci.att.com [127.0.0.1]) by zlp30493.vci.att.com (Service) with ESMTP id 5436E40004BF; Fri, 22 May 2020 21:50:36 +0000 (GMT)
Received: from clph811.sldc.sbc.com (unknown [135.41.107.12]) by zlp30493.vci.att.com (Service) with ESMTP id 2F9B240004A1; Fri, 22 May 2020 21:50:36 +0000 (GMT)
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clph811.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id 04MLoZ46116098; Fri, 22 May 2020 16:50:36 -0500
Received: from mail-blue.research.att.com (mail-blue.research.att.com [135.207.178.11]) by clph811.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id 04MLoWoA115695; Fri, 22 May 2020 16:50:32 -0500
Received: from exchange.research.att.com (njbdcas1.research.att.com [135.197.255.61]) by mail-blue.research.att.com (Postfix) with ESMTPS id 7820210A3018; Fri, 22 May 2020 17:50:31 -0400 (EDT)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0468.000; Fri, 22 May 2020 17:50:31 -0400
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: =?utf-8?B?TGVuY3NlIEfDoWJvcg==?= <lencse@hit.bme.hu>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt
Thread-Index: AQHWLnmtoOX0Lok0+0aBCQF6UrSJLai0fxFA
Date: Fri, 22 May 2020 21:50:30 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF0108A5BA22@njmtexg5.research.att.com>
References: <158995996438.13925.2934780472900149847@ietfa.amsl.com> <14002442-9713-d474-8012-bca5dcd6976c@hit.bme.hu>
In-Reply-To: <14002442-9713-d474-8012-bca5dcd6976c@hit.bme.hu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [135.75.114.78]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-22_10:2020-05-22, 2020-05-22 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 bulkscore=0 phishscore=0 impostorscore=0 suspectscore=0 adultscore=0 cotscore=-2147483648 mlxscore=0 malwarescore=0 spamscore=0 clxscore=1011 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2005220168
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/xEhrqdP59PAphKJES9viKM8Tt_E>
Subject: Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 May 2020 21:51:00 -0000

Hi Gábor Lencse,

Thanks for your message and the draft you wrote with Keiichi Shima.
I remember you from the draft that Marius Georgescu led in BMWG,
resulting in RFC 8219.

The BMWG has been updating areas of RFC 2544 as we determine the need
for better procedures. This practice began with RFC 4814 [0] in 2007 
(Hash and Stuffing), and continued with RFC 5180 [1] for IPv6. Many
others have followed. RFC 8219 is yet another example.

BMWG has had some help keeping RFC 2544 up to date. The ETSI ISG on
Network Function Virtualization has a Testing and Open Source Working 
Group. This WG has prepared and frequently updated their specification
containing many of the topics you propose, but not all. The Specification
is TST009 [2], and it directly deals with your proposed topic of:
"An Optional Non-zero Frame Loss Acceptance Criterion for the 
Throughput Measurement  Procedure". The TST009 benchmark "Throughput" is 
equivalent to the RFC2544 Throughput (zero loss), and the TST009 (clause 8)
Metric Variant of "Capacity with X% Loss Ratio" covers the non-zero 
loss case.  Besides Section 26.1 of RFC 2544 (Throughput), 
TST009 clauses 9 and 10 go on to expand Section 26.2 (Latency), 
and RFC 2544 26.3 (Frame Loss Rate) expanded in TST009 clause 11 (Loss).
Each TST009 clause contains one Benchmark and several Metric Variants. 

Getting back to BMWG's own updates, RFC 2544 Section 26.6 (System Resets) 
was updated by RFC 6201. It also seems to me that RFC 2544 
Section 26.5 (System Recovery) was treated more completely in a 
recent effort, but reference escapes me at the moment.

The astute reader has noticed that I skipped over RFC 2544 Section 26.4
(Back-to-back Frame Benchmark) until now. BMWG has work in progress 
to update Section 26.4 [3], and we have discussed this draft at our
May 15 Interim meeting and again on the list this week.

Updates for another RFC 2544 Section, Section 24 on Trial Duration 
are included in ETSI NFV TST009 [2] clause 12.3, with other work
in progress under consideration in the BMWG: Multiple Loss Ratio Search [4]
and Probabilistic Loss Ratio Search [5].

This was, by no means, an exhaustive roadmap to benchmarking best practices.
However, after revisiting all the RFC 2544 Section updates above, 
I conclude:

1. that there is benefit in some of the work you propose, specifically
"Requirement of Statistically Relevant Number of Tests", and perhaps 
some of "the Novelties of RFC8219" could be generalized, if not covered
by the work discussed and cited above.

2. some of the work you proposed joins with current efforts to update 
sections of RFC 2544, but it does not constitute a "RFC 2544-bis" 
by any means (as the file name indicates; I suggest to change the file
name in the next release to make this more clear and maybe separate
the draft in two or more specific topics).

So, in summary, I encourage your work in non-overlapping areas.

best regards,
Al
(as a participant)


[0] https://datatracker.ietf.org/doc/rfc4814/

[1] https://datatracker.ietf.org/doc/rfc5180/

[2] https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.02.01_60/gs_NFV-TST009v030201p.pdf

[3] https://tools.ietf.org/html/draft-ietf-bmwg-b2b-frame-02

[4] https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03

[5] https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch-03



From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Lencse Gábor
Sent: Wednesday, May 20, 2020 3:38 AM
To: bmwg@ietf.org
Subject: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

Dear BMWG List Members!

On the basis of our experience with RFC 8219, we have made four recommendations to upgrade RFC 2544 in our new draft "An Upgrade to Benchmarking Methodology for Network Interconnect Devices".

Could you please read and comment it?

It is really short: we tried to summarize the essence of our recommendations. We are happy to work out the details, if there is interest / support in BMWG.

Any feedback is welcome!

Best regards,

Gábor Lencse


-------- Továbbított üzenet -------- 
Tárgy: 
New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt
Dátum: 
Wed, 20 May 2020 00:32:44 -0700
Feladó: 
internet-drafts@ietf.org
Címzett: 
Gabor Lencse <lencse@hit.bme.hu>hu>, Keiichi Shima <keiichi@iijlab.net>



A new version of I-D, draft-lencse-bmwg-rfc2544-bis-00.txt
has been successfully submitted by Gabor Lencse and posted to the
IETF repository.

Name: draft-lencse-bmwg-rfc2544-bis
Revision: 00
Title: An Upgrade to Benchmarking Methodology for Network Interconnect Devices
Document date: 2020-05-20
Group: Individual Submission
Pages: 9
URL: https://www.ietf.org/internet-drafts/draft-lencse-bmwg-rfc2544-bis-00.txt
Status: https://datatracker.ietf.org/doc/draft-lencse-bmwg-rfc2544-bis/
Htmlized: https://tools.ietf.org/html/draft-lencse-bmwg-rfc2544-bis-00
Htmlized: https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis


Abstract:
RFC 2544 has defined a benchmarking methodology for network
interconnect devices. We recommend a few upgrades to it for
producing more reasonable results. The recommended upgrades can be
classified into two categories: the application of the novelties of
RFC 8219 for the legacy RFC 2544 use cases and the following new
ones. Checking a reasonably small timeout individually for every
single frame in the throughput and frame loss rate benchmarking
procedures. Performing a statistically relevant number of tests for
all benchmarking procedures. Addition of an optional non-zero frame
loss acceptance criterion for the throughput measurement procedure
and defining its reporting format.



Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat